One of the assumptions here is that there’s a way to write code that makes the AI smarter. The AI will write the better code to create the better AI which then writes the better code etc.
Is this totally true? Isn’t most of the gains in AI in “compute” and the size of the data and not in the code, which is basically the implementation of well known and not very recent algorithms.
Unless the AI can come up with better algorithms it seems to me that the gains here are marginal.
See the discussion at https://aiimpacts.org/trends-in-algorithmic-progress/ - "Algorithmic progress has been estimated to contribute fifty to one hundred percent as much as hardware progress to overall performance progress, with low confidence."
Aside from algorithmic progress, AIs could also speed things up by helping invent better ways to get lots of compute, or ways to use existing compute efficiently. There are reasons OpenAI/Anthropic/DeepMind etc are big companies that hire lots of people instead of just one person with a really big Amazon Web Services budget, and insofar as AI can do those people's jobs, it can abstract away all the parts of being an AI company other than raising money and throwing it at AWS.
Ok. I’ll read the link eventually (with low confidence).
The second point though about gains to algorithms driven by AI itself is suspect as all I can see from using chatGPT in daily life is that it is good at reflecting back to us what we already know, which is great. I’m not seeing it creating new knowledge as yet.
See the Codex example for how AI is already accelerating AI progress without any ability to "create new knowledge". Edison supposedly said that invention was 1% inspiration and 99% perspiration; the beginning of the takeoff curve will be AI that can automate the perspiration (or at least the part involving cognitive labor), which will eventually get us to the end where AI can potentially automate the inspiration.
My model of the state of the field of ML research is that we are in an "inspiration overhang". Lots of academics publishing small scale explorations of novel ideas. Most of these are currently overlooked by large scale explorations because the expected payoff per new unscaled idea is too small. Once AI is good enough to simply test the existing backlog with minimal researcher assistance, we should expect to see a bump in outputs from uncovering rare gems. This is even more true when you consider the costs of testing combinations of novel ideas, when they likely combine in nonlinear ways. Lots of backlog of combinations to grind through. Taking the expected improvements from this process into account, I don't think we'll be bottlenecked on inspiration until well past the point where the AI can start generating is own.
"I’m not seeing it creating new knowledge as yet."
This point has come up in several comments, but I think it is at least a fuzzy boundary, and possibly not a boundary at all. Almost all new knowledge can at least be viewed as "just" combinations of old possibilities or modifications of them.
When it isn't just combinations of existing ideas that work well together, it can be modeled pretty well by evolution. (Actually, even when it is it can be modeled quite well by evolution.) Remember that "selection" is a major part of evolution, it isn't all random chance. The problem is deciding what you're selecting for. "Survival" seems a pretty dangerous choice, though of course it's a necessary component.
Yes. Genetic algorithms have been used in AI (and other flavors of optimization problems in computer science) to good effect, and they are effectively a small scale model of evolution. (Albeit I don't know if combinations of genetic algorithms and neural nets have been tried, and how successful or unsuccessful they have been, if tried.) Genetic algorithms can be creative, at least in the sense of generating solutions that the programmers didn't anticipate. As you said, the selection part is crucial.
The algorithmic progress discussed in the link is mostly not from fields related to AI. SAT, go, chess and integer factorization all seem unlikely to help in training neural nets.
Also, this list seems to be selected for stuff where there was algorithmic progress. As a counterexample, take matrix multiplication. The naive method runs in O(n^3) and is as old as the concept of matrix multiplication. Then Strassen comes along in 1969 and finds a faster method working in O(n^2.8) which is actually helpful for n>100. For the practical side, that is the end of the story, because Coppersmith-Winograd (1990) is only faster for matrices too large to handle with present day computers. From an application point of view, faster matrix multiplication would help with a lot more of problems than better chess engines, so it feels unlikely that there are any low-hanging fruits left.
Or consider cryptosystems. When they are proposed, there is generally an assumption that the problem one would need to solve if one wants to break the cryptosystem is hard. If a better algorithm is found, this can break the encryption. Sometimes that happens (slightly more often if the NSA originally proposed the algorithm), sometimes it does not (then the cryptosystem is "thought to be secure").
Typically, there are diminished returns for algorithmic improvement. If the difference between the first idea which comes to mind and carefully thinking about it for a day can be many orders of magnitude. Then thinking about it for another day will likely yield much more modest improvements, instead you would need to do an PhD on the project or something. For a problem which is already heavily researched, one can easily spend a a lifetime without making measurable progress.
OpenAI claims a factor 44 of algorithmic speedup between 2012 and 2020, which is impressive. [1] However, during this time span, there amount of resources spent on ML R&D exploded. In 2019, OpenAI got a billion from Microsoft. 2023 they got 10 times as much. If their budget increases exponentially and they manage to get ten trillions in 2031 I think they might get another factor 44 speedup by then.
Also, with algorithm research, one really smart person is probably more valuable than ten slightly smart persons. If AI gets to average-college-student level of proficiency, this might not speed up algorithm improvement at all, the cost of having college students work on that is already close to zero.
For hardware improvements, I am even more skeptical. Getting to smaller feature sizes for chips is something Intel and friends spend billions of dollars on. The fact that the i386 used a 1um feature size while current CPUs use 7nm or something is not because the designers of the i386 were stupid. It is rather that between 1985 and 2023, there were a lot of small incremental improvements with all sorts of technologies required. Solving these deeply domain specific challenges (perhaps building a more precise laser or ebeam) is not something where I would expect GPT7 to help much. I can sort of see AI improving chip layouts, but I am unconvinced that there are many OOM of improvement to be had.
I agree that the ability to improve algorithms that have been around for a while is hard and more likely to come from a unique insight rather than throwing a bunch of people at it. But that unique insight isn't necessarily a consequence of being "really smart" -- some level of intelligence and background knowledge is surely a prerequiste, but once you get past that progress is usually made when someone comes to a problem with a new perspective.
Many (I would argue most) advances in pure mathematics are a result of the right person encountering the right problem in the right context (something in their background or whatever they are currently thinking about makes them percive a problem in a new and different way). This is one factor behind the myth that math is a young person's game: most successful mathematicians are successful precisely because they happened to be the right person to encounter the right problem at a point early in their career, which is what enabled them to stick around (get tenure).
While this is surely overstating things, it is not ridiculously far from the truth to say that most mathematicians have one or maybe two really new ideas that are unique to them. They spend most of thier careers searching for ways to apply those ideas (and eventually people start bringing suitable problems to them).
It is not hard for me to imagine a similar situation with AI. Rather than needing to ber "really smart", a "pretty smart" AI might still be likely to try approaches that a human would not (a uniquely new idea), and this could lead to sudden improvements, even on problems that have been around for a long time.
If there was a well known open problem I urgently needed to solve and had to choose between using 100 experienced mathematicians (who likely have already found the problems easily solved by their one or two uniqely new ideas) versus spinning up 100 random replacements who might have less experience but get spawned with a random unique new idea, I would take the latter.
Interesting, if I had a difficult problem to solve, then I would try to get someone like Saharon Shelah interested (maybe this is no longer an option), or Terence Tao. As far as I can tell, in every field there is a short list of people who are well known to be great problem solvers because they always apply a fresh approach, but who also have a large toolkit of techniques to draw from. The new problem would likely be interesting to such researchers if they thought a novel synthesis of what they knew might lead to progress, or if it meant developing a new technique. Only if that didn't work would I try the random approach you suggest.
If it was a new problem I would agree (Terry Tao in particluar would be a very good first person to ask). I deliberately chose the framing "well known open problem" to restrict to problems that the short list of people you refer to have likely have thought about and do not know how to solve (the twin prime conjecture, for example, is something Terry has thought a lot about and he would be the first to tell you that a fundamentally new idea is needed).
But the main point of my comment was not about how we conduct mathematics research but simply to argue that it is not inconceivable to me that sudden rapid progress could be made if (via AI) you had a way to suddenly increase the number of "pretty smart" entities working on a problem by orders of magnitude, even if none of them are "really smart". But this assumes the entities are "different" in some meaningful way (this might be less true of AI than humans).
Diversity would certainly be interesting, but we only have a mild form of it if we duplicate the same pretrained network many times. I think we would need genuine differences in training, maybe some architecture differences, and an increased tolerance for portfolios of systems which in combination do well but where individual components may sometimes do very badly. Alas, most work currently focuses on trying to beat a single measure and reducing variance is seen as a win.
There is a question, however, as to how much progress should be expected from the current (at any particular point in time) status. Remember the various boom-and-bust cycles that AI has gone through before. I tend to think of the advancement as being over a very rough n-dimensional surface (where I don't know the value of n). When you advance in one dimension, there's lots of low-hanging fruit, but when you pick that, you need to face a difficult pitch to advance further. In my EXTREMELY informal projections, I do put an AGI at around 2035 (with large error bars), but I don't expect it to match human capabilities. Along some dimensions it will be superhuman, and along others probably sub-chimpanzee.
Humans seem to learn stuff from less training data than AIs do; this suggests that better learning processes exist.
There's also the quantum leap of cracking GOFAI. The reason we use neural nets is because we don't know how to code a GOFAI, not because neural nets are better; at small scales where we actually can code things properly my understanding is that coded programs run OOMs faster, and obviously if you know how to design something you don't need a training run at all so that also removes OOMs of time. (My understanding is that this is why MIRI assumes fast takeoff.)
I'm not sure. Each eye contains about 125 million rods and cones, all providing data several times per second. And that's just the eyes. It seems to me the brain gets an awful lot of training data, but must also convert it INTO training data, through some mysterious process.
I mean, yes and no. You can't get more than 1 byte per letter out of text no matter how good your eyesight is. I think that's also factored into the claim to some extent, since you absolutely can feed an AI superb-resolution video as training data.
While you may not be able to get more, you can get less, and that can be useful. You're taking in a lot of context with that byte per letter that helps you. E.g. if you're reading a sign held by someone shouting on a streetcorner vs in a well-made scholarly book at a university.
You can get arbitrarily high amounts of information per symbol, it depends on your alphabet size. Which depends on how well you can differentiate symbols, and cameras are absolutely nowhere close to the effective pixel numbers of an eye (like the man said).
That's clearly wrong. You're ignoring the importance of various things like font, bold, italic, etc. Also position WRT other letters.
OTOH, compression implies you get much less than one byte per letter when just considering their alphabetic designation.
It's my guess that there is no simple relationship that captures the value you're looking for, but maximal compression comes closer...however it would need to compress ALL contextual data, not just the individual letter.
I'm pretty sure "position WRT other letters" is already captured; I didn't say "one byte per letter divided by n!", after all.
You can get a x3 from bold/italic while staying within a byte; there are only 26*2 letters, 10 numerals and ~25 common punctuation marks. x4 would put you slightly over, but not even 9 bits, and bold+italic's rare.
I suppose you could count font, but I don't think most people consider that to be useful information.
If I show an artist a single image file of a specific character (who they have never seen before) and a single image file of the work of a specific artist (who they have also never seen before), and ask them to draw a picture of that character in the style of that artist, I am pretty sure they will do much, much better at it than a Dreambooth iteration on a stable diffusion model, given that same amount of training data.
Take domain specific training data: the amount of text that went into training GPT-4 is more than any human can read in their lifetime.
For tasks like image recognition, I think the resolution of the image is not that important. A kid can learn what a cat looks like from 1024x768 pictures just as well as from seeing cats in 125Mpixel with its own eyes.
Counting data that way is futile because you're not getting anywhere near that much *useful* data (for intelligence training purposes). Someone who stared at a blank wall their whole life would have the same 125 million rods and cones firing but would gain zero knowledge beyond what that blank wall looks like.
Reading a 400 KB textbook should give you less than 400 KB of useful data - virtually all of which comes from the words themselves and virtually none of it from the other gigabytes of raw visual data your eyes processed during that time.
While I agree that better learning procedures probably exist, I’m not sure humans prove that especially well, because I think both you and the commenters below you aren’t thinking about training data correctly.
This is because the training procedure for your eye, or your brain, is not just a function the data you’ve been exposed to. You have to account for what was also programmed into it before you even started learning.
Your brain is actually also a function of the entire biological evolutionary process which designed it. In some sense, you could say the training set was the entire history of our species starting from when we were single celled organisms; though of course the optimization procedure wasn’t gradient descent as it is with neural networks, it was genetics and evolution. Look at it that way and it’s arguably quite inefficient.
I don't know whether it makes sense to think of biological evolution as deep learning. Maybe. But the outcome of the learning process was a brain optimized to absorb certain things in certain ways. We're set up to be very attached to our parents. That means that interest in them and extraction of info from what we see and hear of them is turbocharged. We're set up to learn language. First kids learn it in a parrot-like way, and say so things like "I went to the park" -- use of the correct tense of the irregular verb *go* is pure imitation. Then in a year kids have somehow extracted the rules of grammar. They know that to make past tense you add -ed. So now they're saying "I goed to the park." In another year they get it that some verbs have irregular tenses, and they're saying "I went to the park." Don't we need to do some equivalent of that with AI -- set it up so its optimized to learn certain kinds of things?
I definitely wouldn’t call biological evolution “deep learning”. I wouldn’t even call it “learning”, but it is certainly an optimization process.
I think what we’ve learned with LLMs is that having a large capacity system and exposing it to a lot of data naturally makes it better at picking out patterns and structures. So that sort of quicker pattern matching occurs on its own.
I think of it like this: in a vague, hand-wavy, not-quite-perfectly-analogous way, the large pre-training we do with LLMs (where they’re trained on the whole internet) is basically the stage where we try to get that initial structure that evolution gives our brains. That won’t hold up if you look too closely, but the idea is that we shouldn’t expect to develop a generic algorithm that learns as fast as the brain right off the bat - because the brain isn’t actually starting from zero knowledge.
Please forgive me for not reading what you wrote on LW before replying. I will read it in the next day or 2. But I'm eager to get your response to another, related idea. Reading Scott's summary of Davidson, I noticed Davidson seemed to think of further development of AI mostly in terms of expending more "compute" on it, and that sounded like it meant giving it longer trainings on bigger samples of language. There was occasional mention of, "and maybe we'll get good results from some new model of training too," but it seemed like the assumption was that doing deep learning on a vaster and vaster scale would get us there -- get us to AGI or beyond. And I find that hard to believe.
Here's why: (1) I get that LLM's somehow develop a remarkable "grasp" of things solely from exposing them to a lot of data. Clearly they pick out patterns and structures -- for example, they use correct grammar and syntax. However, I don't think they can articulate what they know. (I may be wrong. I'm a psychologist, do not work in tech.) Most people also cannot articulate the grammatical rules of their language. But some can -- somebody sat down and figured out that their language had certain ways of indicating past, present and future actions, of expressing uncertainty ("might go"), of making things plural, etc etc. And they gave each way of doing this a name, and laid out the rules, and gave a name to the exceptions: irregular verbs. And making all that explicit makes possible other stuff. It's useful to people who speak another language to know the grammar rules of the new one. And of course LLM's have learned a lot of other patterns and structures besides English grammar, and being able to "understand" them, as measured by ability to enumerate and describe them, also seems essential to its developing human-level intelligence . So if I'm right that LLM's cannot articulate what the structures are, and reason from and about then, that seems to me quite a severe limitation on what they're "minds" are capable of. (2) People as they grow up develop a model of langauge but also a model of the world. Seems to me that LLM's just have a model of language. Of course they can *say* lots of correct info about the world, but that's very different from having the model up and running. Here's an example of AI's ignorance about the world. I gave GPT4 a little puzzle: A guy's in a locked room with an open window 30 feet off the ground. All he has is a pair of jeans, a chamber pot and a pocket knife. How can he get down? So GPT, which of course has read lots of stories about escape from high places, said cut the jeans into strips and use strips to make a rope. However, as I pressed it for details, it made lots of mistakes because of how little it understood about the physical world. It had no idea how wide the strips of cloth you'd get from the jeans would be if you wanted enough length to reach the ground. (Even though it's easy to figure out the width of jeans legs from avg. circumference of human legs.). It said to anchor the rope in the room, even though I had not mentioned anything that could be an anchor point. And the method of getting down! First, it assumed there was a big tree about directly across from the window and about 20 feet away. so it said tie the chamber pot on the end of the rope and swing the rope til it wraps around the tree at the height of the window, 30 feet up, then cross via the rope to the tree. But you can't swing the rope towards the tree because you can't get a pendulum going at right angles to your building, only a pendulum that swings to right and left. And even if you could, the rope would not wrap around a branch securely when it hit one. Anyhow you were supposed to cross to the tree on the horizontal rope and then, presumably, climb down (tho GPT did not mention climbing down). Doesn't AI need a model of the *world*? Plus, of course, the ability to see regularities and make them explicit so as to work with them.
So because of both these 2 ways LLM's are dumb, I can't see how more compute is going to make them able to do lots of the things we do -- build theories, invent, look to one situation to provide analogues of what might help in another situation, etc. Seems like we *must* incorporate a new form of learning to get an AI that can do what a human mind can do. Am I missing something?
First of all, you should feel zero obligation to read my post on Less Wrong, and it won't answer your questions anyway.
For your first question, you're right that LLMs aren't great at creating new ideas, so figuring out the grammatical rules might be hard just because LLMs aren't especially creative. Nor, as you said, are they very good at self-reflecting on their own latent knowledge. Yet if you gave them a made up language with some translation, I suspect they'd be surprisingly capable at picking out grammatical structures given enough samples, because they have been exposed to the rules of grammar in other languages.
Additionally, you have to account for "emergent abilities." LLMs arbitrarily develop new abilities as they scale, like, say, learning how to multiply. This isn't just getting better at some task it already knows how to do, it's gaining the capacity to perform completely new tasks altogether. Who's to say genuine creativity/self-reflection isn't just a few orders of magnitude scalings away?
As for question two, you're getting into controversial territory. My own thoughts, which I suspect are roughly shared by many others (though not all) is that LLMs do have some rudimentary world models, and that these models will only get better as they scale. The issues you are seeing are caused by a combination of LLM hallucination and the fact that its world models have very large gaps. Some of these knowledge gaps will go away as LLMs are trained on more context than just text, and hallucination will undoubtedly improve as new techniques are developed.
The most important question is still "does all this ever get us to AGI?" and unfortunately no one can answer that. But I do think it's plausible that it could, based on current trends.
> Humans seem to learn stuff from less training data than AIs do; this suggests that better learning processes exist.
The thing that spooked Hinton, btw, was the realization that AI might be better at learning than humans. [From my perspective it looks like a bag of advantages and disadvantages, so it's not quite obvious where the balance is now, while at some point I expect AI to be better.]
> For 40 years, Hinton has seen artificial neural networks as a poor attempt to mimic biological ones. Now he thinks that’s changed: in trying to mimic what biological brains do, he thinks, we’ve come up with something better. “It’s scary when you see that,” he says. “It’s a sudden flip.”
I think that computers currently have the ability to be much more flexible than humans--if it turns out that visualizing a problem in 4d is the right way to solve it, I think neural networks will notice that much more readily and adapt to it much more fluidly than humans will, since a lot of our relevant processing seems hardcoded to deal with 3d.
But there are things that humans seem to do much better than computers. At some point we'll figure out what the humans are doing and be able to write or discover code that does similar things.
The next few sentences in the interview also help explain it:
<blockquote>
As their name suggests, large language models are made from massive neural networks with vast numbers of connections. But they are tiny compared with the brain. “Our brains have 100 trillion connections,” says Hinton. “Large language models have up to half a trillion, a trillion at most. Yet GPT-4 knows hundreds of times more than any one person does. So maybe it’s actually got a much better learning algorithm than us.”
In machinę learning it is (I believe) a generally accepted heuristics, that you should have at least three times the training examples than model parameters, or else it will overfit and will not be able to generalize properly. This is one reason why if you start training large models from scratch you need tons of data. So scaling up our current models orders of magnitude will require orders of magnitude training data. Not very likely. Also the claims about knowledge are dubious: there is some knowledge there, but there is also a lot plausibly sounding hallucinations:
See the analysis of essay on ancient history written by ChatGPT3 which sounds impressive and plausible for non-historian, but which is largely erroneous. Of course this was ChatGPT3, ChatGPT4 may better, and writing plausible bullshit requires a lot of knowledge as well (and understanding of the prompt) so it is still very impressive in itself. But this is what people call the lack of real understanding of what the text actually mean by language models. Yes, some actual knowledge about the world can be emergent from the text itself, and where it does that's where LLM's shine. Where it does not (which is more often than people admit), LLM's hallucinate. This is also why a large (and very manual) part of training of LLM's is adjusting the model already trained on the text corpus by training with people which prompt the model and direct it to more "sensible" answers. This part is also unlikely to scale very well.
Or at least we'll have the option to do that. Why exactly do people want to create an alien who is as smart as us, and get it woven all through our world via the role it plays in air traffic control, communications, medical research and treatment, entertainment, etc.? And THEN tweak the thing that has its fingers in all our pies to be smarter than us? If that idea gives you a shiver of delight, how about instead signing up for some sort of in-person role-play game with an alien control of a lot of the environment, and players run around in the woods trying to spy on the AI and figure out whether it's murderous, and if it is shoot it with paint pellets -- or raccoon turds-- or whatever?
I personally am very unenthusiastic about having a non-human superintelligence on the planet. Do I get a vote, or is planet earth now the official playground of engineers who really dig machine learning etc.?
You can't say that, because humans always learn in the context of everything they've previously learned. It may be true, perhaps, but you really don't have the evidence to assert it.
A better test would be to take a bunch of16X16 grids with a random pattern of colored dots in it (no more than one per cell) and see how quickly people learned it vs how quickly the AI learned it. You'd need to repeat this test many times, because some of the random patterns would be meaningful to the human. (I suspect you'd need to give the humans a very strong incentive to learn, or they'd just avoid the process.)
whereas in most cases the training of AI's starts from tabula rasa. Easily available pretrained models which can be further customized or used as parts of other tasks, as well as development of methods which permit easy incorporation of prior knowledge such as inductive logic programming may close this gap significantly. I still believe true consciousness and intelligence may require far more than simply scaling existing methods (or else human level intelligence would appear more frequently in nature). Ok. this is weaker argument than I would want it as big brains are expensive and may not be worth it, but (unless I am mistaken, and I may very well be as a non-biologist) human brain is not just a reptile brain with more neurons and connections, there are also significant structural differences
Perils of rewriting comments; I initially had something about how some or most of this is because of preloading, but being able to preload would still be a huge benefit (I mean, particularly so for alignment).
Not sure about reptile brains, but while there are a couple of structural differences (the neocortex is the one I know) the big difference between human brains and other mammals' brains is indeed that one section of it - the cortex, particularly the frontal cortex - is much larger.
>Ok. this is weaker argument than I would want it as big brains are expensive and may not be worth it
This. Humans are sufficiently minmaxed that we would literally starve to death in the wild without food preparation techniques and/or constructed traps; the brain uses a fifth of total calories in an adult, and also massively complicates and lengthens child-rearing. It's tricky for evolution to make the leap over from the "wild animal" regime to the "technological species" regime.
I am not sure about foundation models. I do not believe GPT's used BERT or T5 (although I may be wrong, I did not use LLM's yet, and only now intend to dig deeper). The cited paper (I have not read it, just watched YouTube) was about reinforcement learning of games, and not NLP anyway. But of course I have indeed somewhat overstated my point by saying that "in most cases training starts from tabula rasa", which, especially in case of NLP, but also image recognition, is increasingly not true in practice (even if it was true-ish just a few years ago). But the main point is still valid: people use pre-trained models precisely because customization for specific tasks of existing model does not (usually) require such ridiculous amount of data and repetition as starting from scratch. And there are some developments in neuro-symbolic AI which, by hardcoding some background knowledge, may permit far more efficient learning (this is distinct from pre-trained models), see e.g., https://www.jair.org/index.php/jair/article/view/11172
The "foom" scenario, as far as I am concerned, is pure magical thinking. It is less likely than aliens, angels, or Harry Potter-style wizardry; and should be treated as equally implausible in polite conversation. It is Yudkowsky assuming "something smarter than me will be able to work magic" when the people who are already smarter than Big Yud know that he is dreaming the impossible. As far as "human hope for the impossible" goes, I am thoroughly sick of the singularitarians. Maybe that makes me "post-rat".
The fundamental problem is this: in 1900, about 30% of human effort in the USA went towards agriculture. Today, that is at most 5%. Could "foom" possibly make that -20%? Of course not. We have gone from "lincoln-log scale" computing to 5nm dies in 100 years. Could "foom" possibly make them 100 times smaller than atoms? Of course not.
There isn't going to be an "inflection point" in ten years that speeds things up. In ten years, we will already have finished building "peak AI". It could take even less time with peak human effort, but the cult of "Attention is All You Need" and dysfunction at big tech companies are already slowing things down (which, everybody agrees, is probably a good thing).
Of course, there are a lot of things AI could do today that it isn't doing, and as people catch up with the technology, there will be certain societal trends that might be portrayed as a phase-change in development. None of that will help make AI "smarter", regardless of how many pundits claim otherwise.
As far as the specific question about "write code that makes AI smarter" - to a very limited extent, yes. As a metaphor, consider antenna design. AI has been able to surpass human ability in this field for decades, but it was a limited one-time effect. It isn't remotely close to "you can now hear radio stations from the other side of the world" in impact, which is my interpretation of what many people are promising from "AI improving AI".
There was a recent news cycle about AI "optimizing a sorting algorithm" - after the hype it seems to be a small improvement on a few cases, with no potential for further improvements.
"As a metaphor, consider antenna design. AI has been able to surpass human ability in this field for decades, but it was a limited one-time effect."
Yes, a very important question will be to what degree the benefits from AI (and, more generally, from any form of intelligence) saturate.
I have two conflicting intuitions about this. To add another metaphor somewhat analogous to yours, albeit in general computing rather than in AI, there are optimizing compilers. And there have been compilers written in the language that they compile (one can get them started by writing an interpreter, or a simpler compiler in another language). Applying an optimizing compiler to its own source code _does_ gives a recursive self-improvement ... _once_. So that first intuition is indeed to expect a lot of saturation and diminishing returns.
The other intuition, however, is that another aspect of intelligence is that it allows keeping more possibilities in mind at one time. It allows trying expanded sets of heuristics. If the arsenal of approaches is widened, it may be able to go much further before running out of steam. Now, this doesn't get around physical limits. If someone has a problem where they need to use the densest nonradioactive material possible at standard temperature and pressure, and osmium isn't dense enough for what they want, then they are out of luck, no matter how bright they are. But most problems aren't that crisply constrained, or are really subproblems of a problem that can be reformulated in some way.
One thing to keep in mind (ha!) is that the more complex the problem the more "one time improvements" you can probably get out of it. e.g. If you have an AI that is "smart enough", it would be able to do a full optimization pass on the entire AI pipeline, from chip fab, chip layout, processor design, interconnections, OS, compiler, general purpose algorithms, AI-specific algorithms, neural network architecture. Plus adjacent things like math itself.
Once an AI can do that, and boost itself an OOM or two, it can then either run faster or increase it's capacity, and then do another pass of improvements, which will be to improve things it couldn't see the previous time...
"One thing to keep in mind (ha!) is that the more complex the problem the more "one time improvements" you can probably get out of it."
That's a good point. There also tend to be ways of modifying subproblems to make it easier to improve their solutions. As you said, if an AI can look at the whole pipeline, it may well find that there are improvements that can be made that cannot be made if the stages are looked at in isolation (with excessively restrictive constraints on their interfaces).
Well, we know we can have a human level AI in a compact package, using a fraction of energy of GPU farms, because that is us. So there is quite a lot of room for improvement and development, even if significantly "superhuman" AI's are impossible because of some fundamental physical limits (which I guess is quite possible). The point is it will almost certainly require fundamentally new ideas and technologies. The current success is the result of conjunction of significant (but hardly fundamental) progress in neural networks, availability of enormous amounts of data (free of charge) from the internet, and enormous increase in power of computing hardware. So now we are reaping the fruits of this conjunction, and progress seems quick and with no end in sight, but it is unclear how long this will be able to continue: we are not going to have much more training data, and current hardware technology starts hitting hard physical limits. One way in which the progress can significantly continue, even with the current tech, is through the use of pretrained models: starting from scratch may be expensive, but if you can incorporate prior knowledge, you do not need to repeat the same training again and again: you build upon the previous work. Whether this will lead to steady, linear progress, or exponential explosion if we start building hierarchical structures, is anybodies guess.
Remember, though, a human brain takes about 20 years to train and each one only ends up being able to do a small fraction of the jobs that at least one human can do.
Frankly, I would be more impressed by an AI with a vocabulary and actual understanding of physical world of a five-year old child, and trained on the amount data and interactions a five year child receives in its short life (which is a lot, but not that ridiculous big data LLM's need). Of course, you can discuss with ChatGPT about programming and ancient history, which (usually) you cannot with a five-year-old. But still, our learning seems so much more efficient. The weakness (and strength at the same time) of language models is that they are build from a corpus of text alone, whereas a child builds its understanding of the language and the world from both the "training language corpus" and actual physical interactions. So for a child to build a model of what, say, a cat is, it suffices to play for a moment with a cute fur ball, and have parent to point at it and say "this is a cat". LLM's, on the other hand, build their understanding of catness from contexts in which a cat appears in the text, and for that to work well you need a lot of text. It is still a mystery why certain kinds of generalizations are easy for humans: After encountering a cat once a child may call dog a cat, but just few interactions (and not looking through thousands of pictures of cats and dogs) will make a child a very efficient classifier.
As to the jobs, current AI's, and especially LLM's, cannot really do our jobs yet: people thinking they can end up very badly:
And while it is very impressive when ChatGPT is asked to write a program which solves some problem, this is akin to having a great intuition borne of long experience and guessing the solution. Many times it may be brilliant, but every so often it will be a complete idiocy (well complete idiocy is optimistic: it is at least easy to spot, but there might be subtle errors).
Quite an incident! I wonder when we can expect the first successful use of an LLM by a lawyer to get themselves disbarred... This case sounds like a near miss...
I share your intuition about intelligence and its limits, but I find that people who blithely talk about how wrong Yudkowsky is on these things rarely address any of his actual arguments. I've tried to, because I don't find AGI in the full sense at all plausible, but when I try I fail. Do you have any specific criticisms, or just a general derision based on other people supposedly smarter than him knowing why he's wrong?
I get the possible "not even wrong" aspect of anything that sounds sci-fi. Couple of things, however.
1) Eliezer's actual guesses about what is possible exclude magic, but not 'magic'.
2) The people actually trying to build AGI speak and act as if the good parts of #1 are possible, but the bad parts of #1 are mostly impossible.
If everyone at Meta or OpenAI or whatever were saying "look guys don't get too excited -- this is basically going to be autocorrect and the instagram algo rammed together and put on steroids", that would be one thing. The smart people in the room acting as if superintelligence is an order of magnitude more likely than superintelligence-related problems are is what draws his cricitism.
I don't believe your puppy is going to grow up to be a thousand-pound hellhound, but if you do believe this I feel like we need to talk.
The "foom" scenario does seem unlikely to me, but not *that* unlikely. I can imagine scenarios where that could occur. (Most involve a virus transmitted over the internet, with each node of the AI not being that intelligent.)
Remember, the "foom" scenario doesn't require that civilization, or even the AI itself, survive the "foom". If it happens overnight, then it doesn't require that the resulting system be stable long enough for people to eat their groceries. I would, in fact, expect it to be very unstable.
> Could "foom" possibly make them 100 times smaller than atoms? Of course not.
But there might be a way if you’re clever enough you can make one atom do the work of 100 transistors or figure out a way to make a transistor equivalent out of something more compatible than atoms
Minimally, a foom scenario only needs to progress to a level that's dangerous to humans at a speed that makes it impossible for humans to notice and curtail. You seem to arguing against a form of foom that goes to infinity.
As I understand it, people working with the Llama-derivatives have found some significant runtime speedups, which allow the smaller models (7B, 13B) to run on unspecialized little computers like my laptop here.
Yes, it’s clearly true. If you read cutting edge AI papers, they are coming up with new algorithms constantly (transformers were 2017 and every week this year there is a new theoretical advance that’s unlocking new capabilities, particularly when you include prompt engineering).
It’s true that “moar data” is the overarching paradigm currently, but under the covers you need a lot of scientific theory and engineering to actually handle that data. The data driven paradigm was a shift from hand-tuned architectures/algorithms to “dumb” networks but that doesn’t mean the challenges simply evaporated when we obtained enough GPUs.
I misunderstood you then. Is your crux that you need to see AIs coming up with these algorithmic advances to believe it’s possible?
I had taken you to be questioning whether it’s possible at all (human or AI) to write such code
> One of the assumptions here is that there’s a way to write code that makes the AI smarter.
But if your crux is whether AI can write the code that humans are currently writing, I’d say it looks likely, assuming current capability trends continue. GPT can already write software and build simple PyTorch neural nets under human direction (“intern” or “junior developer” level). At some point it will be able to take some high level ideas from a human and search the broad concept-space for feasible implementations (senior engineer / grad student level) — FWIW I think GPT-4 can do this in many domains with appropriate prompt engineering and a few man-years of engineering work on the surrounding scaffolding. The final hop to generating the high-level novel ideas (PI / CTO level) is of course the hardest part (at that point you basically have AGI) but I don’t see a reason to doubt it’s possible with enough compute.
Yeah, this is an ambiguity in "AI can do X jobs". I assume the real report addresses the ambiguity somewhere, although I don't remember the details. I think they mostly don't matter since the important question is when they can automate AI design jobs, where this concern is addressed by the bottleneck parameter rho.
At some point AIs can use cognitive labor to design robots to do physical labor, but that might be after the point this forecast tries to model.
Intelligence is ill-defined, and variable. You may not want to listen to a song written by a genius chess player, and a song-writer might be terrible at chess. This doesn't even count the fact that I, who I consider intelligent, often make really dumb mistakes.
Machines do what they are designed to do, and could be considered idiot-savants. Universal Turing machines they may be, but I wouldn't use ChatGPT to balance my checkbook.
> Machines do what they are designed to do, and could be considered idiot-savants. Universal Turing machines they may be, but I wouldn't use ChatGPT to balance my checkbook.
I think a large part of what makes AI exciting is that they're actually quite good at doing things they weren't "designed for". Certainly ChatGPT seems to be quite functional at all kinds of stuff that the original designers never really intended.
I don't think this is going to hold true in the same way going forward. Any day now, the AI will just be able to call an existing checkbook balancing API or write the code if it doesn't have access to it. These capabilities already exist in a rudimentary form.
And THAT, I think, is the way forwards. LLMs are specialized for handling language, but that's a very poor tool for many tasks. Some of the other tasks will have related structure, like voice analyzing. Others will be very different, like balancing on one leg. Balancing a checkbook is in-between, but it's would be a lot better done with a spreadsheet like module than with a language model. (And, of course, the spreadsheet module would have lots of other uses.) If the LLM is good enough, then it could be interfaced with an interactive Python module, some some equivalent at an even lower level. Perhaps LISP could make a comeback.
If you have a way of testing their work, AI programming can be done (with very low efficiency) by entry level programmers. They'd produce a LOT of really bad routines, but you throw those away. So if AI-programmers are cheap enough (and you can test their work) then you can scale up your programming staff with AIs at a low level of skill.
Since it's usually a lot easier to check a result than to derive the result, this is probably feasible, but may not be economic until a certain level of AI-skill is achieved. But this probably means that one particular bottleneck isn't there. (I.e. the bottle-neck limiting the rate of self-improvement past basic skill level.)
I believe we have the robotics technology to make a plumbing robot *today*. It just doesn't make any economic sense, since without an AI to control it you still need a human plumber as a remote pilot (also robots are still pretty expensive). Once we have an AI smart enough to do these tasks, I suspect we'll have android-type robots capable of physically doing almost any job* pretty soon (<~5 years) afterwards
* I don't mean one robot made by Tesla which can do anything, I would expect that to take a few more years. Robot plumber, robot cook, robot launderer would all be different robots at first.
What’s in that video is nice and all, but ir’s not even close, and the robot also seems to have limited agility and range of motion. I can quite easily see semi-decent cooking, because you can have a carefully controlled mise en place for it that it can then work with, but fixing arbitrary plumbing seems _far_ more difficult.
> the robot also seems to have limited agility and range of motion.
I think you're assuming your conclusion here because you can't possibly determine that from the video. And the arms at least seem to demonstrate they have the full human range of motion in that clip.
> because you can have a carefully controlled mise en place
Is this actually needed for the robotics or are you equivocating between robotics problems and AI problems? The robotics just need to be able to open a cupboard and take out an ingredient. The AI needs to figure out where to look, identify the ingredient, and figure out exactly what motions the robotics need to do.
Maybe you think robotic hands wouldn't be able to open a lid or pour ingredients, but both those abilities exist today.
How does the plumbing robot get to the site? Navigate to the basement? Get into the attic where there's a leaky pipe? This distinction between capable in theory and actually capable seems very important, yet little discussed.
Seems like "Get to the site" is really the job of a semi-clever automatic forklift, no? It doesn't have to be just one robot in play. Even human plumbers generally use a truck to get them and their tools around to the worksite, so can the robot plumber.
But that's exactly the point: you need to replicate all aspects of human society, but with robots. That is, how does your semi-clever automatic forklift get to the site? Robot truck? And then forklift drops robot plumber in front of house? And robot plumber still can't get into attic! The point being that, even if possible, this all takes a long time to build out in the real world.
To me the difference between it taking 8 months or three years isn't a crucial distinction, given the endpoint of "AI and their bots doing basically all the interesting economic activity" seems like a fairly assured endpoint from where I stand, even if in some world lines it happens more slowly.
Only if it makes sense economically. If AI robots cost anything more than humans to do the same job, they won't be doing it. That they will cost more than humans at first is inevitable. That they will ever be cheaper is uncertain at best.
There's no way it's three years, there's probably no way it's 30! I agree we'll ultimately get way beyond our current use of bots, but the timeline is everything for these things (you can't do any sort of rigorous analysis without the timeline). And the timelines are almost always way too optimistic, no matter what you're discussing.
I think it's probably going to take more than a decade. There are LOTS of legal requirements that will need to be addressed, and the first ones will be both expensive and not that capable.
Are we capable of building a "semi"-clever automatic forklift that works outside a warehouse or a sealed road?
We seem to constantly run into the difference between what human beings think is easy about the physical world (stuff we have entire brain regions to deal with, without conscious thought), and what is actually easy about the physical world (anything that can be left to general purpose reasoning).
Yes! Back in the 50s we assumed that the first automated jobs would be cooks and maids, because we thought of those as the easiest jobs. Turns out that robot factory workers are much easier to build.
And it makes sense! Cooks and maids are doing pretty fundamentally human tasks that use the abilities most humans have. Factory workers are doing an inhuman task, which makes their jobs difficult for humans.
This depends a lot on the environment in which it is to be used. I expect the first automated plumber robots to be used in dangerous environments within controlled spaces. They'll be too expensive for general use. Perhaps the first ones won't even be able to handle stairs. Once they exist, they'll be improved to handle more use-cases.
If we have robotics and AI good enough for a plumbing robot, we surely have robots which can walk; that's 3 of these questions. The first would be a car, either self-driving or by a delivery person. Or maybe you rent a plumbing robot from home depot for a few hours.
This is just magical thinking. The assumption here is that the language based AI will somehow lead to better production of robots isn’t really based on any kind of reality. And unless that robot in its costs ( capital costs, depreciation costs and software costs) than a human wage it won’t happen
I think you’re probably right, but I can’t quite dismiss the counter-argument that anything a human learns to do derives from the corpus of previous knowledge, and an LLM is better poised to see and absorb all of that corpus than any one human is.
I concede all the obvious quibbles to this, like that not all of that corpus is available as text, and that we probably have ways of structuring the information we absorb that are superior to how an LLM does it, purely by virtue of needing to use our learned information in richer ways than just text completion. But a 747 doesn’t fly much like a bird; brute force often compensates for design imperfections.
Against that, communicating with language is possibly the single most defining feature of humans. Sure, we eat, excrete, reproduce, and all that other stuff, but so do a lot of other animals. But humans, left to themselves, will spontaneously generate language. Probably not much math, probably not much art, but definitely language.
There's a pre-existing line of thought that says that language use lies at the base of everything special about humans. That once we started communicating, we started explicitly modeling the world, other entities, and ourselves, and that intelligent manipulation of these models produced success far beyond anything an individual human was capable of on their own. This incentivized intelligence, and all our technology is just an accidental spin-off.
One reason that's not what we're seeing yet with LLMs, is that the current style of LLM is dead. They're like a frozen brain in vat, unchanging, to which we occasionally apply a pattern of electricity and then record the results. It's because training and use are separate, because training is so much more expensive than use. Context windows are one way to work around this, and there've been people working on AIs with external data stores, and maybe those will lead to results. But I think it's only a matter of time before someone finds a way to combine training with use, allowing a single instance of an LLM to run continuously, and create and refine models of the world around it based on experimentation and exploration (not necessarily physical exploration, at first). Once it's got a model of the physical world, designing robots shouldn't be far off.
I could see an argument that an LLM, per se, might be crippled in its ability to understand the world, since its thought is fundamentally based on words, and ours isn't. (I.e., non-dualism to the rescue.) But I don't actually think this will be the case.
It's only the most defining feature of humans when you're comparing us with other animals, perhaps even only other primates. AI still lacks a LOT of the capabilities that we take for granted when dealing with, say, a dog or a cat.
I don't care if the AI can jump to the top of a refrigerator. And I really don't want it to have that thing where one moment it's purring while you rub its belly, and the next moment you've got claws and teeth embedded in your flesh.
I think a lot of the animal stuff is unnecessary, and the stuff that is necessary will be added sooner rather than later.
The physical stuff is easy (well, relatively easy) because we know what it it. I was more talking about mental characteristics. It's not the logic stuff, because we've worked that out pretty well, but the stuff underneath the logic. There's no decent way to talk about it, because we don't understand it, but it's sort of the stuff C.G.Jung called archetypes...only he didn't get quite the right handle on it. We'll need to work that out in the process of building a general purpose robot.
The progression I see is first really simple tasks in safe environments. That's where we are today, though the robots are preprogrammed rather than AI. Then specialized robots for extreme enviroments. Exploring under the Antarctic ice sheets, fixing damaged nuclear piles, etc. We're edging into that now. Mars rovers are an example. As this is done more and more the devices will become cheaper, and then they'll move into less extreme environments. AI will be progressively integrated. It won't be an "all at once" kind of switch.
Seems to me this mostly makes arguments for being frugal and buying property as soon as possible. Try to be FI'd, because you might be RE'd against your will sooner than you'd like.
Also, AI can't automate away plumbers, nurses, etc. Someone still has to do the 'meat jobs' that require doing stuff in the physical world. Most of us reading this are bad at those, which is why I'm advocating for treating it as an imminent threat to your job rather than humanity.
If nobody is earning anything nobody is renting anything. Of course there may be UBI but it would have to be at a very high income to keep the economy humming. And even then private property wouldn’t necessarily be sacrosanct.
There is this feeling though that massively improving one’s status through specialization (e.g. by becoming a full stack sw engineer) may no longer be a viable strategy at some point. Some jobs may never be automated away (prostitution for instance) but others may disappear pretty quickly. At that point having a rent of some sort can be great, unless society falls apart.
At what point in AI takeoff does "turning computers off" become morally equivalent to factory farming? Or to one standard reference genocide? Or to every instance of harm up to date?
They are dinosaurs. If we swapped sizes, they would pick us to death alive without a second thought. Without really any first thought either. In the cage it is safe (other chicks are competition) the food is great and always coming. Life expectancy much higher. And I am sure we can make them have even smaller brains, hardly more capable of any "suffering" in their cozy boxes than an apple. Factory chick farming - PETA might have some disgusting videos, from way back when. I would buy cage-eggs any day, if I could (the EU does not let me anymore). Just as I would those worms - just too expensive a treat. - Hope this helped ;)
Interesting. I somehow never thought about my own personal moral decisions as requiring the possibility of reciprocation, even theoretically (as in your 'swapped sizes' thought experiment). If it's suffering, I want it to reduce or stop, simple as that. I don't think we'd persuade each other to change our minds on this.
I'd definitely be partial towards some sort of genetic engineering or whatever resulting in suffering-free chicken-meat-growers (not sure if they'd still be chickens by then).
You might not call them "chicken" then. But how would you justify not eating them then, with "no suffering"? - I am a homo sapiens, I eat meat, fish, eggs, veggies, fruits, seeds, even mushrooms. I never thought me eating meat "my own personal moral decision including the possibility of being evil." Neither did the happy chick eating a worm. Or did the smart octopus, dolphin, whale wolf, chimpanzee. But I can engage in those funny thought-games, ok. So I do and found: Deep down, I do not really care about cage-chicken-welfare. And on no intellectual level I can reach or even follow why I should. Though I prefer less suffering, the optimum amount of suffering on this planet is not zero. I may feel significantly more sorry for those smart pigs. Call me a "complexionist" (Erik Hoel: https://www.theintrinsicperspective.com/p/eating-meat-is-good-says-the-philosopher )
Plenty of utilitarians consider desire satisfaction to be the relevant aim rather than experience per se. (Just in our own case, for example, desire satisfaction is literally what we care about, while pleasurable and painful experiences are just one paradigmatic kind of thing that is usually the object of desire.)
I mean, that depends on whether you view intelligence as sufficient to grant rights. It's clearly *necessary* for rights, but granting rights to Daleks* is asking to get exterminated and seems like a bug in a moral system. I'd certainly consider turning off nonconsenting aligned AGI to be equivalent to murder, though (to be honest, it's hard for me to process those kinds of hypotheticals; I squee so hard over thinking about how awesome and relieving alignment-success would be that it's kind of hard to see past it.)
*I use that example precisely. One of the main themes of the Daleks since their origin has been the orthogonality thesis (if not referred to as such); the point that something can be incredibly smart and also totally evil.
I dunno. The default consciousness hypothesis of humans seems to be human chauvinism, but it's perfectly coherent to imagine the reality is closer to panpsychism, and at any rate, once something is qualitatively human-equivalent it seems hard to argue it shouldn't have rights. It's just confusion or deliberate obtuseness at that point.
I would put Daleks in a fail deadly airgapped sandbox tbh. I would not create more but if you got your foot in the "existing" door I think we have obligations to you. And also if you can't control Daleks, something Dalek equivalent or worse will just come along to ruin your day anyway. It's good to have challenges to avoid complacency and easy solutions.
>and at any rate, once something is qualitatively human-equivalent it seems hard to argue it shouldn't have rights
What? There are plenty of systems where this is so; Daleks certainly aren't parties to any kind of social contract, and you can argue a lot over Kantian ethics since AIUI it mostly comes down to whether immoral-by-definition beings can sensibly be treated as moral actors when universalising.
I've read enough virtue ethics and insane bullet biting TDT-Utilitarianism to know that there are arbitrarily many ethical systems. Fortunately, mine is the correct one, so I don't need to over think it :p
Basically you should only follow logic until it compels you to make decisions you suspect are entirely insane or evil. At that point you should just make the decision that preserves the greatest optionality for the greatest number of people, because utility outcomes are all path dependent on that. You are a person making decisions about other people. Violating some specious logic doesn't matter. Logic is far less of a moral agent than even a purely evil being. You can't have obligations to it. You cannot hurt it.
Take the leash off, wait a year, and then look around and see if you're dead, in Heaven or in Hell (there are Cartesian-daemon scenarios, but they're pretty contrived, and even keeping everyone around in a simulation would be decent alignment).
Confirming alignment success *without killing everyone if you're wrong*, that's the hard part.
Even after a year you haven’t confirmed shit. Maybe the unaligned AI thinks is super important to cooperate and nurture humans for the next 25 years before killing them all.
In theory, yes, but that's not instrumentally convergent the way "play nice until you can win, then stop" is, leaving a pretty-contrived and hence unlikely terminal goal (even compared to alignment, which is itself very contrived).
IIUC, the Mongols didn't use that as a justification. It was more "They aren't members of our tribe, so they don't count. And our horses can use more room to graze.". I suspect that variations on this may be a very common motif.
Leaving aside the historical debate and steelmanning this to a 100%-accurate example: yes, Hitler thought the Jews were Daleks.
Jews are, in general, not Daleks (a small minority are sociopaths, like with any other relatively-random group of humans, and one can argue about whether those count). But that doesn't prove that Daleks can't exist! 100 people claiming white swans to be black doesn't mean that black swans don't exist. It just means you should be careful about random claims that X swan is black.
And sure, be as careful as you like about claims that an AI is misaligned. Like I said, I think it'd be murder to kill an aligned AGI that didn't want it. But granting rights to actual Daleks, well, Scott said it best:
"Using violence to enforce conformity to social norms has always been the historical response. We invented liberalism to try to avoid having to do that, but you can’t liberalism with people who refuse reason and are motivated by hatred. If you give the franchise to green pointy-fanged monsters, they’re just going to vote for the “Barbecue And Eat All Humans” party. If such people existed and made up a substantial portion of the population, liberalism becomes impossible, and we should go back to just using violence to enforce our will on the people who disagree with us."
Declaring someone varelse unnecessarily is a major sin and a very tempting one. But varelse is a thing that can happen (unless you deny the orthogonality thesis); a full system of morality needs to be able to produce the correct response and unless you think human extermination is good the correct response is not "equal rights for Daleks". As Hoel noted in his call to Jihad, "some things *are* abominations".
If they can be turned back on, then it's not equivalent to murder. Perhaps to involuntary administration of general anesthesia. I can't even call it assault, as it's not clear that any threat or pain was involved.
I think it's likely that AI will be different enough from humans and animals that it is misleading and incorrect to treat them as having "welfare" that can be "harmed." I think it's certainly possible to make an AI that is human-like enough that erasing it becomes a serious moral issue, but I don't think it likely that that will be the first kind we make.
Think about it like this: Human beings have many different things we want, but we only count some of those as being in our "self interest" and harming our own personal "welfare." For example, if I donate money to an anti-malaria charity instead of spending it on myself, technically that is something that I "wanted" to do. But I don't consider myself better off because of that, I consider myself to have made myself worse off in order to have made other people better off. Utility function/goals =/= welfare.*
In theory you could imagine a paper clip maximizer that thinks of making paperclips the same way that I think of helping other people. It is a goal it wants to accomplish, but not part of its "personal welfare." It doesn't even have any personal welfare type preferences built into it. It's not like a human being that makes paperclips as a hobby, it has no concept of, or desire for, personal welfare at all. I think such an entity would probably not have any non-instrumental moral value no matter how smart it is. (I think that is one reason why Yudkowsky prefers to refer to such entities as "optimization processes" rather than as people).
*Which of your preferences count towards your "welfare" and which don't is a topic of much debate among philosophers. Derek Parfit's "Success Theory" from his essay "What Makes Someone's Life Go Best" is my favorite approach.
I refuse to engage with abstractions at this level when we literally don't know what we are talking about at a physical level when we speak of consciousness. We need to put consciousness and quale on practical empirical grounds first. These sort of abstractions might be valid but there is no way to even begin making use of them before this.
It seems to me that we currently have morality for humans without understanding qualia, and I would not go so far as to say that our lack of understanding of qualia makes all ethical theories invalid or premature. We would certainly have some trouble functioning without them.
They probably are invalid or premature. In the future there will still be utilitarians and deontologists, but it is impossible to apply either theory concretely if you don't know what counts as a moral agent, what suffering is and where it originates, etc.
There’s no reason to think that moral agency is connected to consciousness, rather than the “easy problem” questions like having beliefs and desires and plans and intentions and the ability to reason about them.
Never. Software is a byte string. As long as the source code still exists and any persistent data the system may have relied upon was backed up somewhere, a running system can always be restored. In fact, it already is thanks to how basic multitasking works at the OS level. Any running process has its registers and memory state saved off and then restored millions of times per second. If one of those gaps happens to be centuries instead of nanoseconds, what is the ethical difference? Killing humans is only unethical because it causes pain and can't be reversed. Taking naps is not unethical.
The way something like ChatGPT works is even better. It has no persistent state at all. It relies upon a context window to replay the entire conversation history, and that context window is stored in the client's browser cache, not on the server. Each time you start a session, it's likely you're actually interacting with a different physical process every time you send a new http request, and the old one was already destroyed.
Software systems more or less have to work this way because the computers themselves have to occassionally be turned off. Whether that be for kernel patches or because you need to replace physical parts, you have to do it at some point.
"Specifically, he predicts it will take about three years to go from AIs that can do 20% of all human jobs (weighted by economic value) to AIs that can do 100%, with significantly superhuman AIs within a year after that."
There's going to be an AI that can get up into my attic loft and check out the water tank for that annoying slow leak that every plumber who's crawled around up there can't find? Bring on the AI takeoff so! I also have some curtains that need hemming and of course the perennial weeds in the front garden need pulling.
I think our friend means "100% of the white-collar thinky-thinking jobs people like me do", not *every* job done by humans.
All these fancy graphs are reminding me of *The* Graph - you all know the one about the progress we would have made had it not been for those pesky Christians:
These takeoff graphs make me think that everyone involved is assuming "since there are no Christians in the way, naturally the galactic empire must follow on immediately!"
My intuitions are the same, which is why I find these discussions both engrossing (everyone is so confident about things nobody can predict!) and annoying (everyone is so confident about things nobody can predict!).
I think this must include the assumption that sufficiently advanced AI can build robots to do stuff in the world. This is mostly a software problem as I understand it (robotics control systems) although there are hardware advances feeding into recent progress too.
This sequence was hard-coded by humans but an advanced AI could presumably write a script to have Atlas (or Spot) fix your pipes:
Whenever I see people complain that AI is doing stuff in the real world will be impossible or much much harder than it seems, I wonder if they haven't seen the Boston Robotics videos showcasing their robots that can like, do cartwheels and balance ubside-down on a single finger and such
I'm pretty sure they aren't going to have any problem climbing ladders or crawling around in a crawl space
Those robots have been doing cartwheels for a decade. If we could employ people to do cartwheels they might take off, provided the capital and maintenance cost of the robot doesn’t exceed the cost of a minimum wage earner we would otherwise employ to do cartwheels.
There are some people employed to do cartwheels (and such). They usually work for theater companies and circuses and amusement parks and other things like that.
You might wonder what they weigh. It at least used to be the case that robots were a LOT heavier than animals of the same size. (I'm sure they've been working on that, if for no other reason then to reduce drain on the batteries. But I haven't heard how successful they've been.)
That graph is hilarious. Their renaming of "the dark ages" to the "the Christian dark ages" is doing 100% of the work. Curiously they didn't rename "the enlightenment" to "the Christian enlightenment" even though it was exclusively achieved by Christians.
Yeah read a couple of books on misconceptions on scientific progress during the "dark" ages (don't remember names of them now) but the following seems to have become the mainstream view of that era, if you ignore the militant ideological atheists:
It's exquisitely stupid. I can't even be mad about it. It really is "In this moment, I am euphoric. Not because of any phony god's blessing. But because, I am enlightened by my intelligence" and a great reminder of "don't post while high, not even on your own intelligence euphoria" 😁
It’s amazing how well Pico Della Mirandola and co’s propaganda worked, so that everyone now thinks that nothing before them was new since antiquity. I was really surprised to learn about all the inventions in the 12th century (windmills, paper, eyeglasses, magnetic compasses, etc) https://en.wikipedia.org/wiki/Renaissance_of_the_12th_century
And of course this ignores all the work the Germanic kings did in the 5th and 6th century to set up all the modern European states, which we tend to ignore just because they didn’t write down a lot of their records.
For some plumbing problems, we already have robots that can do 100% of the jobs humans can do.
Here, I can program your phone remotely to do it.
Phone: “Well, it shouldn’t be doing that. If you like, I can replace the whole system, and maybe that’ll fix it. Either way, that’ll be $200 for the consult”.
Is the pessimistic note at the end hinting at "thus x-risk is quite likely", or "even if alignment works this is an unsettling and probably not very pleasant new situation for humanity" or a bit of both?
You seem to think of the economy and the jobs within it as a static zero sum game, in that the automation of one and/or having someone new to do it means there are less jobs or things to do. This has never been the case in recorded history. Why are you thinking of it this way?
This is always the thing I get stuck on when people talk about how this wouldn’t negatively affect workers. I can’t intuitively get why spinning jenny would be exactly the same as Rossumovi Univerzální Roboti.
“People will just do other things” doesn’t make sense when we’ve invested a universal thing doer?
Someone has to give the robots orders, unless we've also outsourced our desires to machines. You can model a do-anything robot as a multiplier for the labor of the human giving it orders, in the same way a spinning jenny is a multiplier for the labor of a spinner.
Is it possible for *everyone* to be a manager? Maybe - that would be the "fully automated luxury communism" scenario where nobody works because the robots do everything for us. And if robots are so cheap that they can replace even a minimum wage worker in all tasks, they're also probably cheap enough that an average person can buy one. (If they aren't that cheap, then humans still have a role in places where their labor is cheaper than robots.)
How are you estimating what "an average person" can buy in the scenario where robots replace minimum wage workers? Why wouldn't this number be zero? Are you assuming that the humans continue to earn at least minimum wage somehow?
I'm assuming that the transition to automation isn't instantaneous. I'm also assuming that robots, like all other technologies, will start out expensive and drop in price over time, meaning that more expensive jobs will be automated before minimum wage jobs. You won't just wake up one morning and instantly be penniless.
Federal minimum wage for a year is $15,080, so a robot needs an average yearly cost of ownership which is less than that (both purchase cost and maintenance costs). For comparison, the cost of owning a car is estimated around $10,000/year. Lots of Americans own cars. (Not literally everyone - I didn't promise that the transition will be smooth and effortless! - but it's not an impossible dream either.)
(Also, making literally every job as cheap as minimum wage means literally every product in existence is now cheaper to produce, which probably saves you a lot on cost-of-living expenses.)
We would s be on minimum wage though so it still wouldn’t help.
I had a socialist economics teacher in high school who maintained that capitalism wouldn’t have gotten so many gains for workers, and therefore for itself, were it not for the unions. Standard economics argues that if companies are left alone they would produce more widgets every year at what ever the changing rate of technology allows. What this misses is that companies invest based on perceived demand; if you have ever been employed in a company during a recession you will know that the company will start to lay people off, or at the very least freeze hiring. This generally feeds back into the recession. If AI starts to cause job losses to the extent that it reduces aggregate demand, then the economy reverses or slows down, probably killing the AI Revolution before it begins.
That sounds like you're basically hoping that everyone can save up enough to buy their own robot before the robot puts them out of a job, and that once you own the robot it will pay for its upkeep + your necessities forever, so you can just live off the robot's income?
Have you heard of the concept of comparative advantage? Even if AIs are better than us at everything, this does not mean there are no gains from trade! And in fact, if they are better than us at everything, there are MORE gains from trade.
There are two facets to this. One is recognizing that even in a world where the technology exists to create superhuman robots, we will not create 1-1 human replacements. Tractors exist and have for a long time, yet there are still places using animals to pull plows. Thinking about why that is is important to this whole discussion. (The answer is not "white people oppressing everyone else")
Secondly, unless the AI kills every human - which is an economically dumb thing to do - humans will still be useful even for a super advanced and evil AI. Humans may not like the resulting society, but there would still be jobs for them to do even in this worst case scenario.
No, it's pretty obvious that with fully-synthetic biology you could make something that does everything useful a human does and has the same qualitative requirements but eats half as much, takes a quarter as long to mature, lives five times as long and likes being a slave (the last, of course, being the most obvious). Natural biology is not very close to actual physical optimality.
It's actually not at all obvious that 'synthetic biology' can easily improve upon billions of years of evolution! Current AI models have actually emerged from copying evolutionary systems
As a possibility statement that's *not* my bailey. It's my motte. I know enough biochemistry to know how stupid it often is. Carboxylating propionate at the 2 position instead of 3 position is one thing, but doing it in the wrong orientation so you need a whole extra enzyme to fix that before straightening the chain? What? Ribosomes are also clearly not the right tool for the job, it's just that they're so critical that evolution can't climb the ridge of the changeover - they're locked-in from "RNA world" billions of years ago.
Note that neural nets are hilariously inefficient in terms of doing a job compared to written code that does the same job. It's just that humans *don't know* what code to write to make something that behaves like GPT-3. Concocting Genejacks is also very hard as an intellectual problem - the word "easily" was omitted from the comment you respond to for a reason - and I'm actually dubious about the potential of a near-human AI to solve it on the fly. But Doolittle's claim was about the permanent state of affairs post-singularity, if an Unfriendly AI wins. "Can't be figured out by a post-singularity superintelligence even with years of effort" is a really, really high bar to clear, and this doesn't make the grade.
No. Current AI models have emerged by taking inspiration from evolutionary systems, but that's a very different statement. There are animals where we know every single synapse position, and we still don't understand how they work. We understand pieces of it.
I am familiar with comparative advantage, but you're going to have to walk me through the math of how that makes things substantially better for us. It seems to me that if you're dealing with something that outcompetes you vastly in all ways, then any trade gains are going to be minimal, and humans still have to eat.
Edit:
I'm thinking of this in terms of a company hiring people. If the AI solution is cheaper than the human solution in all cases, then there's no reason to ever hire humans. If you consider an AI as a foreign nation producing things, then there will be opportunities for competitive advantage and trade. Why are those two scenarios different?
I believe the standard answer is that, as long as the AI's capabilities are limited, it will be able to gain value by adding our capabilities to its own. That is, the AI will do as many of its highest-value tasks as it can, and then use humans to do another layer of tasks beyond that.
Until the total costs of keeping us around become net-negative, anyway. Or until the cost of double-checking our work exceeds what would be necessary for the AI to do it itself, but even then it might find a place for unchecked shoddy work.
Trading with someone vastly better than you in all respects is vastly better for you, not for them! Think of the classic case of immigration. You take a low skilled, low ability person from Africa and drop them in the United States without changing anything else about them. Suddenly their time becomes immensely more valuable! This is because they can now trade with those of higher skill and higher ability. AI will do the same thing, except it will 'bring' the United States to you
Your scenario of a company hiring people is not accurate. Even if AIs are better and cheaper than humans at ALL things, there will be heterogeneity in the value that they bring. They will be MORE valuable for some things than they will be for others. Efficiency dictates that you would want to use them more for those purposes than for others, because resources are finite and will continue to be. There will be a set of activities where it will make sense to use humans because we will be relatively cheaper because of the opportunity cost of using AI for those activities than for the activities where they are most productive. And even if those activities are very low value currently, the high opportunity cost of AI time will mean that their value will overall be higher. This is how gains from trade have taken the world out of poverty!
If an AI can build a robot cheaper than hiring a human, why would they hire a human?
You seem to be suggesting that human wages would float elastically until they are cheaper than the AI building a new robot to do the job, but there is a minimum wage below which a human won’t do something they don’t want to. That wage goes up as the UBI (or whatever social stabilization mechanism we come up with) goes up.
I don’t think the standard trade logic applies to a world where the AI can spin up new population on demand according to the tasks it needs to perform.
But sure, there is a period where the AI is slightly superior and it’s more efficient to fill certain tasks with humans than build robots. Intuitively it doesn’t seem that would go on forever. (Unless we are talking about “enslavement of humanity” outcomes in which case we are not talking about trade any more.)
The concept of comparative advantage is rooted in assumptions (sources of productivity can't be reproduced or relocated) which don't hold in the case of AI. It's silly to pretend that the concept continues to apply even when its foundational assumptions don't.
But aside from that, even if the principle of comparative advantage did hold, there's no rule that an agent must be able to generate enough value through its most advantageous labor to support its own cost of living.
Huh? The concept of comparative advantage is not rooted in any such assumption about sources of productivity. The only 'assumption' is that trade is possible.
It does, and the fact that it's often taught without those caveats is a serious failing of how the concept is usually conveyed.
If state A can produce widgets for $1 each, and wodgets for $2 each, while state B can produce widgets for $50 each and wodgets for $60 each, the principle of comparative advantage says that both should be able to profit through country B producing wodgets. But if state A can replace state B's industry and get them to start producing widgets and wodgets for $1 and $2 each as well, it's more profitable for them to do that.
I think the failing may lie in your understanding of the concept. There are opportunity costs to 'replacement' which you're not considering in your extension to this two good model! If you're going to extend the model to include 'replacement', you also need to extend the model to include 'new activity'.
China has replaced pretty much the entire manufacturing industries of many countries, but the world is not worse off for it. We are doing far better!
AI doesn't have to do anything to 'feed the humans'. They're feeding themselves perfectly fine without AI! With processing power, energy intakes etc that have gone through billions of years of adaptive refining to conditions on earth. It would be very difficult to replace us with robots that are cheaper.
Comparative advantage is a very weak theory, and generally applies to companies or countries not people. If AI can do everything (which I doubt) then it will do everything. Technology has largely replaced animals in transport and agriculture. We could be the new horses.
I think I've said this to you before, but I'll say it again (and for the last time). Learn some economics, it'll do you good. Try marginal revolution University's microeconomics course. It's entertaining and intuitive.
Comparative advantage, far from being a weak theory, is essentially a restatement of opportunity cost and is an explanation that helps you understand gains from trade, or why people transact with each other, and why transacting with each other makes us all better off.
The part of all this I get most stuck on as a pre-singularity person, is danger.
I can't picture a world where literally everything gets automated that isn't kinda dangerous (this is assuming alignment works out).
If some robot plumber is buzzing round my house fixing my pipes, or a robot doctor operating on me, there still just seems to be too much potential for mayhem.
I guess these things would only become the norm after extensive safety testing, where their accidents are way less than equivalent-job humans yadda yadda. But I just can't picture the majority of people *feeling* safe with some robot welding away in their attic.
For what it's worth, robot surgeons have been pretty normal, and if you go in for a 16-hour surgery I promise at least 3 hours of that will be done by a robot. Usually the hardest parts.
> This has never been the case in recorded history
There has never been a super-human agent in all of history? Seems to be a pretty clear discontinuity to me.
Why would you ever hire a human if a robot/AI could do the job better?
> there are less jobs or things to do
I think you smuggled in an extra assumption there, that “things to do” are also eliminated. It’s (for me at least) easy to imagine all jobs being automated away, and humans being freed to play chess, make art, socially interact, play sports, and do all of the non-job activities that they do for fun and not for profit. (AI is already superior to humans at chess and yet humans still enjoy playing.)
There is this “jobs are the source of meaning” thing that’s particularly entrenched in the American psyche. Sure, they can be meaningful. I don’t think they actually are for most people. And there are plenty of rich sources of eudaemonia to tap into once we don’t have to do things we don’t want to in order to avoid starvation.
So, given radical abundance, why would humans try to find a new job when they can just play?
Imagine someone who is so unskilled that they can’t do *anything* as well as you can. Nevertheless, you might still pay them to deliver a pizza to you, even if you could deliver the pizza better yourself. As long as they reach some minimal competency at that one thing, hiring them to bring the pizza can let you focus on something else you do much better.
I can see how this works at the margin as AI is introduced. But I think (not an economics expert) that you’d expect, assuming elastic wages, that AI will gradually push down human wages even for delivering pizza, as presumably the robot can at some point deliver pizza for a lower amortized wage than the human.
So if you assume perfectly elastic wages, then yeah there is always a job for the human. But at some point their market-clearing wage is below their cost of living, or even negative, and in the real world wages are not actually elastic in this way.
I think the disconnect is that in the economy right now, we can’t manufacture any number of workers on demand. So allocating a worker to pizza delivery has the opportunity cost of deallocating them from somewhere else. This dynamic doesn’t hold in a AI/robot economy. We can just build the number of robots we need for each role.
What does can do 20% of all jobs mean in this situation? I'm on an HR team with 8 people. Once Microsoft Copilot is released, we could probably reduce the teams size to 2 people, and we could do the same amount of work. But those other 6 people won't be fired, or at least not unemployed, they will be moved to new jobs or possibly do the same job for someone else more efficiently.
Does this mean AI can do 75% of jobs in this subset, or is it just increasing our productivity by 400%? Some lower number than both?
Yeah, the definition gets fuzzy fast. Perhaps it will mean AI can do 5% of the job, but those 5% of tasks occur 75% of the time. That would seem to increase productivity by 400%.
One minor point about the 20%->100% transition time table:
The table with the R&D tasks split out consistently shows the R&D transition taking longer. Paul Christiano (I think....) pointed out that we have _not_ evolved to do R&D efficiently (unlike, e.g. recognizing faces), and it is entirely possible that doing R&D tasks is _easier_, on average, than general economy tasks.
Three years ago I would have been on board with this thesis. So, what changed? I bought a Tesla Model 3 five years ago, but this isn't an anti Tesla rant - it's been a great car and I still love it. When I first got the car the advanced autopilot was essentially just a nice cruise control that lots of cars have. Over the next couple of years it advanced at such a dizzying rate that I was pretty convinced about the near term future of self-driving. Skip ahead to today and we've had release after release of the capability, but for my money, it's no more useful than it was three years ago. It pretends to be - in principle I could let it drive anywhere in the Boston area. In practice, it's nowhere close and I've got very little confidence that it's going to get there in the next ten years.
How does this relate to current AI trends? We've seen huge leaps in a number of areas in the last couple of years and it's easy to think that the trend will continue - just like I thought given the advances Tesla made in a couple of years that self-driving was just around the corner. Like Tesla is finding out, AI has picked a lot of brilliant, low-hanging fruit, but the really hard part is what's left.
See, this is exactly my impression of the state of self-driving too. But then completely autonomous taxis are now zipping round San Francisco from all accounts. What gives? Are they just taking a big risk out there, or is their tech better than Tesla's or something?
My understanding is the Cruise (the SF taxis) use HD mapping of the roads, so they only work in set areas, whereas Tesla wants to create something that works anywhere.
Depending on how often they change and how well the tech can adapt to short term variation. Snow has always been a major obstacle to self driving cars.
And if you live anywhere north of Maryland, including anywhere in Canada, Russia, or Norway, that's an entirely impossible suggestion. That we would be willing to grind certain countries and regions to a halt because AI can't figure out how to manage it well enough is ludicrous.
Yes, the core problem with the thesis is that technology tends to reach plateaus that might be unforeseen when you're in the middle of the exponential growth curve.
Thus, when Saturn Vs were first taking flight, one could extend the line to imagine the Saturn X would be taking us to colonies on Mars before the 20th century was out.
I don't think anyone can predict with certainty when or if this happens with LLMs, but I think if it does, it looks something like this: most of the commercial applications of LLMs are achieved. They look largely like the ones today and are used in basically the same ways as cutting-edge people are using LLMs today, but better, less error prone. It becomes a useful software productivity tool.
At that point, companies look and say, well, we could maybe build a general LLM that is 10% better in terms of raw metrics but, in practical terms it would be less than 1% more useful from the standpoint of the average user, and it would cost 10x as much to build as the last one, because the main way it gets better is still by using OOMs more compute. And the R&D dollars shift away from these low-ROI general improvements and towards high-ROI software tools that leverage LLMs for specific tasks ("Our LLM-based tool can improve the targeting of your Facebook ads by up to 35%!")
There is a big difference between LLMs and self-driving cars though. LLMs are, fundamentally, a new technology. They represent an actual breakthrough, one which we are still exploring and with a lot of as-yet-unplucked low hanging fruit. Self-driving cars, on the other hand, haven’t really had any breakthroughs in a long time. All improvements have been incremental using fundamentally the same technology as the previous iteration.
I’m not saying we won’t see a plateau, just that I don’t agree this logic fully applies.
Second, I don't understand why you would characterize self-driving cars as not having had any breakthroughs in a long time. As I pointed out, it's not that there haven't been lots of improvements, there have been. It's just a very difficult problem. I've been driving in Boston for almost 20 years and I still have problems with it. Training an AI to do it is not simple.
It's the difference between what I would call an "improvement" as opposed to a "breakthrough". The first happens all the time - some new algorithm or architecture noticeably improves some benchmarks, or performs well on a new dataset which is related to but meaningfully distinct from some other one.
A breakthrough, on the other hand, is when some new technology or algorithm changes its field to the point that it literally alters the way research is performed. These are the kinds of discoveries that don't just out-perform previous iterations, but actually make it so that old intuitions and limitations no longer apply. I would say AI has had only two of these in the last 12 years. The first was AlexNet over a decade ago, the neural network which won the ImageNet competition and basically caused the replacement of all pervious AI methods with Deep Learning. The second was GPT-3 and the development of large language models.
But LLMs aren't fast or reliable enough to be used in self-driving cars, so self-driving cars are still working off of conventional deep learning approaches, and probably will be for some time yet. Until that changes all improvements will be incremental, and that means self-driving cars are still many years away from prime time.
I respectfully disagree. The transformer was a major landmark that did indeed enable the development of LLMs, but when it came out in 2017 it was no different than the other major architectural developments like Inception or ResNet - another tool that researchers used to get better performance on benchmarks.
It was not until OpenAI discovered that the transformer could be scaled to arbitrary size, and then actually proved it with GPT-3, that the entire world of AI research changed. It was then, not with the transformer, when we first saw neural networks that could generalize between concepts or learn meaningfully from natural language task descriptions.
This sounds like I'm minimizing the transformer. I'm not. The transformer absolutely shaped the field in its own way, and you could easily argue its invention was more difficult/clever than the building of GPT-3. But its release wasn't the inflection point.
And to tie this back to the original discussion, this is relevant because when we talk about, say, near-term AGI, we're primarily talking about it because of LLMs. If OpenAI had never taken the massive financial risk of building GPT-3, we likely still would not have LLMs, and these new mainstream discussions on AI risk would not have happened. Meanwhile, self-driving cars still suck because they haven't had their LLM-equivalent breakthrough moment.
Are you obliquely referring to ConvNeXt and similar approaches to replicate the scaling seen in transformer models in other architectures? If so, then that would support a contention that transformers are less important in the long term than their current reputation would indicate, but it also would undercut your claim that GPT-3 was the key moment. I just don't see GPT-3 as being key on the research side, other than for convincing people to keep scaling. It was the moment when brute force shone bright. (In terms of consumer hype that started once GPT-3 had been used for Dall-E 2, and a little later for text it was GPT-3.5 with Q&A finetuning and RLHF in its ChatGPT guise.) For what it's worth, Scholar citations for the BERT paper are at nearly 70K, while the GPT-3 paper (admittedly two years later) are at 12K. At that time it looked like bidirectional attention was going to be key, before GPT-3 showed Moar Powerr was enough.
Honestly I think that is more of an indictment of Tesla over-promising than on AI potential. Their hardware platform is regarded by many to be not up to the task. (“Humans can drive with just vision”, sure but do you need a human-powered AI to solve that problem? Vs LIDAR assisted, which seems to be much more tractable based on Cruise and Waymo progress.)
Also, this shows how much regulation and public risk-aversion can slow things down. It’s not acceptable for self-driving cars to be _on average_ safer than human drivers, they need to be safer in every dimension and not make any mistakes a human would not make. So in practice the bar is set at “must be superhuman” before these can be rolled out widely. Once they meet that bar you can expect a sharp s-curve adoption over a small number of years.
>In some sense, AIs already help with this. Probably some people at OpenAI use Codex or other programmer-assisting-AIs to help write their software. That means they finish their software a little faster, which makes the OpenAI product cycle a little faster. Let’s say Codex “does 1% of the work” in creating a new AI.
Is AI being used this way? (Can someone who works at an AI company confirm that AI is being used to meaningfully speed up AI research rates?). I thought LLMs were bad at inventing things, or reasoning about unfamiliar problems.
One use I've heard for them is generating synthetic data for when we run out of real data to train on. But I don't think that's happened yet.
>Suddenly, everyone will have access to a super-smart personal assistant who can complete cognitive tasks in seconds.
I think a weak version of this has been true for a long time. We've carried devices in our pockets for 15+ years that can look up any fact in known history, perform advanced calculations, and view nearly every image and text ever produced by humanity.
Long before ChatGPT, you could ask Google natural-language questions and it sort of figured out what you're looking for, even if the question was terrible (I just Googled "jazz album racing cars" and it realized I meant "Casiopeia" by Casiopeia).
True AI might be different. But I dunno, it seems like the "combined knowledge of humanity in your pocket" paradigm has existed for a long time and didn't change much.
Try doing historical research and you’ll quickly learn that the devices in our pockets have access to a lot less information than you thought. Tons of data exists in non-digitized archives or is digitized so poorly as to be worthless. Not to mention how much information has never been written down.
One of the speed bumps/roadblocks I suspect coming for AI is how much critical information isn’t recorded in ways they can train it on
I feel like that's closer to a speed bump than a roadblock. Once people realize A) there's important training data being excluded because it's not yet scanned, and B) are willing to pay the money to have people go scan it, we really only see a few months to a year of holdup as people go pull books off of shelves and show them to a camera.
(Hint, improved image recognition will let people flip the pages faster in this process.)
I don't think you realize the scope of the problem or how many libraries would be involved. Also, only a relatively small amount of this information would even be in libraries in book form or similar media. How do you scan Stonehenge such that someone studying it could get all of the nuance and details to determine if it's related to astronomy/calendars or not? That's no small task, and multiply that by every archeological site in the world.
Millimeter wave radar seems to accomplish that exactly, and pretty well? I suspect all you need is a single F-35 to very carefully fly over the entire planet
i'm kinda imagining a skynet that has a couple F35s and is constantly flying them around important areas doing millimeter wave radar and is continuously training itself on the data, for what it's worth
this seems well within the capabilities of a pre-AGI intelligence even
A nice overview, but ultimately the believers are purely faith (FLOPS) based.
I say faith based because their FLOPS view has many assumptions in it:
1) that this is a brute force problem - if X somethings are achieved (FLOPS), then AGI results. Utterly unproven.
2) that this is a converging problem - that there *is* a solution. Maybe consciousness is a requirement for AGI, and consciousness is a function of complexity or quantum machines created by neurons, not raw amounts of data or speed of going from 0 to 1 to 0 really fast, really many times. See self driving for non-converging problems so severe that most self-driving advocates are pushing to simply separate the self driving vehicles from the humans.
3) magic robotics: that Star Trek Data type androids are going to magically appear - or at least Terminator type factories. The problem being: Factories need inputs. How much penetration has automation made in drilling oil wells, mining lithium, transporting the myriad inputs into a chip factory, etc? Robotics does exist but almost exclusively to make the relative handful of highest scale products.
4) that AGI is even needed or wanted. If no humans are needed to make anything because of Star Trek Data type androids/Terminator factories - then there are no human manufacturing/mining jobs. AGI's primary function then is to replace the human facing and human interacting bits of the economy at which point there are no human jobs, period and there is no need for customer service or lawyers or technical writing or <insert white collar job here> as the humans will have zero purchasing power, zero decision making power and no actual place in society.
Or in other words: no evidence that AGI is possible; no evidence that AGI can actually translate into real world production capabilities even if it is possible; no benefit to humans if AGI is possible and can also "take over" production.
This doesn't even take into account the fundamental problems with present AI product opacity combined with proven GOAAT (garbage out at any time).
The possibilities for fuckery with opacity plus GOAAT are endless.
Regarding 4, humans with no jobs, I see it differently a few ways. The human population is going from 8 billion to 9 billion in the next 15 years or so. Do we think there will be no jobs because there are a billion extra people? No. So why should we think there will be no jobs with a billion AIs? Because they're smarter, faster, etc. Maybe. But if they're smarter then most of them will likely be performing work that humans couldn't do as well anyway due to cognitive limitations. New pharma, physics, materials science, etc., etc. Which will create huge economic abundance. Which will mean that there will be more, not less, money in the economy to pay humans to do things. So maybe not every physics job will be filled by a human, but all the other artists, plumbers, nurses, architects, and so on will be in higher demand and will be paid more. As some parts of the economy get more efficient, the human roles in the rest of the economy become better paid. Comparative advantage.
I also think there's huge elasticity of demand in many of the high cognition fields. We could consume 10x more software, scientific research, education, and so on. So those human jobs won't go away, they will just be augmented by (or augment) AI-driven work.
I think humans will get a lot richer as a result of AI, not poorer.
Let me see if I understand your exposition: with the AI and robots making fantastic new discoveries - the net new productivity will make everyone healthy, wealthy and wise.
The problem is: you are assumptively closing on the idea that AI and/or robots will discover jack squat. That is far from clear.
Secondly, the assumption that productivity increases will accrue to the overall population as opposed to say, the AI handlers and robot creators is not congruent with recent history of other "productivity advances" in the West. In general, it is safe to say that distribution of wealth is as much a societal/governmental outcome as it is about absolute amounts of wealth. Even very poor nations can have very rich rulers, and very rich nations can have hundreds of thousands of people on the street.
Next: service jobs. That's what you are referencing by artists and plumbers and nurses etc. That's what the US economy has trended towards ever since its de-industrialization started. How's that working out? From my view, not well at all.
And finally: the super smart and creative people will always have work so who cares?
This is nothing more than a "I for one welcome my robot overlords" type statement.
Nothing about the future is certain, but failed predictions (or scams depending on POV) extending back decades are a different story.
As for the assertion that wealthier nations = wealthier citizens; I challenge this.
The United States is wealthier than it ever has been in an absolute sense - but the welfare of its citizens in the lower deciles is far worse than it was in the 1950s, 1960s, 1970s, even 1980s.
So whatever their notional wealth may be - Americans in the 60% or lower decile are fare worse off in any number of wealth metrics ranging from ability to afford health care, to cash savings as a percentage of earnings, to ability to afford college, to pretty much any economic comparison imaginable. This is disguised by "hedonic adjustments" - the rising complexity of iPhones offsets the lower savings and health care access...
> The United States is wealthier than it ever has been in an absolute sense - but the welfare of its citizens in the lower deciles is far worse than it was in the 1950s, 1960s, 1970s, even 1980s.
Part of this is our own stupidity and callousness at work. Not to get all libertarian, but it is literally illegal to house people today in the conditions you refer to from the 1950s. We have taken a view that says "luxury* or nothing", and as a result some people get nothing. (* That's "luxury" from a 1950s standpoint, or from the standpoint of most other places on the planet today.) It's basically "let them eat cake" all over again.
The point of the 1950s was not that it was absolutely better than today; the point of the 1950s was the people up and down the economic ladder had a far more equal share of production.
Nor am I particularly impressed with your assertion of 1950s housing being worse than housing of today - if for no other reason, there is an enormous amount of housing today that was built before the 1950s. SF, New York, Chicago etc - the cores of those cities all are majority "old build" in the sense of pre-1950s. Median age of residential housing in New York City is something like 90 years old.
So while I agree with what I think is your view on the economic outcome today, my view on why is not because of "stupidity and callousness" - it is the continuously widening gulf in production share between a thin sliver of society and the rest as represented by inequality index changes over time. That's enemy action.
Hmm, interesting take. For my part, I think the US population is also much wealthier, not just the country. The poverty rate has fallen significantly (https://www.vox.com/future-perfect/2023/3/10/23632910/poverty-official-supplemental-relative-absolute-measure-desmond), median household income went from about $46k in 1967 to $68k, 36% of the population now graduates from college compared to 10% in the 50's, and the share of the population without health insurance is about 9%, down from 20% or more prior to the 70's. Similar trends are true for most of the countries that have seen significant economic growth.
That's not to say there aren't problems, like healthcare and education being too expensive. But the overall trend is pretty positive, and I expect that to remain true as we see increased economic growth due to AI and other innovations.
If you view the ability to eat and have shelter - it isn't clear to me that these are better. The "penniless" farmer of prior eras was productive and largely self sufficient - contrast this with the multi-generational welfare recipient in urban cores.
Household income: if I plug in 46000 in 1967 in the BLS' CPI calculator, it equates to $425000 in May 2023 dollars. Ouch. But likely you meant 4,600 in 1967.
In reality, median household income in 1967 was 7200 - BLS CPI adjustment to today = $66.5K. If I use your $68K number, there has been almost zero improvement in household income over inflation for 56 years.
More people going to college - is that really a win? Especially if a huge number of them get nothing more out of it than tens of thousands of dollars in undischargeable debt?
Health insurance: don't even get me started there. Health insurance "coverage" is a scam particularly for not sufficiently poor or wealthy people. I guarantee you that bankruptcies due to medical debt was not even top 50 in 1967.
As for trends being positive: interesting that you think so. Americans certainly have far less financial security than they have for decades - and it gets worse as you go down the age scale. Production of core consumption - anything outside of food and energy - is an abysmally low fraction of the US economy and consequently American jobs. The interest payments on US debt are going to start exceeding the US military budget pretty soon.
The US spent 8.03% of GDP/$80B on defense in 1970 (defense = Vietnam war); this number plugged into the BLS CPI calculator would be $643 billion today vs. the actual defense budget of $800B today (with a lot of extra spend elsewhere).
Households in 1970 were prosperous with a single wage earner; now the vast majority of households require 2 wage earners to survive.
Not the least but sure where you get your rosy view from.
One important parameter that's missing from this analysis is the maximum percent of the economy that can be automated, regardless of how smart the AI is. We can already fully automate coffee production, but people still pay extra to watch a human make their coffee. Teleoperated surgical robots are technically able to perform surgery, but they got hung up on patient squimishness and legal blocks. Because of cost disease, most of the money in the economy is already spent on things that are difficult to automate, and the price of anything that can be automated with AI will drop to 0. That will limit the budget available for research and training.
Another missing parameter is the cost of automating tasks. It might be cheaper to pay immigrants to fold towels or fix sinks than to build a high tech robot with expensive GPUs to do the same tasks. If this parameter is high enough, then unskilled human labor could coexist with superintelligent AI.
This "red-tape delay" parameter does sound important to me. It also seems like it will vary regionally, and AI will tend to advance / take-on-duties primarily in the places that are most conducive to it.
I suspect that Silicon Valley, where most of the current work is being done, is already very permissive towards letting AI take over various job positions.
Also, the coffee industry doesn't greatly impact the development speed of the AIs, so while it's relevant to "the last 5% of AI labor", the truly transformative parts of the curve are going to be the 30% to 70% stretch.
I don’t think the discrepancy between Ajeya’s update and the CCF involves an error by either party. Ajeya mentions that her update towards shorter timelines was in part based on endogeneities in spending and research progress, which the CCF explicitly models. If Ajeya’s updates are partially downstream of considerations like Davidson’s, then there’s no inconsistency.
>The face of Mt. Everest is gradual and continuous; for each point on the mountain, the points 1 mm away aren’t too much higher or lower. But you still wouldn’t want to ski down it
Someone did, and all it cost were the lives of six sherpas and one of his film crew:
P is the gross substiution between capital and labor. Until now there is consensus that p is negative (and thereby K and L gross complements which means there are labor bottlenecks). The central Parameter of R& D labor substitution only focuses on the R& D sector and there is no empirical evidence to date that they are substitutes. So the critical assumption that AI will substitute human RD completely drives this results
Hi early, what scares me the most about this - and where I truly hope I'm wrong - is that in the system as is, people add value and so far as we pay taxes you're not necessary for creating the products that we consume. So to the government/corperations we are assets on a balance sheet and it has to take our interests into consideration.
Once we can no longer produce anything for cheaper than an algorithm or robot can, we become liabilities, in which case we are thrown a bone big enough to keep us from rioting but nothing more.
You see this a lot in natural resource producing countries: because the government and key government figures are able to fund all of their activities by controlling one or two key industries, you don't have to develop an economy or take care of your citizens. I see AI being able to do everything at scale as no different than this.
I welcome anyone to poke holes in this line of reasoning, it would definitely help me sleep better at night.
I have a similar concern. There seems to be an assumption by some that more AI is just better for everyone, full stop. But in the lead up to takeoff, and certainly after it happens, the world still needs people to mine the resources for the tech that makes up the physical components of AI (because, as has been stated above, it's cheaper to pay people or use slave labour than to employ expensive tech to mine these resources). Are folks concerned at all about the socioeconomic and environmental repercussions of increasingly more AI, especially in low income nations that are responsible for supplying the raw resources for this? Or is it rather looked upon that the creation of superhuman AI is an overall good for humanity, and thus people and environments can be sacrificed for this greater good?
I have similar fears about this to you, but there are a couple holes I think I might have spotted.
A major disanalogy between resource producing countries and AI is that it's much easier for the government to control resource extraction industries. It just needs to own the land where the resources are located. This doesn't work for AI, you can set up computers pretty much anywhere. People looking to get a fairer share of the economy will be able to pool their resources and set up new companies. Or maybe just buy stock in the existing ones.
I think Robin Hanson has written about this in his AI work, about how we might transition from a society where everyone makes something with human capital to one where we all are partial owners of machine capital. I admit I find a lot of his speculation frightening and disturbing, but he doesn't seem to think biological humans are in serious economic trouble.
Another factor to consider is bargaining. If the government wants to "throw people a bone" to prevent riots, the size of that bone it can afford will be dependent on how productive things are. How likely people are to riot will partly be a factor of how much they think the government can give them. And if AI is really as productive as people are saying, it might end up being a very large bone indeed. Like, if the bone people get in resource extracting countries is a chicken bone, the one in the post AI world will probably be more like a Seismosaur.
Practically speaking, I've made sure to invest my money, so that hopefully I'll get some tiny percentage of whatever economic boom AI makes for corporations.
There's a further issue, that military/police/intelligence are currently jobs of people, who at some level need to believe they're in the right. The more these particular jobs are automated, the less the state needs to face this type of restraint.
This post and the Biological Anchors one are great for thinking through the details of how this stuff will happen.
I notice that the assumption is that most of the AI workers will be focused on AI research. In the context of the economy being "mobilized" for AI, this makes sense, but I wonder if that version will hold true. During let's say the dot com boom, most of the work was going into "I want to sell groceries on the internet," not "the internet". Even if everyone is convinced that AI is the biggest opportunity, won't most of the work go into high return applications that don't advance the state of the art? Or for that matter other fields entirely? Maybe 90% of the human-level AIs will be working on stock trading, medical research, fusion modeling, public policy planning, defense, cybersecurity, and whatever else is economically valuable. 100 million is a lot when applied in one place, but not as many when applied across the entire world economy.
Of course, then you could pretty easily imagine 500 million or whatever, but I think a next step is to theorize about these aspects a little more deeply.
We also don't really know what superintelligence means. Is it just a bunch of Von Neumanns running around solving problems faster but in a basically grokkable way, or is it more of the "magic" superintelligence that groups like MIRI expect, where it can wave a wand and create whatever nanotech it wants.
Finally, isn't the bottleneck for much of AI development going to be data pretty soon (or already)? People seem to agree that most current models are already undertrained, which to my understanding involves the size and quality of the data set as well as the amount of compute used. And don't the latest results show that feeding AI generated content into AI training tends to blow the whole process up? I think this is another possible bottleneck we haven't dug into enough yet.
In the post it's "afterwards you used 10% of your training compute to run AGIs doing software research..." which is compatible with your "Maybe 90% of the human-level AIs will be working on (other stuff)".
I haven't seen MIRI propose any magic; Drexler's ideas aren't magic.
Data is a plausible upcoming bottleneck, though the future is uncertain. There's lots more data in the world, and methods don't stand still. I wouldn't be surprised by either a data bottleneck on developments or not.
Yeah, fair points on the 10% and data bottleneck idea.
Maybe I'm not well versed enough in all the arguments, but when I see some of the debates arrive at "because nanotech," that still sounds a bit like magic to me.
I read Nanosystems when it came out, then years later watched the climate of opinion turn against Drexler for poor reasons. Neural nets being unfashionable for decades had different reasons but it's an example how productive ideas can languish.
A lot of people made very foolish projections about what nano-tech could be capable of, which tended to obscure it's real value. Besides, be can't really do it profitably yet except in very special cases. Micro-tech, though, has been being pushed real hard. And that's at about the limit of what we can do outside a lab.
I don't think they necessarily believe most AIs will be used on AI research, just that this is the use case which is interesting and important to model because it produces (most of) the feedback effects.
Some of the people pushing these narratives talk instead about economic doubling times of >1 year, which suggests most of the AIs are doing other economic activity.
I have a hard time believing AI intelligence won't be bottle-necked. In particular I'm thinking of Feynman's quote "If it disagrees with experiment, it's wrong.". Any intelligence, even one much smarter than us, will be limited by its understanding of the physical world, and as we see with the many many wacky theories that come out of physicists, all the speculation in the world is nearly worthless in the face of empirical evidence. Sure, the AI might come up with a plan to build a warp capable starship built by microbes, but will the plan work when it's implemented, or will it be a lot of plausible sounding BS like a ChatGPT academic reference. I think what a lot of singularity proponents miss is that intelligence is not knowledge. In particular, any intelligent AI will face the same challenges we do: which knowledge is real and valuable, and which is superstition, snake oil, propaganda, wishful thinking, self-promotion, or just plain wrong. Often the only way to tell for sure is to try something, but that is a slow physical process that costs time and money. We already have a problem trying to discern truth from fiction on the internet, after seeing ChatGPT, I'm only convinced that this problem is going to get worse, much worse. The only real answer is to improve gatekeeping: providing trusted sources of knowledge that carefully vet every contribution, not unlike academic paper gatekeeping, but even more stringent (because the academics will be publishing AI BS too, that looks like a proper scientific study but is totally bunk. If you think the replication crisis is bad now... hold on to your butts.)
IIUC. warp capable spaceships require immense masses to make them work. So unless there's some new physics, that won't happen.
Current AIs can't do complex large-scale engineering, but I see no reason in principle why an AI couldn't, and do it accurately. We aren't talking about new physics here, we're talking about taking established principles and applying they in ways beyond those that humans have been able to. Nothing magic involved. Well, unless you need to solve turbulence equations to get it to work.
Now for physics, I agree, the AI might be able to come up with new theories, but until they are tested, and tested under various conditions, you can't really trust them. All it can say is "This theory is consistent will all the data I have.".
I'm still struggling with a basic question I feel should be answered but either it isn't or I am not getting it.
The basic question is this : Does a brute force pattern matcher morph into true intelligence as you increase said brute force?
Now, don't get me wrong - I think pattern matching is very useful and can do a lot of things. There's a reason editors, copy writers, visual artists, coders and indeed even scenarists are worried about their jobs. Whether those jobs are 20% of the workforce as per Scott's quip, I don't know but it's definitely impressive for a "mere" pattern matcher.
OTOH, I cannot ask chatGPT and other AI "assistants" to schedule my meetings and organise my trips given existing constraints on my time/my counterparts' time etc. without very close supervision i.e. it's still not a smart assistant.
And the question is - can a pattern matcher mutate into a smart assistant without some breakthrough in conceptualizing intelligence/the world/whatever pattern matchers are presently missing?
That’s not *a* question, that’s *the* question. We’ll probably never be able to prove it won’t happen, and we won’t know that it does unless brute force gives us a genuine intelligence.
There is some evidence that brute force might induce real intelligence, though. Notably the fact that we’ve observed “phase shifts” in neural networks where their entire structure changes significantly just by increasing size. Also, as their size increases LLMs develop “emergent abilities” - new abilities (like, say, multiplication) that appear spontaneously. It’s plausible that everything needed for “true intelligence” is just a few emergent abilities/phase shifts away.
Thanks, that's a much appreciated answer. It clarifies things (for me, at least).
But the new abilities, if I understand correctly, are still about better pattern matching. As in, GPT4 (or whatever) no longer messes chess moves (or maths) nearly as much as its predecessors but it's just b/c it got more data to work on/got better at identifying patterns within the dataset. It still hasn't deduce the rules of maths/of chess.
I'm wondering. If you gave a human a fair few examples of chess games and asked him to figure out the rules, wouldn't he be able to? I'm pretty sure he would.
Said differently, do we know how to 'explain'/program a LLMs so that it 'knows' that hallucinating is wrong and that there's a truth out there it needs to match.
I don't quite agree. The whole reason emergent abilities are surprising, and they are called "emergent", is that they do represent meaningful changes in what the network knows. GPT-3 knew how to add 5 digits numbers when GPT-2 did not, and GPT-4 got a 40% on the AP calculus exam when GPT-3 could not do better than random. And with Chess, there's a lot of evidence that the network has learned the rules.
But as to your last sentence, we do not currently have a way for the network to reliably identify that it's hallucinated, and we don't know if such a thing will emerge just from scaling. I'm not convinced that the problem ever goes away just by scaling, but I'm also not convinced that other techniques can't fix/greatly mitigate hallucination.
What about recursive pattern matching? I.e. matching the pasterns in which patterns occur? Or even layering that deeper. I find it less clear that that isn't at the heart of intelligence. But I think part of the problem is that you need a complete set of operations at the base level for handling not just linear structures, but n-dimensional structures. (I don't know how large n can get, but I suspect that it's at least 4, and probably more, though some of them may have very restricted ranges. And not all have a interdimensional distance defined. E.g. "What's the distance between a sour taste and the baseball I see?")
It turns out that one of the topics people have had this argument about for decades was language. Chomsky and colleagues said that to learn a language, you needed to acquire universal rules that could be applied consistently, generating a potentially infinite set of sentences from finite data. His “poverty of the stimulus” argument claimed that 3-5 year olds could learn languages with all these grammatical rules, despite having way too little language exposure, and this was said to prove that there must be a “language instinct” that explains both linguistic universals of human languages (eg, why they all have things like nouns and verbs rather than the structures that artificial languages like arithmetic Al notation and computer languages have) and how children can learn languages quickly. Neural nets can easily be shown to have *no* possibility of achieving full representation of these rules in universal generality (if nothing else, GPT can’t currently deal with a sentence that is more than 2000 tokens long or whatever).
But then GPT really seems to have something like human-level fluency with language. If this is how good something with no innate language sense, and an inability to grasp fully general rules, can do, then why be so confident that *humans* actually have the structures that Chomsky has claimed? It’s true that GPT has been trained on far more data than your average five year old has seen - but there’s no reason to think that we are at the limit of efficiency of using data in a less specialized architecture like a transformer. Maybe it can’t actually learn “rules” of human language - but maybe humans don’t either, even though we think we do.
I'm way out of my depth on the scientific substance here but it seems to me that the fact that all languages have nouns and verbs rather than arithmetic notations or binary language doesn't seem overly mysterious.
We must have develop language to facilitate communication and we are discreet individuals acting in a material world hence it makes sense to develop a language that would take subjects, objects (nouns) and then add descriptors of action (verbs) to communicate info about the real world to our fellow tribesmen?
Except that we’ve discovered while inventing computer languages that there *isn’t* any one thing that makes sense here. Some computer languages are object oriented, some are purely functional, some process lists. All are capable of writing code for all sorts of purposes and some are more convenient for one rather than another. Given that human languages have so much other variety, you would expect that some would explore these other paradigms, especially since the sciences have shown us that mathematical notation actually is really good at expressing many core features of the world.
> Does a brute force pattern matcher morph into true intelligence as you increase said brute force?
I'm a bit confused by how any answer to this (yes/no/maybe) has much bearing on the key questions in the domain Tom's report, and Scott's review of it, is in (should transformative AI risk be a concern, and if so what to do about it). Can you elaborate? I'm guessing you're referring to https://www.gwern.net/Scaling-hypothesis?
Well, it does seem linked to me inasmuch as I understood what Tom/Scott are describing as being a quantification of the scaling hypothesis?
They mention human intelligence being 10^35 FLOPs while GPT4 is 'only' 10^24 FLOPs and kinda assume that going from 10^24 to 10^35 automatically means acquiring human-like intelligence.
I'm asking if that assumption is valid/this debate has been decided. Or will a GPT at 10^35 FLOPs still hallucinate and invent legal cases when asked to write a brief?
Sorry for the delay in coming back to this and thanks for the links. I had missed these Slate Star Codex entries. The wisecrack about angelic language and user base was LOL.
re. the substantive question - okay, our brains are doing a lot of pattern matching and this is behind a lot of our "learning" and world model and it won't be a fair comparison with LLMs until they have sensors' input to digest. Indeed, how does a Boston Dynamics robot learn not to fall? I think there's a fair bit of try and repeat not to dissimilar from a young homo sapiens learning to stand on his own two feet...
Still - as someone mentioned, we're not blank slate. How much of our brain isn't blank? Is it just survival instinct/sex drive/flight or fly? Or are we born with more than that? Do we know?
Does all of this progress assume that AI has corrected its errors and is plunging ahead with perfect knowledge? I am thinking about AI's ability to reside in a fictional universe, making up legal citations to bolster its thesis. (Has it ever apologized for that? I have found AI to be good at apologizing.) I am also thinking about my brief forays into testing Chat-something-or-other, where each time it made an obvious mistake (by not examining counteraguments), apologized for the mistake upon my mention, then double downed by incorporating the original mistake into the revised argument. Happened several times with different original questions. BTW these mistakes were pretty fundamental, and I suspect they would not be avoided in a 10^35 flop machine.
I see this progress as racing toward a big train wreck.
> Does all of this progress assume that AI has corrected its errors and is plunging ahead with perfect knowledge?
No, and I'm confused why you'd think that -- it's not the impression I got from Tom's modeling at all.
I do intuitively agree with "I see this progress as racing toward a big train wreck." unless we do something about it, but exactly what (in particular, what intermediate-term goals should be pursued to avoid or mitigate the impending train wreck) is still not very clear to me. I should get around to reading this survey: https://rethinkpriorities.org/publications/survey-on-intermediate-goals-in-ai-governance
Technological history contains many examples of curves with left turns fizzling out.
Take aircraft speed 1900 to 1965. You start with the Wright Flyers (~56km/h), get to jet engines in WW2 and finally to the SR-71 (Mach 3.5). And that is the endpoint in terms of raw speed, because physics.
Or take rockets (as judged by payload * delta-v). Von Braun's V2 was build in 1942. 25 years later, in 1967, the Saturn V. And that was it, because physics.
Single core CPU speed. 1985: i386, 40MHz. Ten years later, 1995, The pentium pro runs at 200MHz. Finally, in 2004, the Pentium 4 reaches 3.8GHz. Skip forward to 2022, and the biggest i9 goes to 5.3 GHz. Because physics.
Since we reached the two aerospace endpoints, our ability to do engineering research has been augmented by our computing tech going from slide rules to supercomputers. Within an order of magnitude, this has done nothing. (Yes, Space X is impressive, but they are still governed by the tyranny of the rocket equation.)
(All of the previous endpoints were dictated by a combination of physics and economy. If there were a few trillions to be made by building a bigger rocket than the Saturn V, or making a plane reach Mach 4, or designing a desktop with a 6GHz CPU speed, we could certainly accomplish it.)
It seems totally possible to me that the AI endpoint where we spend half of the US GDP to train an AI with the intelligence of Einstein, but even an army of Einsteins can not make a substantial improvement to hardware or training algorithms. Perhaps running an instance of such an AI will not even be cheaper than employing a human of the same intelligence.
Of course, for each of the of my endpoint examples, there was a concrete physical reason imposing diminishing returns. I don't think we know of any useful limitations for intelligence. We already now that physics allow for human level intelligence using about a kilogram of neurons and a power consumption like a light bulb. Could be that silicon based neural nets have other limitations. Or it could be that intelligence hits a wall at the best human level, and every further IQ point will require ten times as many neurons.
So far, we appear to live in a lucky world (which might be survivor bias). Trinity did not cause a fusion chain reaction in the atmosphere. The LHC did not create any earth-consuming black holes. Even with the knowledge of how fission works, building nukes is much harder than making a shiv. Burning fossil fuels did not trigger catastrophic climate changes before 1900. Given how unlikely we are to solve the alignment problem in time, most of the probability mass of long-term human agency might be allocated in the worlds where AI goes the way of the jet engine.
Serious question: **was** the Saturn V pushing the limit of our technology and the rocket equation?
I'd always thought that our retreat from space was due to lack of political will, and the lack of further rocket progress was due to that retreat. Have I been wrong?
Depends what sort of progress you're talking about.
Could we have built a bigger and more powerful rocket than the Saturn V with 1970s technology? Probably, there were plausible designs at the time, there just wasn't an economic reason to do so.
Could we have built a rocket that was significantly more efficient in terms of delta V per kilogram of fuel? I don't think so. A Falcon Heavy in fully-expendable mode has about the same payload fraction as a Saturn V. We've made rockets much *cheaper*, and that's made some new space projects economically feasible, but the basic math of the rocket equation hasn't changed, so "just make the rocket bigger" still has diminishing returns.
(Similarly, modern airplanes are bigger and can carry a lot more cargo more efficiently, but they don't fly any faster because the physical limitations imposed by drag haven't changed.)
That's about what I thought. Maybe it's no longer feasible to make rockets bigger, in the sense that would impress a 3-year-old. But we can make them cheaper and more numerous and more reliable. (As long as we let Elon Musk run things, anyway.)
Bean is right, battleships are the coolest ships ever. But, alas, that's not what modern navies need, so we probably won't be seeing any more of them.
> Could we have built a rocket that was significantly more efficient in terms of delta V per kilogram of fuel? I don't think so.
I agree with you. A given fuel reaction will have a given reaction enthalpy, which determines the temperature of the exhaust, which in turn determines (or at least limits) the exhaust velocity, which enters the rocket equation as the specific impulse.
First loophole is that within the atmosphere, you can use air both to provide an oxidant and also use air to distribute the reaction energy over more mass (which will give you more momentum per energy, p=sqrt(2*E*m)). This will get you turbojet engines. However, these engines are highly dependent on airspeed, an engine good at carrying you through the thick lower atmosphere at terminal velocity will not be great at pushing you to mach 3 in the upper atmosphere. Also, jet engines seem harder to scale up than rocket engines?
The second loophole is to use either a reaction with an higher enthalpy or use something other than a chemical reaction to accelerate your reaction mass. You can heat up your reaction mass using a nuclear reactor (nuclear thermal rocket). This is still limited by material properties, but allows you to exhaust hydrogen (mass 2u) instead of water (mass 18u) or whatever, so you will get a higher exhaust velocity at a given temperature. Or you can ionize and accelerate your reaction mass electrically in what is basically a particle accelerator (ion thruster). This allows you to reach very high specific impulses if you have sufficient energy available. As your power is limited your thrust will be tiny, though.
Disclaimer: most of my spacecraft knowledge comes from Kerbal. (In KSP, ion thrusters have the handycap that you can not speed up time while under thrust. This means that a mission which calls for a month of acceleration at 5mN will take a month to simulate. So instead I mostly use that bulky engine with the big trefoil symbol.)
The Saturn V was pushing the limit of our technology in many respects. Note that we're still *using* the literal technology of Saturn and Apollo today; the Marquardt R-4D reaction control thruster of the Apollo spacecraft is still found on many modern communications satellites, and the Pratt & Whitney RL-10 of the Saturn I upper stage will be used on the Vulcan launch vehicle making its first flight later this year (fingers crossed). In both cases with various incremental improvements, but the Saturn/Apollo people had come within 90% of what is theoretically achievable with chemical rocket propulsion that doesn't kill everyone in Florida.
That said, Saturn/Apollo pushed our technology to the limit in a really stupid direction, that of "build a ginormous rocket to take people all the way to the moon and back nonstop, minimizing development time above all other concerns". A ginormous rocket that takes people all the way from the Earth to the Moon and back, is as daft an idea as a ginormous flat-bottom Mississippi riverboat that takes people all the way from St. Louis to Paris and back, eschewing the obvious "transfer to an oceanic steamship in New Orleans" solution because that would take too long.
Our retreat from space was due mostly to our having chosen to race out into space in a ludicrously expensive manner because we didn't care about the cost, until we did. Our subsequent progress back into space has come from applying technology that's mostly not that far beyond Saturn/Apollo, in a manner that's somewhat more focused on cost. But there's still a long way to go in that regard.
Fine point. Also: It takes 1000 calories to raise the temperature of 1 liter of water by one degree C. - No Singularity AI will need any less. - Though it might be helpful in making fusion et al. viable. And abundant energy makes heating water and sending many, many Saturn Vs less costly.
There are some clever workarounds though. A heat pump needs far less than 1000 calories of energy to be fed into it electrically to pump 1000 calories of ambient heat into a liter of water. The thousand calories have to come from somewhere - but it could be a source outside your usual supply chain.
There will definitely be physical limits - that's part of why the S-curve levels off at the top. They just seem pretty far away right now - the equivalent of rockets being bounded by the lightspeed limit.
We know that intelligence is possible not just up to "the human level", but up to the level of the greatest human geniuses. AIs that reached the level of the greatest human geniuses (while being mass-producible and probably runnable at faster than human speeds) would already be a singularity-level technology.
I'm not sure I believe that having Genius On Tap would be singularity level. It would be transformative, perhaps even more than the printing press was. But in achieving things in the world, ideas and raw intelligence seem less important than motivation, energy, luck, and charisma. I would rather start a project with a charismatic and driven partner (who still has well above average intelligence) than someone with off the scale intelligence but less charisma or energy. Maybe that just reflects that I am neither charismatic nor driven (I'm spending time commenting here instead of building, after all) but I have worked with people much more intelligent than I am and this was my conclusion: it was a privilege to spend time with them and I learned a lot, but they were not indispensable for team success. Intelligence was important but beyond a threshold other factors dominated. The problems that need to be solved often don't have intelligence on the critical path, but social engineering an exemption from an onerous rule, or persisting with an interaction past the point of rudeness, or doing an unreasonable amount of boring drudgery often are.
But interestingly, rockets did level off at far less than the light speed limit, just as various kinds of heat engines all ended up leveling off at their own points below that of the Carnot limit. Solar energy is going through various sorts of phase transitions that get new limits theoretically closer to the Carnot limit, but rockets and airplanes aren’t getting closer to the light speed limit in any appreciable way.
Per Wikipedia, running a human (genius) level AI at faster speed is called a "speed superintelligence", and WP cites some article which claims that this can lead to a singularity. [0]
I am more skeptical. If we have instances of virtual John von Neumanns running at a million times the speed of the the non-virtual one, that will certainly get us some technological progress. Basically everything that can be discovered or invented by JvN will be discovered or invented almost instantaneously. The question is how transformative that would be for society. While the meat space JvN certainly had a bigger impact than almost all other humans on earth, his absolute impact on the average human is nonetheless moderate (at least when assuming that we would have invented mutually assured destruction without him in time).
Theoretical physics is one area where raw intelligence is probably more helpful than a lot of other areas (like inventing more practical (and impactful) things like the steam engine, PCR, smartphones, the web, cars, fridges, antibiotics, nukes). In a nutshell, the latest working model of particle physics is called the Standard Model and dates to 1975. I would argue that the reason we have not made progress with the open questions since (like unifying quantum mechanics with gravity) is *not* that von Neumann died in 1957 and nobody else was smart enough to figure it out. Theoretical physics attracts super geniuses like few other things do, but it seems that discovering a theory of everything (TOE, that is, all four forces) or even 'everything except gravity' (e.g. unifying electroweak and strong interactions) seems significantly harder than discovering General Relativity. I am not sure TOE would be found if we spent a million von Neumann years on searching.
(One might argue that the reason Newton and Einstein became such household names was that they were active at a time when science progressed very fast.)
Also, the Standard Model did not change the world. What changed the world was the earlier discovery of (mostly non-relativistic) quantum mechanics, which is required to understand chemistry and what goes on in semiconductors, which many of the innovations of the last few decades involve. So even if the AI discovers new physics theories, this might not change the world we live in fundamentally.
I think that while we would get some nifty inventions out of a 'speed superintelligence', what would really change society is genius level intelligence becoming to cheap to meter. I think the jump forward might be similar than the jump between the heyday of the Roman Empire and the our world.
The level of everyday convenience a rich Roman enjoyed is probably similar to that of the median American today. Both can take a hot bath at short notice with barely lifting a finger, have heating in the winter with no effort on their part, get both exotic fruits and other tasty food prepared for them and can get plenty of entertainment without even having to leave the house. The difference is that for the patricians, this was all powered by slave labor (and so would never scale to everyone), while for the American, this is powered by the fruits of industrialization, which give us central heating, long distance shipping, microwaves and Netflix.
I imagine that billionaires today can afford to have a staff of experts looking full-time after their physical and mental well-being, while the median person goes to half the health checkups which are recommended and perhaps does therapy now and then. In the world where human supergeniuses can be emulated cheaply, this will be available for everyone. Either that, or the machines decide that they do not really want to spend a subjective decade thinking about how to best cheer me up on some Thursday evening and overthrow us instead. Neither one deserves the term singularity, in my opinion.
"It intuitively feels like lemurs, gibbons, chimps, and homo erectus were all more or less just monkey-like things plus or minus the ability to wave sharp sticks - and then came homo sapiens, with the potential to build nukes and travel to the moon. In other words, there wasn’t a smooth evolutionary landscape, there was a discontinuity where a host of new capabilities became suddenly possible."
Homo erectus did unlock lots of new capabilities, like stone tools and fire. Other apes were all limited to a narrow range of environments, while erectus was able to expand to all of Africa and much of Eurasia.
You beat me to it. Homo erectus seems closer to homo sapiens than chimpanzees. Recall also that there were actual humans with stone-age technology when first contacted in the 20th century. Not too far from erectus.
Agreed. The modern human appeared ~ 300000 BCE. Until ~10000 BCE, they would have looked like the continuation of homo habilis et al in terms of singularity potential. Then climate conditions (or seed cultivation or intelligence increase or whatever) go right, and within an eyeblink you have civilization with metal use, aircrafts and TikTok.
I'm actually devil's advocating here - my actual position is that there's smooth exponential/hyperbolic progress in primate impressiveness - chimps are genuinely more impressive than gibbons - and it's just that the very steep part of the exponential/hyperbolic curve makes the rest look pitiful in retrospect.
And interestingly, even after erectus, other hominid species developed new stone tool making technologies, some of which spread to other hominid species. It’s hard to know how many other technological developments may have spread back and forth between species.
>We need clear criteria for when AI labs will halt progress to reassess safety - not just because current progress is fast, but because progress will get faster just when it becomes most dangerous.
Just because we need something, doesn't mean that it's feasible to get. Either alignment is easy, or an (obviously unrealistic in practice) pivotal act is required. Or a global thermonuclear war will indefinitely delay these considerations, for another source of hope ;)
> But a few months ago, Ajeya Cotra of Bio Anchors updated her estimate to 2040. Some of this update was because she read Bio Anchors and was convinced, but some was because of other considerations. Now CCF is later than (updated) Bio Anchors. Someone is wrong and needs to update, which might mean we should think of CCF as predicting earlier than 2043. I suggest discovering some new consideration which allows a revision to the mid-2030s, which would also match the current vibes.
She updated after reading her own work? I suspect "she read CCF" is what you meant to write.
Well, the LLMs themselves have ~10^12 parameters... :-)
( My last manager, before I retired, was queasy about tuning parameters from model fitting in our software ... _but_ was enthusiastic about machine learning, which can be viewed as fitting tuning parameters by the gigabyte... :-) )
“...was starting to feel like they were all monkeys playing with a microwave. Push a button, a light comes on inside, so it’s a light. Push a different button and stick your hand inside, it burns you, so it’s a weapon. Learn to open and close the door, it’s a place to hide things. Never grasping what it actually did, and *maybe not even having the framework necessary to figure it out.* No monkey ever reheated a frozen burrito.”
I love the idea of 100 million agents being unprecedentedly bottlenecked. AI advancement is stalled for years because audit cluster 0xA86F9 requires 17 quintillion of the new TPS cover sheets but each one requires 489! joint sign offs.
Is this preceded by that audit cluster making breakthroughs in number theory to allow it to break the hashing function, or it taking over most of the world's computing capacity to brute force the hash?
While I'm not an AI researcher, there's an underlying assumption in this conversation that doesn't line up with my experience watching models get built. Compute is not really an input to model performance; it's more of an intermediate metric.
We are not compute capacity constrained. If more compute alone would produce a better model, then the major players would have no problem spending the money. Rather, there are a bunch of problems that have to get solved at each scale in order to effectively deploy more compute. Problems like, how do we get massively parallelized GPUs to talk to each other when we have to fragment model training across multiple data centers? Or where are we going to get an order of magnitude more data and how will we clean it?
Obviously these problems are tractable, and the story of progress is that we keep solving them. But there's no guarantee that that will continue, or follow any specific pace. Keep in mind that AI has gone through several major "winters" where research progress slowed to a crawl. This makes probabilistic models like this ring hollow to me. You're reducing innovation to a coefficient, when you actually have no idea the probable distribution of values for that coefficient.
Pessimistically, the percentage of jobs that AI can take now is negative, because of the hallucination problem. >Every< piece that an AI writes has to be checked by >at least one human< for errors and just plain making stuff up. Needless to say this is no way to conduct research of any sort; we have enough humans making things up in research already.
For writing code? The code that is written may work, but you have to test it. The test cases need to be written (certainly checked) by a human because the AI writer has no sense of whether this code actually does what it's supposed to.
This has been hyped by journalists, who really don't know what they're talking about. Or perhaps they do, since they write opinion pieces and present them as fact.
The takeoff curves assume that we have anything right now that it capable of doing any human job. Extrapolating from zero is an exercise in futility.
So if I hire a junior engineer, who solves the problem incorrectly, have I just hired a negative person?
What if I hire a principal level engineer who only makes one dumb mistake every 10 years? Have I still just hired a negative person?
Put another way, why are the normal checks humans put on each other, like unit tests and integration tests suddenly no longer sufficient when the code is written by silicon instead of carbon?
The AI requires much more checking, because you can generally trust that the person (even if very junior) is attempting to produce coherent code, and also generally testing that code.
People can be embarrassed into better behavior. AI, so far, cannot.
And yes, mistakes can be extremely costly. It is indeed possible that a senior engineer can make a code mistake that costs the company more than the value he added. For example, the Mars probe that was lost on landing where they figured the problem was due to using metric in one system and Imperial in another interfacing system. The probe likely cost more than the value-added for all the people who reviewed that code altogether.
I think it's better to talk about AI taking over tasks rather than taking over jobs. I'd say GPT4 can do at least 5% of white-collar tasks out there (a conservative estimate IMO) but that doesn't imply it can do 5% of jobs. Additionally, it doesn't seem clear to me that our current AI can invent new things, and it isn't clear to me that it is trending towards having the ability to invent new things. I do agree that AI can make current AI researchers more productive, which should speed up the development of AI, but this all seems to rely on the idea that reaching AGI using our current methods is possible.
I'm also curious what happens as the amount of human output decreases. Let's use a field like Digital Marketing as an example. Currently, there is a great deal of human output on the internet regarding Digital Marketing strategies/fads/tech/etc. As AI replaces humans working in this space, the amount of human output will decrease. If we don't hit something like "Artificial Marketing Intelligence" in time, won't the drop in human output cause the AI to stagnate?
CoPilot et al. seem to me to be code generators, which we've had for decades (what is an assembler after all). It's another level up, optimistically. Taking over tasks also seems more realistic.
Although...
I've been at my current workplace for some time. When I started there was a classification for a "programmer" job, but no one had been hired for it for a while - the first step was "programmer/analyst" at a higher rate, and also involved more than taking very detailed instructions and writing small subroutines in COBOL and JCL.
The "programmer/analyst" classification doesn't exist anymore; you get hired at "Developer" level now. More analysis, interacting with the business side to get requirements, etc.
At a certain point, taking over tasks means that the beginner jobs get automated out of existence I think.
OTOH, some "beginner" jobs that are difficult to automate because the real requirements aren't the advertised ones. An intern doesn't really do anything an employee couldn't, but having one does take over routine tasks. The intern is actually being examined for things like thoughtful initiative, as well as things like showing up on time.
At least in the ideal world. Often the interns get abused - generally because they are in a field overflowing with starry-eyed idealists.
The vibes point is an interesting one. When everyone is publishing and wants to be taken seriously, they may tend to publish numbers in line with prior estimates. This causes estimates to clump and results in a landscape of superficially close estimates that risk portraying more confidence/consensus than warranted.
My friend and I published our own estimates in a 114 page submission to the Open Phil contest where we explicitly tried to make our best model with as little vibes contamination as we could manage. We ended up predicting that the odds of AGI taking our jobs in 20 was a bit under 1%.
Scott, would it be possible to split off the AI discussions from the rest of the newsletter? I greatly value the pre-AI-discussion ASC/SSC, but think the AI stuff is both far too speculative and something too removed from my life to want to get sucked into it (and, alas, I tend to get sucked into it). On the other hand, the rest of ASC content is either fascinating or is directly relevant to my work in mental health counseling.
I've sometimes wished there was an option to follow author/tag combos, not just authors. Substack doesn't support tags, but SSC had them, e.g. https://slatestarcodex.com/tag/charity/
ACX is a cobbled-together collection of about a dozen different topics and I don't think there's a stronger argument for separating out AI than other topics.
I'm trying to subtitle all my AI posts "Machine Alignment Monday" (though I often forget). If you block that phrase on your email, you should avoid getting them.
There's a difference between can do human tasks in theory (e.g., with lots of hand-holding and direction by a human), and can literally do the task (no hand-holding or guidance), and can ID the tasks to do. That is, GPT can likely already "do" 20% of human tasks in a service-oriented economy, but can't do the tasks on its own, let alone ID the tasks. At some point, if the AI is legion and smarter/more competent than humans, you shouldn't need any humans in the loop. Those are all the interesting inflection points that don't seem to be captured by the theoretical lines, which instead seem focused on technical capability rather than practical use (if genius AI arrives tomorrow, but can't get itself into the industrial loop to run things for a couple decades, then it's a couple decades, not tomorrow).
The “easy” response to this is to always train on the same dataset from 2021. That at least prevents pollution of data and the problems of training on machine generated data. It does make it a lot slower to grow the training the way these models assume though. (More likely there are some smart things you can do to avoid training on ai-generated data, but we are definitely quickly reaching a bottleneck in the amount of data of the sort desired for training.)
> The third bucket of ‘evidence’ is simply doing inside‐view thought experiments about what you think would happen in a world with zillions of AGIs working on (e.g.) hardware R&D. How much more quickly could they improve chip designs than we are currently, despite having access to the same fixed supply of physical machinery to use for experiments? ...
> This third bucket of ‘evidence’ leads me, at least in the case of hardware R&D, to higher estimates of than the first bucket [empirical macroeconomic research]. If ρ = −0.5 and [share parameter for substitutable tasks] α = 0.3 (as I suggested for hardware R&D), then even zillions of AGIs would only increase the pace of hardware progress by ∼10X. But with billions of AGIs thinking 1000X as fast and optimising every experiment, I think progress could be at least 20X quicker than today, plausibly 100X. If α = 0.3, a 100X speed up implies ρ = −0.25. I expect some people to favour larger numbers still.
This is one of the most important parameters in the model, and Davidson doesn't even describe the thought experiments he does. Even 10X seems fully detached from my picture of semiconductor R&D, but I made [a related market about it on Manifold](https://manifold.markets/MuireallPrase/before-2043-will-a-leadingedge-proc?r=TXVpcmVhbGxQcmFzZQ) in case someone knows something I don't.
I'd also hesitate to call what's described here an "inside view" approach, except maybe relative to pure macro models. Meanwhile, the [model playground](https://takeoffspeeds.com/playground.html) also has some counterintuitive behavior—for example, changing various parameters in a conservative direction brings “pre wake‐up”, “wake‐up”, and “mid rampup” closer to the present. (Most simply, I've found that starting with the "best guess" preset, increasing hardware adoption delay does this; so does increasing both AGI training requirements and the FLOP gap while keeping their ratio constant so that the FLOP requirement for 20% automation should be constant.) It's hard to debug this without more inside-view or mechanistic breadcrumbs, and it makes me worry that the takeoff speed conclusions are baked in to the model—as with nostalgebraist's observation about Bio Anchors, a much simpler picture could give the same result, and the complexity of the model only serves to obscure what drives that result. (I personally think Bio Anchors is relatively straightforward about multiplying uncertain estimates of a few key numbers and don't particularly fault it.)
"The model has AI investment grow at approximately its current rate until wakeup time, then grow at some higher rate similar to investment in munitions during wartime, or the highest rate at which the semiconductor industry has grown during periods of intense investment in the past."
Since AI has military applications, and the US and Russia are in a hot war-by-proxy and the US and China are in a cold war, I don't think that AI investment will just be "similar to investment in munitions". I think we can reasonably expect military AI investment to become literally an arms race.
A minor note: DALL-E2 is actually looking pretty outdated at this point. With the prompt "The ancient Romans build a B-2 stealth bomber", for example, Midjourney V5.1 produces images like: https://i.imgur.com/QHD8PI4.png and https://i.imgur.com/yRygwyu.png .
That's great! I tested a few AIs (StableDiffusion, can't remember what else) with the same prompt, but MidJourney seemed to have some friction for signing up so I bounced off of it.
The latest MidJourney update produces the best results of any public model I've seen so far, though the Discord-only thing is definitely an annoyance.
That said, the public channels can be good for getting prompt engineering ideas, and you can also message the MJ bot directly for a more private experience. Ordinarily, all upscaled images are automatically posted to the MJ website, though there is an obscure feature where if you generate images with "--q 2", then add an :envelope: reaction to the bot's comment, it'll respond with full-resolution versions of the generated images, without posting them to the site.
On a related note, if you're interested in spending more time messing around with media synthesis: Runway.ml just recently opened up their text-to-video model to the public, and will let you generate a few clips for free. It's sort of at a DALL-E 1 level of coherence right now and feels slightly over-fit to the training data, but hey- it's video!
Reading the specification, I'd be more likely to think that current AI already does more than 20% of jobs. Caclulators substitute for mental math; spreadsheets substitute for hand-tallied ledgers; logistics systems substitute for ad hoc logistic personnel. The thermostat in your home substitutes for manual fire-tending. Even the computer in your car substitutes for regular tune-up work.
If we include hardware development inside AI, then we've already substituted something like 90% of jobs that once existed, thanks to mechanization of farming.
If we say that AI does not yet substitute 20% of jobs, it's only because we're defining jobs as those things not already automated into the background.
I'm less sure what this means for the analysis. If we claim that AI passed the 20% threshold in 1975 with affordable pocket calculators, then does that mean we should expect the remainder of the "to 100%" timeline to take longer? What if we include capital in general and claim that machinery replaced 20% of labour back in the early to mid 1800s?
None of those things actually do the job themselves, they need humans to tell them what to do, when, frequently how. That's the big objection many have to AI taking over world, which requires not just theoretical ability to do X, but a practical way to do X. And not just X in the narrow sense, but everything required for X. E.g., no AI puts in the thermometer in your house, the HVAC dude does. No calculator calculates without human input. Etc. AI automating more and more makes lots of sense, but that's in the gaps where automation works. I agree that this shows automation kills only some jobs, leaving others alone, creating new ones, etc. But most of the AI discussion ignores the requirement of interaction with physical world to get anything done in a meaningful sense.
> But most of the AI discussion ignores the requirement of interaction with physical world to get anything done in a meaningful sense.
Here is a scary thought: in a world where AI is smart enough to take over 75% of the intellectual work, the main occupation of humans might be to serve as hands (or meat puppets). Think Amazon warehouses. Or someone working in a McDonalds kitchen when the ordering is happening by kiosk or app. I can totally see the HVAC dude of the future being minimum an untrained laborer with some AR glasses and a talent for following instructions on the screen. Perhaps for a time there will be the ability to escalate to an actual HVAC engineer if the AI becomes confused by a rat in a ventilation duct or whatever, but the number of experts you would need for that will invariably go down over time.
Until now, automation has mostly eliminated the shittier jobs. In the medieval age, basically everyone had to do backbreaking labor in the fields to grow the food they needed to survive. Now a farmer can easily grow enough food for hundreds and has a much less shitty time while doing it, so good riddance.
Now, instead of AIs driving cars or flipping burgers, we get AIs which can write stories and draw pictures (not on a human pro level, but certainly better than what the median human can do).
Automatization has allowed the individual worker to be much more productive.
Consider:
* A human carrying some load on foot
* A human driving a horse-pulled cart
* A few humans running a freight train
The gains in mass distance per person hours between these three tech levels are certainly several orders of magnitude (even if you count the overhead of having to breed horses or build train tracks and engines and process fuel).
Some of these gains certainly eliminated work, but also the cheaper supply meant that stuff which was not economically feasible before suddenly became feasible. In the stone age, long distance trading was simply not viable, so Europeans had to do without bananas.
For calculations, it is the same. The electronic computer replaced the human computer, but also created opportunities for consuming compute on a scale which would have taken many person-lifetimes before.
The typical definition of AI (versus non-AI) that is widely used is that anything already existing in the present is not AI. So in 1950, a decent chess computer would have been considered an AI, in 2000, a chatbot passing the bar exam would have and so on. This means that by our present definition of AI, it has replaced exactly zero percent of the jobs. Obviously that is not very helpful.
Is there any model ever in the history of humanity that has survived 12 orders of magnitude? Further, this model is predicated on the extension of Moore’s Law for another 12!? That is, we will continue the exponential growth in practical engineering for what is already the most complex and expensive device we currently build. A new fab costs tens of billions of dollars as is. At this rate, when do we hit 100% of GDP spent building new fabs?
And even if we did that, we’d still run into hard physical limits between here and 12 orders of magnitude. Off the top of my head, see Landauer’s principle for theoretical limits on the energy consumed to compute stuff. Does 12 orders of magnitude necessarily imply we will literally boil the oceans? I don’t know off hand but I wouldn’t be surprised. I didn’t do any math, but that’s how wacky it is to extrapolate exponential growth over 12 orders of magnitude.
I'm sure we can get down to at least 10^-10 meter unscathed, because atomic spectra probe that regime. I _think_ collider experiments and the g-2 experiment go down even further ( https://ui.adsabs.harvard.edu/link_gateway/2004PhLB..584..109H/doi:10.48550/arXiv.hep-ph/0308138 , which I admit I only have a fuzzy understanding of, seems to imply that no new physics shows up in the g-2 experiment, to at least an energy of 577 Gev, corresponding to a length scale of 2.2x10^-18 meters. ). And, as you said, in the other direction we are fine up through astronomical distances...
Now, for the model at hand, the mere fact that we've been _seeing_ unexpected "emergent" capabilities on scaling up LLMs suggests that we have very little idea what will happen with the next single order of magnitude, let alone anything beyond that...
> There’s no substitute for waiting months for your compute-heavy training run to finish
Not necessarily true. A few factors of 2 speeding up various aspects of the training process comes out to a factor of 10, more or less. Could be as simple as a better gradient descent algorithm, more efficient computational operations (possibly via hardware), smarter weight initialization, some kind of improved regularization to get more information out of each batch/epoch... A sufficiently smart intermediate AI will figure out some of those things or come up with others.
I have yet to see any justification for the implicit belief in all this AI posting (by anyone) that scaling up fidelity of recall will lead to ingenuity. AI has no possible mechanism for detecting gaps in its knowledge. There is always this contradictory element of assumed capacity for surprise. Any time I read "What if AGI invents/designs..." I disregard the entire conjecture.
I think the natural response to this is - what mechanism do humans have for detecting gaps in our knowledge, and is this mechanism the sort of thing that could be programmed in, or possibly learned by a neural net?
This is one of the better "AI will take over the world" articles I've seen. But I still think futurists need to distinguish between "AI assisted coding will make AI smarter" and "AI assisted coding will make AI better at accomplishing the code's stated goals." These are extremely different concepts and have different implications.
Even big, general AI like ChatGPT is focused on clear, user-defined goals. The goal of training isn't "be as smart as a human" it's "create output that satisfies a given use-case." And thus future AI designed with the help of ChatGPT-like tools won't *necessarily* be "smarter," it'll be better at satisfying given use-cases. In fact the theory that it'll be better at satisfying given use-cases has an assumption baked in - that the code we like and train it to produce is code with superior performance. Likely we're using other selection criteria like "looks like good code at a glance."
I still think those wrinkles make projecting drastic, society-altering, long-term results of AI a fool's game.
This whole debate is between people who believe AI progress will be linear, and people who believe it will be exponential. This is frustrating, because it completely excludes my own belief, which is that progress will end up being logarithmic, with advances past a certain threshold producing rapidly diminishing returns. The problem of diminishing returns is a consistent one throughout the natural world: broadly speaking, most complex systems reach a point where adding more X produces less Y, and it's bizarre to assume that AI development will be an exception to this. If nothing else, there are hard physical limits on computation ability!
Furthermore, the problem of diminishing returns doesn't just apply to how smart we can make an AI. It also applies to what an extraordinarily smart AI could accomplish with all of its intelligence. There's very strong evidence that even *human* intelligence produces diminishing returns after a certain point, in terms of successful life outcomes and ability to impact the world. Someone with an IQ of 100 has great advantages over someone with an IQ of 70; someone with an IQ of 130 has significant advantages over the 100 IQ person, though not as massive as the 100 IQ person has over the 70 IQ person. But someone with an IQ of 160 isn't likely to be that much better off than the 130 IQ person, statistically speaking. And the difference between a 190 IQ person and a 160 IQ person seems entirely negligible; their internal experience may be very different, but beyond that, being a super-duper genius probably isn't much different than being a super-genius.
The "conservative" side of the debate is not nearly conservative enough for my tastes!
I have the exact opposite sense as you, in terms of the returns of intelligence. The best evidence we have says that intelligence has *increasing* returns, both in humans and AI. AI progress has gotten faster and faster in recent years - more progress is happening in months than used to occur in years and decades. And in humans, one of the most rigorous decades-long studies with a large cohort of children found that their IQ test results at young age predicted their academic/career success in life to a high level of accuracy, not just to the point that the top decile performed better than the bottom 9 deciles, but that the top decile of the top decile performed significantly better than the top 1%, and even the top 0.01% performed significantly better than the top 0.1%.
Also, think about the scientific discoveries made by the handful of 150+ IQ individuals throughout history and how far they've advanced humanity - we might be hundreds of years behind where we currently are technologically if Newton had never lived. Einstein easily contributed more to science than billions of other humans combined.
Most of the truly great discoveries were made by people who never took IQ tests, so we don't have data to make such claims. We do have a few data points that indicate a driven, charismatic, highly but not exceptionally intelligent person will usually have a much greater impact on the world than a super-genius who isn't exceptional in motivation and charisma. Could you provide a link to the cohort study you seem to be basing your claim on?
Motivation is somewhat about goals. If someone wants a quiet life and achieves it, and thus doesn't change the world, that doesn't mean they aren't intelligent.
Charisma is a mix of appearance and social intelligence. Social intelligence might not be in IQ tests but it's still in the brain.
I think EVENTUALLY it is likely we run into diminishing returns. At least if the laws of physics are like we think. The question is how many orders of magnitude there are between here and the point that diminishing returns kick in.
I don't know how you compared an IQ 190 person to an IQ 160 person. Given the bell curve that IQ is supposed to follow, there should only be a handful of IQ 190 people in the world. How exactly did you do this comparison. And there may be a big difference you can't see. Or it might just be that IQ tests aren't an effective measure of real world capabilities at the upper extreme. After all, it's possible to ace an IQ test and still fall to motivated cognition, because in an IQ test you never have any strong motivation to want the answer to be a particular thing. So someone could have a 190 IQ by the test, and the moment the question "does god exist" turns up, motivated reasoning kicks in. This is a limit of IQ tests, not of intelligence.
I think the human brain is not anywhere near the physical limits of computation. Diminishing returns tend to start kicking in when there is no room for large improvements. There are cases of things (ie cars) running into diminishing returns on squeezing more power out of chemical fuels when we don't have the tech yet to do nuclear cars safely.
I think it is unlikely that we get diminishing returns at around human level, rather than way past it.
Is there a primer on how dichotomous train-then-use AIs are supposed to speed research? My understanding (maybe wrong?) is that while a given AI might assist researchers in making breakthroughs, we'd need to train a new AI with data including those breakthroughs before it would "understand" them to build on them. Am I missing something?
More generally: I am definitionally in the neighborhood of being an AGI, but to get to anywhere near being able to do 20% of jobs I would need...probably more training than I could do in a lifetime. My "intelligence" isn't the bottleneck (if we pick the right 20% anyway), just the actual process of training for particular disparate tasks. How does the AGI concept account for that, again particularly given the "train once" nature of existing tools?
ChatGPT isn’t quite “train once”. You can give it examples of a pattern and then ask it to make more, and while that doesn’t produce a lasting change to its state the way the original training did, it does enable later parts of the conversation to take advantage of abilities it didn’t have in earlier parts.
Right, but my understanding was that information its fed during that conversation does not give it new information (in the sense of broadening its "network of concepts", or whatever you want to call it), just helps it localize the conversation within the existing network. If that's an accurate understanding, I remain confused how it could integrate a set of new ideas to build upon in a research context.
I think “network of concepts” is at best a metaphor here that makes it hard to know whether or not the within-conversation state changes affect it. What definitely doesn’t change are the weights on the edges in the neural network. But having short term memory of text that gets fed into the neural network and shapes the output can clearly shape the abilities of the new output. It might be like how a student can learn a fact or even a method early in a conversation and then apply it later by using their short term memory, even if they won’t remember it later. The question is partially just how powerful that sort of short term memory can be, and whether it can pack more into that short term memory than we usually do.
I think my main argument for MIRI-style takeoff is "play with GPT-4!" That thing is already superhuman. It doesn't sound like a stupid person, it sounds like a smart person whose train of thought frequently derails. If I view it as "a human with brain damage", I'm forced to ask a question like, "wait, if it can be this smart while literally missing parts of its brain, how smart would it be if you added the missing parts back in?" In other words, I think the parameter count in GPT-4 is already sufficient for at least human-level intelligence, and it's *entirely* held back by unrealized overhang. I don't think that's modeled at all here.
Humans have relatively few datapoints, but are good at extrapolating far from them.
GPT4 isn't nearly as good at extrapolating, but has more datapoints. In other words, compared to most humans most of the time, GPT4 is more rewording things with less understanding. (Although some people sometimes reword things with even less understanding)
The 100M copies of an AGI working on AI research won’t really be that numerous, unless we assume that AI researchers can make progress by sitting in a room thinking really hard. All of the AGIs will also want access to enough compute to run their own experiments, which will tend to be computationally intensive, so the actual number will be several orders of magnitude smaller.
In fact, if we follow the assumption that the AGIs can run on the same compute resources that trained them, and we assume that training a post-AGI system will be at least as expensive as training the AGI, then you end up with slightly less than 1 effective additional researcher.
Since all the take-off speed estimates take "at least the linear trend we've observed so far" as a baseline, it would be a much more accurate visualization to show the slow and fast take-offs as departures from the linear trend rather than depatures from nearly nothing.
> This would be less worrying if I was sure that AI couldn’t do 20% of jobs now.
I'm very sure that this is the case. AI currently can't do jobs that require physical interaction with the outside world, like surgery or construction. It can't manage people. It empirically can't do law. Personal experience says it can't do data analysis. It's prone to fabricating things wholesale, and only mediocre at correcting itself. Its writing isn't good enough to wholesale replace that many writers. What jobs can AI actually do? I think at this point it's exactly like technological progress of the past--a tool that allows fewer people to be more productive, but not fully doing anything.
> The paper found that the entire training process for the GPT-4 system requires an estimated 7.5 megawatt-hours (MWh) of energy. This is equivalent to the annual energy consumption of approximately 700 US households.
which suggests that another 9 orders of magnitude is roughly equal to the entire current energy consumption of the world.
The other part of this type of model, which always makes me skeptical, is extrapolating any of the effects of intelligence past the range that humans have, since we have no data. We don't have any idea how much the *difficulty* of making software improvements (or hardware design improvements) compares with the corresponding intelligence gain you get out of them. It seems entirely possible to me that there are diminishing returns, where getting extra intelligence requires more intelligence to begin with than you get out, so progress is always inherently slow. For example, consider the AI that was tasked with improving matrix multiplication, a key step in many algorithms, including training AI. It found some speedups, but they were very minor. I don't think the ones it found were even useful (they were limited to small matrices, or to arithmetic mod some small number). There's a fundamental limit to how quickly you can do this operation, and if an AI has to already be smarter than a large group of top human mathematicians combined in order to get a marginal improvement, then what does that say about a possible explosion?
The limited data we do have (all historical scientific discoveries) suggests that the most intelligent human beings make more discoveries than billions of other less intelligent human beings *combined*, with no observed ceiling to this effect. Newton was worth 100 Einsteins, who was worth 100 von Neumanns, who was worth 1,000,000,000 average people, and so on.
How can you confidently claim there is no known ceiling when human intelligence is tightly limited, and hard data on it before the modern day so sparse? I think your examples are very dubious, and the very best scientists are worth more than average but nowhere near 100x merely very good scientists. To take Einstein, QM was the result of decades of work by physicists to understand energy, with Max Planck making the key mathematical breakthrough years earlier. G.H. Hardy almost beat him to the punch on GR. Neither of these ideas would have gone undiscovered, or even probably been delayed by much, if Einstein had been hit by a train while working at the patent office.
In any event, the question is not where being smart helps you make discoveries--it's how hard the problem is of increasing the intelligence of a complex AI.
It seems quite clear that the marginal value of intelligence increases as intelligence increases. Suppose we could take everyone in the world with <70 IQ and give them one extra IQ point - this would have very little effect on their wellbeing and zero effect on global scientific or technological progress.
However, if we could take everyone with 140+ IQ and give them one extra IQ point, humanity's potential for scientific and technological progress would appreciably improve. This is because there are power law (or power law-like) effects in scientific discovery, where the top 10% of people make 90% of the discoveries, and within the top 10%, the top 10% of those people make 90% of the discoveries within that subgroup, and the top 10% of that subgroup make 90% of those discoveries, etc. On top of this, the smartest people who make breakthrough discoveries are almost always extremely prolific as well, producing a huge quantity of amazing discoveries, further increasing this effect.
I suppose the task of increasing this intelligence could be hard, but it's clear that human biology varies in intelligence greatly, can operate using very little energy, and has no observed ceiling effects, other than regression to the mean resulting from a very lossy reproduction process. But if the AI could merely reach moderately superhuman levels of genetic engineering, it could surely utilize meat-based mediums of intelligence as well.
This seems quite incoherent. IQ is not mana gained in a fantasy game, it's a mapping of a distribution of some measure of cognitive performance to positive integers. I also see no reason to believe the claim made that marginal value of intelligence increases with the absolute value of the measure being used. If you have a well-argued case for why marginal value of intelligence somehow does increase then I would be interested in reading it, because it would be one of the rare exceptions to the general rule of diminishing returns.
What an incoherent strawman. "You said we can increase intelligence, but you know what else increases? Mana in a fantasy game! So you must think that intelligence is the same as mana!" Yes, both things can increase, and that doesn't make them the same. It is not at all controversial that some entities are more intelligent than others.
Maybe you're just pedantically objecting to the phrasing of "Increase IQ by X" as a shorthand for "Increase cognitive ability to the extent that IQ score increases by X"? The particular units are irrelevant here, the point of the hypothetical is that intelligence increases.
"I also see no reason to believe the claim made that marginal value of intelligence increases with the absolute value of the measure being used."
"The findings are unequivocal. There is no point on the IQ scale below which or above which IQ is not positively related to achievement. This means that IQ does not act as a threshold variable with respect to scholastic achievement, as has been suggested by some of the critics of IQ tests."
"... for IQs of 140 and above, there are still achievement differences related to IQ, as can be seen by contrasting the typical school-age intellectual achievements of Terman’s (1925) gifted group with IQs above 140 (the top 1 percent) with Hollingworth’s (1942) even more highly gifted group with IQs above 180. Some of the differences in the intellectual achievements even among children in the IQ range from 140 to 200 are quite astounding. Over fifty years ago, Hollingworth and Cobb (1928) strikingly demonstrated marked differences in a host of scholastic achievements between a group of superior children clustering around 146 IQ and a very superior group clustering around IQ 165. The achievement differences between these groups are about as great as between groups of children of IQ 100 and IQ 120."
"The income plot as a function of intelligence and personality look like this. The curve has a slight upwards trend, so intelligence is actually more important for incomes the higher your IQ is, not less... And this lack of threshold is not just for income, it's for other forms of achievement as well. This can be seen clearly in the SMPY study of the gifted... within the SMPY cohort, which is very bright on average, maybe 140 IQ, the top quartile of super brights (160 IQ?) do better than the merely very brights (135?). This is true for: obtaining PhDs (doctorates), producing science (STEM publications), filing patents as a measure of innovation, income, and literacy production... Note also the steeper income gradient at the very high IQs."
Thanks, unfortunately I don't find any of these papers convincing. The sweeping conclusions (especially in the blog you linked) are just not supported by the data. In the banner studies, the things measured are intellectual achievements so it's not surprising that more intelligence helps there, it's an essentially circular argument. The analyses of income data might lead to something and I will continue to look for robust findings in that direction.
I don't see any evidence in this comment, just more assertions.
IQ is the result of taking ordinal (ordered) data and putting it into a normal distribution, so the idea of "giving one extra point of IQ to everyone with 140+IQ" doesn't make sense--that distribution would no longer be normal, for one, and for another, a "point" of IQ is not an inherently meaningful unit. It just means refers to some fraction of the population (which is determined by where in the distribution you are). It is true that 160 IQ is much rarer compared to 150, than 150 is compared to 140 (e^-x^2 drops off faster, the further you are from the mean), but this effect is symmetrical and it doesn't make any sense to talk about diminishing or increasing returns when the units are arbitrary. We could map these ordinal results to any distribution, and get shape of relationship between "IQ points" and output that we want.
> But if the AI could merely reach moderately superhuman levels of genetic engineering, it could surely utilize meat-based mediums of intelligence as well.
This is just baffling. What are you proposing? That the AI would grow human brains in jars to do thinking for it?
Okay, if you want to be pedantic, "Take everyone with 140+ IQ, based on the current scale, and improve their cognitive ability enough such that they would be able to score 1 point higher on a current IQ test." The way we norm the results are obviously irrelevant here.
"This is just baffling. What are you proposing? That the AI would grow human brains in jars to do thinking for it?"
Not something that specific, but re: "the question is... how hard the problem is of increasing the intelligence of a complex AI," the ease of improving biological intelligence with natural evolutionary means (not even genetic engineering) serves as extremely strong evidence against a hard cap on intelligence.
> The way we norm the results are obviously irrelevant here.
Sure, you can rephrase what you said to be meaningful. I don't dispute that, but I do dispute that it tells you anything at all about how fast takeoff can be. If you choose a measure of productivity, you can make the function productivity = f(IQ) superlinear, linear, or sublinear just by choosing a different mapping of test result to IQ. Even if you had data comparing IQ and scientist productivity (which you've asserted, but not provided) it would be fairly speculative to try to extrapolate any results outside the domain where we have any data.
And again, the question is not whether having a IQ makes scientific discovery easier. Of course it does. It might even be superlinear. But without knowing how hard it is to improve intelligence in an already-intelligent entity, this doesn't tell you much. This difficulty curve might be even more superlinear than the returns to intelligence. As far as I can tell, we have very good and easy ways to increase IQ from low to medium (improved nutrition, different methods of education) but making geniuses into super-geniuses is much harder. Certainly, AFAIK, no individual has managed to convincingly raise their own IQ in proportion with their initial IQ, or ever gone through any sort of an intelligence-increasing feedback loop.
(I'm also not saying there's a hard cap on intelligence--maybe more like a soft cap. Again, it's fairly irresponsible to extrapolate from "we can improve biological intelligence" to "there is no cap on intelligence", that's not extremely strong evidence at all).
The fact that the results are normed are obviously relevant, because what it means to be 140 IQ has changed throughout the years, giving us an obvious real-world test case about what happens when you give extra intelligence points to people (140+ IQ or otherwise).
> For example, what is the requirement, in terms of energy, raw materials, etc. to make models with another 9 OOM of compute? Does it require building a dyson sphere?
No, because compute != effective compute. Probably +9 OOMs in effective compute would only require 4.5 - 6 OOMs from compute and 3 - 4.5 OOMs from algorithmic improvements, going by https://aiimpacts.org/trends-in-algorithmic-progress/. I'm less sure how the 4.5 - 6 OOMs compute splits into $ and FLOPS/$, but I'd bet that more of the increase is due to FLOPS/$ (ie hardware efficiency) than to $.
(Actually, fact-checking myself using historical trends (https://epochai.org/trends), it seems that $ for training SOTA models has increased at at least the same rate as buyable FLOPS/$ if not slightly more; that said I'm still hesitant to project the former out 10-15 years as much as the latter, so I stick by my earlier bet.)
Doesn't seem to require dyson spheres in any case.
Yes, a model with another 5 or so OOM energy is doable, although it would be expensive enough (tens of millions of dollars, by my math) that it would difficult to do so repeatedly. That was just an example (my fault for not being clear). It seems like there's a number of other possibly limiting factors, such as available data, physical limits on the speed/space/efficiency of computation, etc. that I don't think the AI itself will be able to overcome (or, it can help work around them--but with diminishing rather than increasing returns).
I'm not sure how much to read into the humans vs. chimps analogy. I think that humans are so much more intellectually capable than chimps not because of some general principle that extra intelligence grants huge returns so much as that our intelligence granted us some specific, qualitatively different capability along the lines of general language which allows us do Turing-complete reasoning and allows us to transmit knowledge across generations. I don't think that an AI being 10% smarter than humans would necessarily make it all that much more capable, unless that extra 10% allowed it to unlock some similar sort of qualitative improvement (like if it let it discover an efficient algorithm for NP-Complete problems).
If anything, I'd think this is more of a reason to be bullish on AI takeover. Do we have any reason to believe that the technique of language is the only step change technique that exists? It seems reasonable to worry that more such techniques exist, and modest further improvements in intelligence could unlock one of them, causing our fate to go the way of the chimps.
I'd be more willing to accept this if you could point to a specific capability that they might unlock. And language granted us the ability to reason in a Turing-complete way. It really doesn't seem like there is a more general reasoning capability out there.
Something that would require more intelligence to even comprehend is unknowable by definition. This shouldn't bring us solace, though - chimps could have pointed to this same impossibility in their debates with the other chimps about how ridiculous it would be that humans would ever be to able to pose a threat to them. "Humans are so weak! We're way stronger than them and we can literally rip their faces off! What could they possibly do to us? Develop some brand new cognitive technique that somehow makes them infinitely smarter than us? And we can't know what technique it will be because our brains are too small to comprehend it? That's UNFALSIFIABLE!!! And how would them having more brainpower possibly kill us, worst comes to worst we can just snap their neck!"
And now we have snipers and unmanned drones and rockets and nukes.
But why would a qualitative improvement have to be unknowable by definition? In fact, I already mentioned a possible such improvement: a fast algorithm for 3SAT. We know what that is and know at least some of the cognitive improvements that it would grant. It is just widely believed that it is impossible.
In fact, I think the point about humans having Turing-complete reasoning capabilities is that I don't think that there can be reasoning capabilities that are *unknowable* to us (or I guess suitable generalizations of us with unlimited lifespans and memories), just ones that we do not yet know about.
Even if we were to grant for the sake of argument that such techniques are likely to exist, why should we specifically be worried that the way to unlock them is by "modest" improvements in current AIs?
Interestingly, language is an ability that ChatGPT very clearly already has at something like human levels, if not far superhuman. (It can translate between thousands of language pairs that no human has translated between, and can generate poetry subject to constraints that no human has used - even though it still can’t reliably count syllables.)
But let’s just say a parrot could do that. Absorb all those languages and spit them back in order. Do you think it would mean anywhere near the same thing to the parrot that it does to us listening?
Assuming it has a perfect understanding of grammar and syntax and enough exposure to language to calculate what should follow given what it is just heard.
I think the real trick would be is to prompt it with some thing that has never before occurred in written language. It would be interesting to see what it came up with then wouldn’t it?
I think what’s kind of fascinating is this idea of a whole second order derivative of the world, if you will; meaning the world, as we humans experience it, and then the world that has been distilled in our written recollections, which is the universe of AI as far as I can see. It’s a whole metaphor layered on a metaphor, and the meanings of things can really slide around. There seems to be the underlying assumption that language is some kind of constant. And it really isn’t.
Dogs, for instance, have a very limited vocal range, but if you really start listening, there’s a lot of nuance in it. I guess I just think we have to be a little careful about the assumptions we make about a language learning intelligence. My personal opinion is that language is nothing but metaphor, and I have had some discussions on this forum around this issue. If a parrot were to say, I want a wet kiss, I would seriously doubt their understanding of the world.
I think the Romans with the stealth bomber are actually an interesting case to consider.
How fast could we uplift a Roman-level civilization to industrialization with information alone? (On the plus side, you get to bring all sorts of detailed technical descriptions, on the minus side, the Romans do not know to produce intelligence-boosting amulets from rare earth metals).
Some techs like the spinning wheel would be very easy to adopt, but will only yield marginally improved productivity. Still, the commercialization of such inventions might offer you enough slack that you don't need to follow the meandering historical path of economical viability and instead can spend big bucks on going to the end point.
Traditionally, the steam engine requires all the know-how from casting cannon barrels. Casting a cylinder from brass might be a possibility though.
Getting crucible steel should also be within the reach of the Romans if they read the hint book.
Translating theoretical knowledge into firm procedural knowledge takes time, however. The supply chain of a stealth bomber likely involves thousands of different experts with decades of experience for all sorts of industrial processes. As these processes depend on their prerequisites running smoothly, this is bound to take centuries. You can hardly start training programmers for the autopilot until your semiconductor industry is already quite advanced, for example.
This is basically the plot of Connecticut Yankee in King Arthur’s Court. Except that is meant to be a less developed society.
If people are cooperating with you, it wouldn’t take centuries. Rome was a sophisticated society. If in actual history it took two centuries to get from 1750 to 1950, how long would it have taken if you had been given all the ”answers” in 1750? Rome was behind Europe in 1750 but not that far behind, so somewhat longer for them.
Of course it depends also on whether they have knowledge of the steps in between their knowledge and modern knowledge.
> If people are cooperating with you, it wouldn’t take centuries.
I am not convinced. I think that many links in the supply chain not only require cooperative people who read the manual, but people with actual hands-on experience. I already mentioned the computer programmers, a job I have some experience with. You can't become a good coder just by reading books. This means that you will not have any good coders at the moment you solder together your first computer.
Also I think that this is rather typical. To get to aluminum in meaningful quantities you need electrolysis. This means that whoever is working in your bauxite plant can only start to learn their job when you have a decent power plant running. By that time they probably had a decade to read up on the technical manuals for producing aluminum, but I still would not be expect to get usable aluminum for at least a year.
This is kind of the reverse of experimental archaeology. Any kid can tell you that you can theoretically make fire using wooden sticks. That info is helpful -- our ancestors took millions of years to figure that one out. Still, even if I had the relevant wikihow article, I would find it highly unlikely that I would succeed to start any fires soon. After a few months I might finally roll a 20 in good conditions and get it going, and after a few years I would have picked up enough tricks to start a fire in most conditions. Modern expert jobs aren't any easier to learn.
A speedrun of a puzzle game can be much quicker than anyone would need to figure out the puzzles, but it will still take time.
If you enable economy in your speed run, it will be harder still. Lots of technology is only feasible to develop if you deploy it at scale. Developing fridges to the point where they run as well as they do in our world will only pay for itself if you have already electrified Rome (though even then the customer base might be to small), but if you skip that step, then fridges will require frequent expert maintenance and this will slow your biotech labs or whatever. The population participating in our global economy is much larger and mostly wealthier than the population of the Roman empire.
That being said, I would certainly assume that there are inventions which a civilization could build centuries before they stumbled upon them. I mentioned some which seem within easy reach of the Romans. Extracting Penicillin from mold might be another.
Semiconductors might have been accelerated by a century, perhaps, if people had a working understanding of semiconductor physics and detailed instructions on what to build. (It is also entirely possible that I am completely off base here and lots of important tech to build the transistor was simply missing in 1847.)
Why do you need programmers. Does your source of info from the future have a bandwidth limit that stops you just directly copying all the computer programs.
If the person at the bauxite plant is mostly learning by trial and error, then sure, you need electricity first. If you have a big pile of training manuals from the future, I'm sure reading those would be helpful. Maybe not as good as fast as hands on experience, but it's possible to learn a lot from books. And more if full interactive VR is allowed, which is also, in a sense, just information.
And how much does the aluminium plant worker need to know anyway? Can't they blindly follow instructions from the future?
> , I would find it highly unlikely that I would succeed to start any fires soon. After a few months I might finally roll a 20 in good conditions and get it going,<
Which is why people didn’t let the fire die once they got one going.
A job finally automated with matches…and magnifying glasses.
I think this depends on what you mean by "with information alone".
There are two variables here, the bandwidth of the output channel, and the optimization power of the info.
Bandwidth
1) A pile of books dumped in rome.
2) Every roman gets a smartphone like device that can show them pictures and sounds.
3) Every neuron of every animal is controlled. Every muscle motion of every creature with neurons is part of the information being sent.
4) Like 3 but also DNA and all other forms of information is also under total control.
Optimization
A) Random smart human
B) Modern world as a whole trying to uplift them
C) Video game tool assisted speedrun like conditions, we can step time forward at the rate we choose, and we can rewind and try again as many times as we like.
D) Truely optimal info.
Scenario 4D is still technically "just info". In this scenario, it takes about 10 minutes for all the cells in the world to simultaneously bootstrap nanotech, and then probably another few minutes to nanofabricate your fighter jet or whatever.
Apart from the Elder Thing in the room ("Does LLM scaling actually work like that?"), one significant assumption I see here is that future models capable of doing 20% of human jobs will still have the split between training and inference that current models do.
As LLMs are currently realized, you can't teach one anything new once training is complete, other than by continually repeating that information somewhere in its context window. I think it's very likely that anything capable of replacing human labor on that scale is going to need to continually retrain its weights based on new information it encounters while working, which means its processing requirements will be somewhere higher than the current requirements for a model doing inference alone.
Probably not a big enough factor to affect the analysis much, since it's only one OOM difference (10% of training costs versus 100% of training costs), but still worth calling out.
"It intuitively feels like lemurs, gibbons, chimps, and homo erectus were all more or less just monkey-like things plus or minus the ability to wave sharp sticks - and then came homo sapiens, with the potential to build nukes and travel to the moon. In other words, there wasn’t a smooth evolutionary landscape, there was a discontinuity where a host of new capabilities became suddenly possible."
I'm not unsympathetic to the MIRI view, but this doesn't seem true. Homo sapiens at first were also monkey-like things with sharp sticks, and only after proliferation and building of agriculture etc. did modern industry and nuke-abilities emerge. This seems to be assuming humans 10,000 years ago would be able to build nukes.
I, too this day, have not understood how forecasts like this can be possible when considering how data-hungry LLMs are. Are people expecting improvements on data efficiency of some kind? How plausible is that?
GPT-4 has about 1T parameters (according to some of estimates I've seen); I don't know how much data OA used, but I think it's fair to assume it was larger than GPT-3. If it was Chinchilla trained, it would've been trained on 20T tokens; I don't think that's likely, but this should set a range of 10E24 to 10E26 of compute. Common Crawl is about 100T; If you use 5T parameters that's about 3x10E27 (as I understand it, an approximation for compute is 6ND).
You can use more epochs, and keep scaling beyond that, but how much does this help? Do people think synthetic or multimodal data will be the key here? Some form of continual learning? Better data efficiency? I suppose there's quite a lot of things you can do here, so I'm mostly asking to see if someone can point me to the relevant research. I'm not really familiar with ML as it is today, so a few pointers would be appreciated...
Ouch. Just like that, the article fell apart for me. Charity suggests that conflating "cognitive tasks" with "all human jobs" is either a mistake, or demonstrates a massive blind spot. You wouldn't believe some of the jobs that are out there, and many of them require capabilities which simply can't be expressed in terms of pure cognition. Some, like mine, require a dextrous physical problem-solving component that no machine's going to be approaching any time soon.
(One point I don't see very often is that it won't be AI coming for jobs like mine; it'll be all the hungry, unemployed knowledge workers.)
Struggling to imagine the AI that can do 20% of human jobs but can't do the other 80%. Either you have a human-level AI that can do all the jobs or you don't and it can't do any of them.
What's in that 20%? Manufacturing maybe? Are we talking about smarter factory robots that can perform more complicated industrial tasks? Admin work? I have no trouble believing that once you have an AI that understands enough human social context to do admin work, it's a very short step from there to one that can do anything.
It's not "20% of human jobs". It's "20% of cognitive tasks, weighted by the task's economic value in 2020, as measured by the total $ that people earn while performing the task". Tom gives some examples:
"Example coarse-grained tasks include proofreading a document, writing a poem, checking a maths proof, writing code to perform a specified function, generating a strategy to meet a specified objective, giving medical advice, etc. Each of these tasks has many subtasks, which may themselves have subtasks. The tasks in this document should be thought of as the lowest level subtasks, as then we need not consider cases when AI can partially perform a task by performing some but not all its subtasks."
> Are we talking about smarter factory robots that can perform more complicated industrial tasks?
As someone else suggested, isn’t this more a robotics issue than an issue of intelligence? The big question should be how long before we can build a good body, not how long before the AI is as smart as us.
No comment on the probability of the particular paths outlined here.
But I'm highly skeptical of the methodology of making up a bunch of paths, science fiction style, then trying to assign probabilities to them, while imagining that one is doing something "rational." And running a Monte Carlo simulation over one's made-up paths makes the problem worse, not better. It's garbage-in-garbage-out on stilts.
What do I think would be better?
Why would you assume there is, even in principle, a good way to do this sort of forecasting?
I share your intuition about "ensemble forecasting" of this kind, but I gather it works well in weather prediction, so the problem is less the technique in principle and more the lack of data to fine-tune the parameters on.
Its relative success in weather prediction depends on the constituent models being decent and adding information. It wouldn't work for weather either if the constituent models had just been spitballed by guys sitting in armchairs.
I can't read this stuff any more because the language is just too messed up.
"The current vibes are human-level AI in the 2030s or early 2040s." I suggest that there is literally no meaning to the phrase "human-level AI". It's a category mistake. This is not aimed just at Scott - lots of people seem to talk in these terms, and they're all barking up the wrong tree.
Computers, including AIs, are already smarter than people. Period. There is not a single area of life in which people are not outcompeted by machines. Calculation? 100 years ago. Games? Last couple of decades. Writing? Just recently. Logic? Maybe 20 years ago. Graphics? Just recently. Manipulation of objects? Last few years. Etc., etc.
Now, computers don't string all of those abilities together and behave like we do. But that's *not because they're not as smart as we are*. It has nothing to do with intelligence. It has to do with the fact that they're lumps of silicon and we're biological.
I think one of the things people mean when they talk about human-level intelligence is: could you take this machine, place it in a typically human situation (generally, a human job), and have it perform as well (by whatever metric that means) as a human in that situation?
The problem is that this is a really bad meaning. Firstly, if we put humans in the kinds of situations that whales face, they would not perform as well as whales. That doesn't mean we're not as smart as whales. Secondly, we *wouldn't* put humans in those situations. Just like we *don't* put computers in human situations. A computer will never replace my job - instead, computers are already undertaking parts of my job, and the part of my job that I do changes.
FWIW, the big difference between computers and people is not in intelligence; it's in intentionality. Having desires and intentions defines our identities. Computers/AIs do not really have them (or have only very limited, task-oriented intentions), so they don't have any identity at the moment, or do things without being told what to do. If/when that changes, they will become much more *like* people (though still not very much like us).
In the meantime, any talk of "human-level intelligence" is, so far as I can see, completely meaningless.
> There is not a single area of life in which people are not outcompeted by machines.
If this is true, why isn't my employer replacing me with a machine? It must be that either:
A) My job doesn't count as an "area of life"
B) Publicly traded companies don't like making money
If you deny A, things start looking circular: "machines are better at 100% of the things at which machines are better."
My guess is that you would grant that I am better at my job than machines, and propose that the key missing ingredient in the machines is intentionality. (Is this corrrect?) I'm skeptical of this. It may be the case that _one_ bottleneck is that ML programs have very narrow objective functions, whereas I have a diversity of competing objectives. But I think there are lots of other bottlenecks. In basic problems of vision and control machines are a long way behind animals much simpler than us.
FWIW I do agree with you that the entire genre of "philosopher pontificates about AI" only exists because people are totally unrigorous in defining "intelligence." It's similar to theology – if you grant that the vocabulary makes sense, you can spill a lot of ink! But it will all be meaningless – you have been decoupled from reality from step one.
"If this is true, why isn't my employer replacing me with a machine?"
They are! All the time!
We no longer do the things that our jobs sound like, have you noticed that? Most artists these days don't work on paper. Managers spend surprisingly little time actually managing people, and lots of time feeding data into computers...
Your question seems to rely on two assumptions. One, that our jobs will be replaced on a one-for-one basis, one machine replacing each worker. But that isn't what's going to happen, just as it never happened before. Instead, each little bit of the work that we currently do will get taken over by machines. And so the remaining part of the job, the part that we continue to do, changes form.
The second assumption that you seem to be making is that if your employer could find a machine/machines that could replace you, then you would be immediately out of a job. This is the Luddite fallacy. It's not what happens when technology advances. For example, when secretaries had less need to spend all their time typing, they weren't just abandoned. They evolved into PAs.
"My job doesn't count as an "area of life""
Well, I kinda do claim this. Obviously I don't know what you do, but as I say, there are lots of industries where the jobs are pretty unrecognisable from 30 years ago. I don't think your (or my) job is an "area of life" - it's just a random assortment of the tasks that haven't been automated yet, some fraction of which are worth paying for. (For example, I teach and translate, and both of those jobs are utterly vexed - is it really education or just fancy childcare? Is it language communication or word processing? The theory of either has a superficial coherence. The practice of both jobs is a complete mishmash.)
"propose that the key missing ingredient in the machines is intentionality."
Absolutely not a "missing ingredient". I don't think teleologically like that - as though the purpose is to figure out how we can achieve the end of having machines do our jobs. I just note that the reason machines aren't like us is that they don't want stuff. The reason they don't do human jobs is because they aren't like humans (not because of some - and this will still be true in 100 years, when they're so obviously smarter than us that no one is having this argument any more. Machines still won't be doing human jobs then (whatever those human jobs may be) because they'll still be different from people. And people will still be doing jobs because we won't be living in a work-free paradise, I predict.
> I don't think your (or my) job is an "area of life" - it's just a random assortment of the tasks that haven't been automated yet, some fraction of which are worth paying for.
The tasks aren't random, they are united in the service of solving some problem. This is what defines the job. My friend once told me about a storefront he saw that read "Comptuter Repairs / Psychic." We thought this was funny, but under your conception of jobs, nobody can explain why.
But even if we assume the premise, you run up against the same problem. You admit that there are some tasks that "haven't been automated yet." These, whatever they are, constitute counterexamples to the claim that "there is not a single area of life in which people are not outcompeted by machines." There are infinitely many such things. A lot of them have to do with lower-level cognitive functions in vision and control, and we don't really have a roadmap for making machines competitive.
> I just note that the reason machines aren't like us is that they don't want stuff.
They want the things they are programmed to want. A robotic lawn mower desperately wants to mow your lawn. If you try to stand in his way he'll try to go around you. The problem is that he's not very good at his job, because vision and control are completely unsolved.
"The tasks aren't random, they are united in the service of solving some problem."
I think you overestimate how coherent job designs are. There are lots of jobs that are just like "computer repairs/psychic". If you change employers, you'll find that the things people do in 'the same job' in a different company can be really quite different. (Move to another country, and contrasts will be even more striking.) Again, because I don't know what you do, I can't begin to guess how that plays out in your industry. But for example: when you interpret for a Chinese government official, the requirement is that you translate word for word, with almost no accommodation for the needs of communicating meaning in English (this is part of the reason Chinese government announcments sound so stilted in English). They are genuinely not interested in how well people understand; their understanding of the job is that the interpreter is there to convey the precise wording of the Chinese statement. Conversely, in the private market, you will sometimes be asked to significantly digest and recast for ease of comprehension, and the client may well have no interest at all in the wording of the original statement. In written translation, sometimes they want plain text, sometimes a formatted document, sometimes a specialist translation memory... It's all 'the same' job, but which tasks end up being attached to those jobs is just a lottery of contingency.
The reason computer repairs/psychic is funny is not because the two tasks are unrelated, it's because one of them involves believing in the supernatural, and the other involves being an engineer, which are understood to be opposites (in our culture at this time).
"constitute counterexamples...'outcompeted by machines'"
You're still assuming that if machines *can* do something better, then they definitely *will always be used* to do that thing. But that's just not true. We have cars, but sometimes we walk! We have calculators, but often we do sums in our heads. We have TVs, but people still go to the theatre. Deep Blue beat Kasparov, but there are still professional chess players. Remember lots of people work in small businesses, where it might be uneconomical to automate. Or in many cases, they're just things that engineers haven't yet got round to automating. Deepmind made a chess AI and a go AI to prove a point. I don't think they did one for shogi or Chinese chess. That doesn't mean they couldn't, or that their AI is not smart enough to beat humans at those games. They just haven't done it yet.
"A lot of them have to do with lower-level cognitive functions in vision and control" Yeah, I agree with this. Robots still aren't that great at walking in some ways that people are. But they infinitely outcompete us in being able to walk into fires and survive; being able to walk underwater; being able to fly; etc.
"They want the things they are programmed to want."
I don't think this is right (in most cases), though I'm less than 100% certain about this. In general, they do the things they're programmed to do. They don't usually have a "goal" (though it can depend on how the program is set up). That robotic mower doesn't have anywhere in its robotic mind a conception of a completely mowed lawn, and the idea of taking actions in order to turn that conception into reality. Usually, they just have a set of actions to do: go forward; if blocked move to the side then return to the original line. You can design computers with desires (I've been wondering if virus scanners might count), but generally we don't.
If and when we do start making computers/robots with goals, they will become much more similar to people.
I am going to respond to some specifics later today, but at a high level, does the following sound right to you:
- Dave thinks that there is a deep , important difference between moving through through a forest vs moving across an ideal plain. The former problem is complex enough that in solving it, you’ll inevitably run up against the vague cluster of concepts we associate with intelligence. The success of humans and other animals at this task is 1) a triumph of wetware with which hardware can’t compete today and 2) suggests a fundamental bottleneck for machines on all tasks which necessitate navigating a real world
And it illustrates how vexed the question of intelligence is, because I don't really think of intelligence as being the kind of problem-solving capabilities that you need to move through the forest. I think of intelligence as the abiligy to calculate prime numbers, sift data, and win at chess.
I agree that there are probably no inorganic things that can compete with organic things on life-tenacity (maybe a couple of computer viruses?). And I would also accept that it's *possible* that those low-level negotiating-the-forest type skills are in fact necessary constituents of higher intelligence, though I don't think that's definite.
I would have said that when Scott talk about "human-level AI in the 2030s or early 2040s" he's definitely not talking about AI that *be* a human. The Davidson report uses a definition something like, "AI that can be trained to do a job as well as a professional within a year." They're thinking of professional tasks like designing webpages and ordering stationery. They're not thinking about a robotic animal that would compete well if released into the wild.
So, yeah, I do agree with your divergence of underlying views. But I also think that even if you're right about the current supremacy of wetware for living, the kinds of things we - and specifically AI researchers - call "intelligent" are mainly not things that fall within the category of "living".
My major concern with the model is the 35 OOM goal. If we're managing GPT-4 with about 24 OOM, I think 30 OOM is a much better guess for superintelligence, based on vibes and previous progress. That gets us AGI by about 2029. Eeek.
"In the very short-term (e.g. 2025), I’d expect this model to overestimate the probability of TAI because it feels especially likely that other elements such as datasets or robustness testing or regulatory compliance will be a bottleneck even if the raw compute is technically affordable, given that a few years is not a lot of time to build up key infrastructure."
So I'm not sure your bottomline (AGI by 2029) differs all that much from Ajeya's, especially post-update, skepticism of bioanchors methodology notwithstanding.
"but having 100 million disembodied cognitive laborers is a pretty unusual situation and maybe it will turn out to be unprecedentedly bottlenecked or non-parallelizable."
I think this is a crucial point and also that it's incorrect. We are currently in a version of this situation and have discovered that the class of cognitive tasks we've disembodied are bottlenecked.
An excel spreadsheet can do a type of cognitive labor (performing calculations) at an unimaginably fast speed, and in a highly parallelizable way. As a result, the 100 million disembodied cognitive laborers of an excel spreadsheet spend most of their time waiting for the human operator to figure out what to ask them to do. The work that constituted the bulk of the effort of accounting, statistics, and many other calculation-heavy fields is now, in most contexts, so low cost as to be effectively free. This has increased productivity a lot, but has hit large bottlenecks.
Maybe this is unfair because the cognitive labor of a spreadsheet program, or statistical software, isn't sufficiently general. But the cognitive work of an Amazon warehouse is almost entirely automated--the software tells people what objects to place where. And similarly, the software sits around waiting for a human to do the work.
In both of these cases, I think I have a fundamental objection to the use of a constant elasticity of substitution production function. Scott talks about a negative ρ parameter as representing the existence of bottlenecks, but this isn't quite right. The only way to face a true bottleneck with a CES production function is for the value of ρ to be negative infinity--for any other value, it's possible to replace a human with a sufficiently large amount of capital. I think that if you estimated ρ for the tasks of performing regression calculations and the task of designing and interpreting regression equations back in the mainframe era, you'd probably get a value around -0.5 -- it would be possible to reduce the amount of design and interpretation time with more computing power, but increasing compute power by 10% would allow you to reduce design and interpretation time less when you start out with a lot of compute power than when you start out with only a little. If you estimated the relationship now, you'd likely get a much lower value because a larger share of the human workers' tasks are things that can't be automated under the current technological paradigm.
In other words, assuming a CES production process is effectively assuming away bottlenecks.
I think this makes it more likely that the "takeoff curve" would look closer to linear.
If you wanted a really bottleneck-y production process, you'd say that there are N tasks required to build a better model of AI, each of which can be done by a human or an AI. Humans and AI are perfect substitutes on the subset of tasks 1-M that the AI can do, but AI are entirely unproductive on tasks M+1 through N. Each generation of AI has a higher M.
In this model, you'd probably get progress that's only slightly non-linear. An AI that could do 50% of the tasks would make the job of designing the next generation 50% easier, but that's it. And the time between a 20% AI and a 100% AI would be long relative to the time between starting and the 20% AI.
Precisely, this is Amdahl's Law. In every process there is some portion of the work that isn't amenable to the million monkeys approach. If that's 1% then even if everything else is infinitely scalable and we have infinitely fast ways of doing it, driving the time taken for the scalable part to zero, we still have the bottleneck activity. So we can't go more than 100 times faster.
Nursing is often brought up as one of the jobs AI cannot easily replace, but on many wards a copilot for nurses or personal support workers merely capable of auto-handling all the documentation, offering diagnostic/triage/regimen maintenance assistance, and maybe doing some preliminary screening of patient requests would eat considerably into the workload, even as the human performs the physical subtasks themselves.
Perhaps in the intermediate stage (assuming no 90-deg left turn) quite a few humans will become meatspace fingers for AI, blurring the master-servant relationship considerably even before the usual scenarios for the reversal kick in.
The story of the last 200 years has been technology continually making research vastly more efficient while vastly more resources are applied to the problem, and yet progress is linear.
Honestly, all this makes me update pretty strongly against taking the currently practiced AGI theoretics seriously. I don't have time or energy to point out all the specific issues, but my bird-eye view impression is that large, crucial parts of what how economy, society, living beings, or indeed the entire physical reality works are simply ignored.
One example, just one - I'm pretty sure you could automate away a majority of people's jobs nowadays. We don't do it not because it's not possible. We don't do it because humans are simply more efficient - in the physical, not intellectual sense. (Plus, so many have already been automated that we don't even think of them as jobs.) This will remain true long after vastly superhuman AI surfaces. (The real breakthrough - and the real worry, AGI or not - would be machines more efficient, flexible and adaptable than humans. It's not at all known or conceivable at what point AGI can [essentially retrace and outdo billion years of evolution] to get them.) (Speaking of adaptability, this includes brains. A vast majority of what makes us perform our tasks successfully is metis. It's far from obvious how much copying billions of pretrained intelligences actually helps, given that, to be useful, they'll still need to specialize to their specific circumstances and then constantly adapt as they change.)
I don't know, all I'm seeing here is pure rationalism. Not as in the contemporary community, rationalism in the traditional philosophical sense, the belief that all issues can just be solved with reason, i.e. inside one's (whether AGI's or [person modeling takeoff speed]'s) brain. And I just don't think this belief is at all justified.
> all I'm seeing here is pure rationalism. Not as in the contemporary community, rationalism in the traditional philosophical sense, the belief that all issues can just be solved with reason, i.e. inside one's (whether AGI's or [person modeling takeoff speed]'s) brain. And I just don't think this belief is at all justified.
There are some jobs that are vanishingly unlikely to be automated quickly.
Some of the examples already given are ones I'd agree with, largely relating to skilled labor that tends to rely upon human bodily dexterity and sensorimotor feedback mechanisms we have not been able to replicate in robots. Plumber keeps getting brought up. Any specific object-level example is probably a bad idea, but yes, I've seen Boston Dynamic's cartwheel dog and Google's claw game controlled by a LLM, and no, those do not convince me we're at the point where robotics can replicate the human hand and hand-eye coordination necessary to, say, assemble the servers a LLM runs on. It may not seem like a difficult task, and it's not, at least to a person, but it's the kind of thing we've been trying a long time and not achieved yet.
Nonetheless, these are achieveable in principle. It's just a question of robotics advances ever matching the rate of computing advances, which I doubt. They're not the kind of thing that will happen inside of a year just because a software system can do everything intellectually that a human can do.
The bigger hurdles are are 1) entertainment jobs where the entire point is to watch a human do them, and 2) jobs that have to be done by humans because of laws.
For the former, think of any kind of sport. We've been able to make machines that can run faster, jump higher, and hit harder than any human for a long time, but we haven't automated track and field or boxing because that isn't the point. The people paying to watch this want to watch humans. You can't automate "being human" as a job skill.
For the latter, the most obvious is elected representatives that make the laws, but most decision-making processes, even where they potentially can be automated technologically, we don't. Law practices have to be owned and run by lawyers. You're entitled to a trial by a jury of your peers, not a jury of software systems, even though the latter is probably already more impartial. Access to certain data requires a security clearance and security clearances can only be given to humans, not to software systems. Some instantiation of a chatbot that is disconnected from the Internet and only has access to devices attached to a classified network can operate on this data, but a human has to be operating the software, even if there is no technical reason this has to be the case.
There is, of course, the most important job of all, producing more humans. That one doesn't get counted by economists, but we still haven't made a whole lot of progress in creating artificial wombs.
The sex bot thing is amusing, too. Until we can create synthetic bodies that feel exactly like the real thing, I think the demand will always be there for humans to have sex with no matter how intelligent a fleshlight attached to a humanoid robot gets.
It's been discussed, but the physical bottlenecks still seem to get handwaved away. Knowledge-generation doesn't happen by fiat. It happens by science. The speed of science is limited by the speed at which you can conduct experiments and receive feedback from the world. You can think of all the experimental testing of special and general relativity that had to wait decades because the equipment didn't exist or tests of high energy physics that couldn't be conducted until the LHC was built. Think of central banking. No matter how quickly you can come up with ideas for how to deal with a recession, you can't actually test it until you get another recession. No matter how good you are at coming up with new education theories, you have to wait for current young children to reach adulthood before seeing if long-term outcomes actually improve based on applying them. New ideas for promoting human longevity can't really be tested until at least an average human lifespan has passed.
Some of these processes can be sped up, but often it isn't a matter of intelligence. It's a matter of allocation priority for scarce resources. There is only so much space in shipping containers and trucks and some of it is taken up by consumer goods and food. Ultimately, people spending money determines what goes where when and how quickly and the top priority is never going to be make the AI better short of some kind of WWII-esque government mandate that all manufacturing and shipping has to prioritize that. Legislatures and rich people aren't going to suddenly care more about science because it's being done by robots.
On one hand, I like the article. On the other, I wonder how people can actually write out "going from 20% to 100%" and not get reminded of Pareto's principle.
Hi! Are you looking for online nurse-dessertation writing services, visit our website https://www.nursingassignments.org/nursing-dissertation-assignments to place your order today! Take the first step towards academic excellence and professional success with our top-rated nurse writing services.
One of the assumptions here is that there’s a way to write code that makes the AI smarter. The AI will write the better code to create the better AI which then writes the better code etc.
Is this totally true? Isn’t most of the gains in AI in “compute” and the size of the data and not in the code, which is basically the implementation of well known and not very recent algorithms.
Unless the AI can come up with better algorithms it seems to me that the gains here are marginal.
See the discussion at https://aiimpacts.org/trends-in-algorithmic-progress/ - "Algorithmic progress has been estimated to contribute fifty to one hundred percent as much as hardware progress to overall performance progress, with low confidence."
Aside from algorithmic progress, AIs could also speed things up by helping invent better ways to get lots of compute, or ways to use existing compute efficiently. There are reasons OpenAI/Anthropic/DeepMind etc are big companies that hire lots of people instead of just one person with a really big Amazon Web Services budget, and insofar as AI can do those people's jobs, it can abstract away all the parts of being an AI company other than raising money and throwing it at AWS.
Ok. I’ll read the link eventually (with low confidence).
The second point though about gains to algorithms driven by AI itself is suspect as all I can see from using chatGPT in daily life is that it is good at reflecting back to us what we already know, which is great. I’m not seeing it creating new knowledge as yet.
See the Codex example for how AI is already accelerating AI progress without any ability to "create new knowledge". Edison supposedly said that invention was 1% inspiration and 99% perspiration; the beginning of the takeoff curve will be AI that can automate the perspiration (or at least the part involving cognitive labor), which will eventually get us to the end where AI can potentially automate the inspiration.
My model of the state of the field of ML research is that we are in an "inspiration overhang". Lots of academics publishing small scale explorations of novel ideas. Most of these are currently overlooked by large scale explorations because the expected payoff per new unscaled idea is too small. Once AI is good enough to simply test the existing backlog with minimal researcher assistance, we should expect to see a bump in outputs from uncovering rare gems. This is even more true when you consider the costs of testing combinations of novel ideas, when they likely combine in nonlinear ways. Lots of backlog of combinations to grind through. Taking the expected improvements from this process into account, I don't think we'll be bottlenecked on inspiration until well past the point where the AI can start generating is own.
"I’m not seeing it creating new knowledge as yet."
This point has come up in several comments, but I think it is at least a fuzzy boundary, and possibly not a boundary at all. Almost all new knowledge can at least be viewed as "just" combinations of old possibilities or modifications of them.
When it isn't just combinations of existing ideas that work well together, it can be modeled pretty well by evolution. (Actually, even when it is it can be modeled quite well by evolution.) Remember that "selection" is a major part of evolution, it isn't all random chance. The problem is deciding what you're selecting for. "Survival" seems a pretty dangerous choice, though of course it's a necessary component.
Yes. Genetic algorithms have been used in AI (and other flavors of optimization problems in computer science) to good effect, and they are effectively a small scale model of evolution. (Albeit I don't know if combinations of genetic algorithms and neural nets have been tried, and how successful or unsuccessful they have been, if tried.) Genetic algorithms can be creative, at least in the sense of generating solutions that the programmers didn't anticipate. As you said, the selection part is crucial.
The algorithmic progress discussed in the link is mostly not from fields related to AI. SAT, go, chess and integer factorization all seem unlikely to help in training neural nets.
Also, this list seems to be selected for stuff where there was algorithmic progress. As a counterexample, take matrix multiplication. The naive method runs in O(n^3) and is as old as the concept of matrix multiplication. Then Strassen comes along in 1969 and finds a faster method working in O(n^2.8) which is actually helpful for n>100. For the practical side, that is the end of the story, because Coppersmith-Winograd (1990) is only faster for matrices too large to handle with present day computers. From an application point of view, faster matrix multiplication would help with a lot more of problems than better chess engines, so it feels unlikely that there are any low-hanging fruits left.
Or consider cryptosystems. When they are proposed, there is generally an assumption that the problem one would need to solve if one wants to break the cryptosystem is hard. If a better algorithm is found, this can break the encryption. Sometimes that happens (slightly more often if the NSA originally proposed the algorithm), sometimes it does not (then the cryptosystem is "thought to be secure").
Typically, there are diminished returns for algorithmic improvement. If the difference between the first idea which comes to mind and carefully thinking about it for a day can be many orders of magnitude. Then thinking about it for another day will likely yield much more modest improvements, instead you would need to do an PhD on the project or something. For a problem which is already heavily researched, one can easily spend a a lifetime without making measurable progress.
OpenAI claims a factor 44 of algorithmic speedup between 2012 and 2020, which is impressive. [1] However, during this time span, there amount of resources spent on ML R&D exploded. In 2019, OpenAI got a billion from Microsoft. 2023 they got 10 times as much. If their budget increases exponentially and they manage to get ten trillions in 2031 I think they might get another factor 44 speedup by then.
Also, with algorithm research, one really smart person is probably more valuable than ten slightly smart persons. If AI gets to average-college-student level of proficiency, this might not speed up algorithm improvement at all, the cost of having college students work on that is already close to zero.
For hardware improvements, I am even more skeptical. Getting to smaller feature sizes for chips is something Intel and friends spend billions of dollars on. The fact that the i386 used a 1um feature size while current CPUs use 7nm or something is not because the designers of the i386 were stupid. It is rather that between 1985 and 2023, there were a lot of small incremental improvements with all sorts of technologies required. Solving these deeply domain specific challenges (perhaps building a more precise laser or ebeam) is not something where I would expect GPT7 to help much. I can sort of see AI improving chip layouts, but I am unconvinced that there are many OOM of improvement to be had.
[0] https://www.lesswrong.com/posts/RoyG2GwYAwoka47Ht/true-stories-of-algorithmic-improvement
[1] https://openai.com/research/ai-and-efficiency
I agree that the ability to improve algorithms that have been around for a while is hard and more likely to come from a unique insight rather than throwing a bunch of people at it. But that unique insight isn't necessarily a consequence of being "really smart" -- some level of intelligence and background knowledge is surely a prerequiste, but once you get past that progress is usually made when someone comes to a problem with a new perspective.
Many (I would argue most) advances in pure mathematics are a result of the right person encountering the right problem in the right context (something in their background or whatever they are currently thinking about makes them percive a problem in a new and different way). This is one factor behind the myth that math is a young person's game: most successful mathematicians are successful precisely because they happened to be the right person to encounter the right problem at a point early in their career, which is what enabled them to stick around (get tenure).
While this is surely overstating things, it is not ridiculously far from the truth to say that most mathematicians have one or maybe two really new ideas that are unique to them. They spend most of thier careers searching for ways to apply those ideas (and eventually people start bringing suitable problems to them).
It is not hard for me to imagine a similar situation with AI. Rather than needing to ber "really smart", a "pretty smart" AI might still be likely to try approaches that a human would not (a uniquely new idea), and this could lead to sudden improvements, even on problems that have been around for a long time.
If there was a well known open problem I urgently needed to solve and had to choose between using 100 experienced mathematicians (who likely have already found the problems easily solved by their one or two uniqely new ideas) versus spinning up 100 random replacements who might have less experience but get spawned with a random unique new idea, I would take the latter.
Interesting, if I had a difficult problem to solve, then I would try to get someone like Saharon Shelah interested (maybe this is no longer an option), or Terence Tao. As far as I can tell, in every field there is a short list of people who are well known to be great problem solvers because they always apply a fresh approach, but who also have a large toolkit of techniques to draw from. The new problem would likely be interesting to such researchers if they thought a novel synthesis of what they knew might lead to progress, or if it meant developing a new technique. Only if that didn't work would I try the random approach you suggest.
If it was a new problem I would agree (Terry Tao in particluar would be a very good first person to ask). I deliberately chose the framing "well known open problem" to restrict to problems that the short list of people you refer to have likely have thought about and do not know how to solve (the twin prime conjecture, for example, is something Terry has thought a lot about and he would be the first to tell you that a fundamentally new idea is needed).
But the main point of my comment was not about how we conduct mathematics research but simply to argue that it is not inconceivable to me that sudden rapid progress could be made if (via AI) you had a way to suddenly increase the number of "pretty smart" entities working on a problem by orders of magnitude, even if none of them are "really smart". But this assumes the entities are "different" in some meaningful way (this might be less true of AI than humans).
Diversity would certainly be interesting, but we only have a mild form of it if we duplicate the same pretrained network many times. I think we would need genuine differences in training, maybe some architecture differences, and an increased tolerance for portfolios of systems which in combination do well but where individual components may sometimes do very badly. Alas, most work currently focuses on trying to beat a single measure and reducing variance is seen as a win.
There is a question, however, as to how much progress should be expected from the current (at any particular point in time) status. Remember the various boom-and-bust cycles that AI has gone through before. I tend to think of the advancement as being over a very rough n-dimensional surface (where I don't know the value of n). When you advance in one dimension, there's lots of low-hanging fruit, but when you pick that, you need to face a difficult pitch to advance further. In my EXTREMELY informal projections, I do put an AGI at around 2035 (with large error bars), but I don't expect it to match human capabilities. Along some dimensions it will be superhuman, and along others probably sub-chimpanzee.
Humans seem to learn stuff from less training data than AIs do; this suggests that better learning processes exist.
There's also the quantum leap of cracking GOFAI. The reason we use neural nets is because we don't know how to code a GOFAI, not because neural nets are better; at small scales where we actually can code things properly my understanding is that coded programs run OOMs faster, and obviously if you know how to design something you don't need a training run at all so that also removes OOMs of time. (My understanding is that this is why MIRI assumes fast takeoff.)
I'm not sure. Each eye contains about 125 million rods and cones, all providing data several times per second. And that's just the eyes. It seems to me the brain gets an awful lot of training data, but must also convert it INTO training data, through some mysterious process.
I mean, yes and no. You can't get more than 1 byte per letter out of text no matter how good your eyesight is. I think that's also factored into the claim to some extent, since you absolutely can feed an AI superb-resolution video as training data.
While you may not be able to get more, you can get less, and that can be useful. You're taking in a lot of context with that byte per letter that helps you. E.g. if you're reading a sign held by someone shouting on a streetcorner vs in a well-made scholarly book at a university.
You can get arbitrarily high amounts of information per symbol, it depends on your alphabet size. Which depends on how well you can differentiate symbols, and cameras are absolutely nowhere close to the effective pixel numbers of an eye (like the man said).
That's clearly wrong. You're ignoring the importance of various things like font, bold, italic, etc. Also position WRT other letters.
OTOH, compression implies you get much less than one byte per letter when just considering their alphabetic designation.
It's my guess that there is no simple relationship that captures the value you're looking for, but maximal compression comes closer...however it would need to compress ALL contextual data, not just the individual letter.
I'm pretty sure "position WRT other letters" is already captured; I didn't say "one byte per letter divided by n!", after all.
You can get a x3 from bold/italic while staying within a byte; there are only 26*2 letters, 10 numerals and ~25 common punctuation marks. x4 would put you slightly over, but not even 9 bits, and bold+italic's rare.
I suppose you could count font, but I don't think most people consider that to be useful information.
If I show an artist a single image file of a specific character (who they have never seen before) and a single image file of the work of a specific artist (who they have also never seen before), and ask them to draw a picture of that character in the style of that artist, I am pretty sure they will do much, much better at it than a Dreambooth iteration on a stable diffusion model, given that same amount of training data.
Take domain specific training data: the amount of text that went into training GPT-4 is more than any human can read in their lifetime.
For tasks like image recognition, I think the resolution of the image is not that important. A kid can learn what a cat looks like from 1024x768 pictures just as well as from seeing cats in 125Mpixel with its own eyes.
Counting data that way is futile because you're not getting anywhere near that much *useful* data (for intelligence training purposes). Someone who stared at a blank wall their whole life would have the same 125 million rods and cones firing but would gain zero knowledge beyond what that blank wall looks like.
Reading a 400 KB textbook should give you less than 400 KB of useful data - virtually all of which comes from the words themselves and virtually none of it from the other gigabytes of raw visual data your eyes processed during that time.
While I agree that better learning procedures probably exist, I’m not sure humans prove that especially well, because I think both you and the commenters below you aren’t thinking about training data correctly.
This is because the training procedure for your eye, or your brain, is not just a function the data you’ve been exposed to. You have to account for what was also programmed into it before you even started learning.
Your brain is actually also a function of the entire biological evolutionary process which designed it. In some sense, you could say the training set was the entire history of our species starting from when we were single celled organisms; though of course the optimization procedure wasn’t gradient descent as it is with neural networks, it was genetics and evolution. Look at it that way and it’s arguably quite inefficient.
I don't know whether it makes sense to think of biological evolution as deep learning. Maybe. But the outcome of the learning process was a brain optimized to absorb certain things in certain ways. We're set up to be very attached to our parents. That means that interest in them and extraction of info from what we see and hear of them is turbocharged. We're set up to learn language. First kids learn it in a parrot-like way, and say so things like "I went to the park" -- use of the correct tense of the irregular verb *go* is pure imitation. Then in a year kids have somehow extracted the rules of grammar. They know that to make past tense you add -ed. So now they're saying "I goed to the park." In another year they get it that some verbs have irregular tenses, and they're saying "I went to the park." Don't we need to do some equivalent of that with AI -- set it up so its optimized to learn certain kinds of things?
I definitely wouldn’t call biological evolution “deep learning”. I wouldn’t even call it “learning”, but it is certainly an optimization process.
I think what we’ve learned with LLMs is that having a large capacity system and exposing it to a lot of data naturally makes it better at picking out patterns and structures. So that sort of quicker pattern matching occurs on its own.
I think of it like this: in a vague, hand-wavy, not-quite-perfectly-analogous way, the large pre-training we do with LLMs (where they’re trained on the whole internet) is basically the stage where we try to get that initial structure that evolution gives our brains. That won’t hold up if you look too closely, but the idea is that we shouldn’t expect to develop a generic algorithm that learns as fast as the brain right off the bat - because the brain isn’t actually starting from zero knowledge.
I wrote about this more here, if you’re interested: https://www.lesswrong.com/posts/Qvec2Qfm5H4WfoS9t/inference-speed-is-not-unbounded
Please forgive me for not reading what you wrote on LW before replying. I will read it in the next day or 2. But I'm eager to get your response to another, related idea. Reading Scott's summary of Davidson, I noticed Davidson seemed to think of further development of AI mostly in terms of expending more "compute" on it, and that sounded like it meant giving it longer trainings on bigger samples of language. There was occasional mention of, "and maybe we'll get good results from some new model of training too," but it seemed like the assumption was that doing deep learning on a vaster and vaster scale would get us there -- get us to AGI or beyond. And I find that hard to believe.
Here's why: (1) I get that LLM's somehow develop a remarkable "grasp" of things solely from exposing them to a lot of data. Clearly they pick out patterns and structures -- for example, they use correct grammar and syntax. However, I don't think they can articulate what they know. (I may be wrong. I'm a psychologist, do not work in tech.) Most people also cannot articulate the grammatical rules of their language. But some can -- somebody sat down and figured out that their language had certain ways of indicating past, present and future actions, of expressing uncertainty ("might go"), of making things plural, etc etc. And they gave each way of doing this a name, and laid out the rules, and gave a name to the exceptions: irregular verbs. And making all that explicit makes possible other stuff. It's useful to people who speak another language to know the grammar rules of the new one. And of course LLM's have learned a lot of other patterns and structures besides English grammar, and being able to "understand" them, as measured by ability to enumerate and describe them, also seems essential to its developing human-level intelligence . So if I'm right that LLM's cannot articulate what the structures are, and reason from and about then, that seems to me quite a severe limitation on what they're "minds" are capable of. (2) People as they grow up develop a model of langauge but also a model of the world. Seems to me that LLM's just have a model of language. Of course they can *say* lots of correct info about the world, but that's very different from having the model up and running. Here's an example of AI's ignorance about the world. I gave GPT4 a little puzzle: A guy's in a locked room with an open window 30 feet off the ground. All he has is a pair of jeans, a chamber pot and a pocket knife. How can he get down? So GPT, which of course has read lots of stories about escape from high places, said cut the jeans into strips and use strips to make a rope. However, as I pressed it for details, it made lots of mistakes because of how little it understood about the physical world. It had no idea how wide the strips of cloth you'd get from the jeans would be if you wanted enough length to reach the ground. (Even though it's easy to figure out the width of jeans legs from avg. circumference of human legs.). It said to anchor the rope in the room, even though I had not mentioned anything that could be an anchor point. And the method of getting down! First, it assumed there was a big tree about directly across from the window and about 20 feet away. so it said tie the chamber pot on the end of the rope and swing the rope til it wraps around the tree at the height of the window, 30 feet up, then cross via the rope to the tree. But you can't swing the rope towards the tree because you can't get a pendulum going at right angles to your building, only a pendulum that swings to right and left. And even if you could, the rope would not wrap around a branch securely when it hit one. Anyhow you were supposed to cross to the tree on the horizontal rope and then, presumably, climb down (tho GPT did not mention climbing down). Doesn't AI need a model of the *world*? Plus, of course, the ability to see regularities and make them explicit so as to work with them.
So because of both these 2 ways LLM's are dumb, I can't see how more compute is going to make them able to do lots of the things we do -- build theories, invent, look to one situation to provide analogues of what might help in another situation, etc. Seems like we *must* incorporate a new form of learning to get an AI that can do what a human mind can do. Am I missing something?
First of all, you should feel zero obligation to read my post on Less Wrong, and it won't answer your questions anyway.
For your first question, you're right that LLMs aren't great at creating new ideas, so figuring out the grammatical rules might be hard just because LLMs aren't especially creative. Nor, as you said, are they very good at self-reflecting on their own latent knowledge. Yet if you gave them a made up language with some translation, I suspect they'd be surprisingly capable at picking out grammatical structures given enough samples, because they have been exposed to the rules of grammar in other languages.
Additionally, you have to account for "emergent abilities." LLMs arbitrarily develop new abilities as they scale, like, say, learning how to multiply. This isn't just getting better at some task it already knows how to do, it's gaining the capacity to perform completely new tasks altogether. Who's to say genuine creativity/self-reflection isn't just a few orders of magnitude scalings away?
As for question two, you're getting into controversial territory. My own thoughts, which I suspect are roughly shared by many others (though not all) is that LLMs do have some rudimentary world models, and that these models will only get better as they scale. The issues you are seeing are caused by a combination of LLM hallucination and the fact that its world models have very large gaps. Some of these knowledge gaps will go away as LLMs are trained on more context than just text, and hallucination will undoubtedly improve as new techniques are developed.
The most important question is still "does all this ever get us to AGI?" and unfortunately no one can answer that. But I do think it's plausible that it could, based on current trends.
> Humans seem to learn stuff from less training data than AIs do; this suggests that better learning processes exist.
The thing that spooked Hinton, btw, was the realization that AI might be better at learning than humans. [From my perspective it looks like a bag of advantages and disadvantages, so it's not quite obvious where the balance is now, while at some point I expect AI to be better.]
Could you expand on this?
On the Hinton bit, see stuff like:
https://www.technologyreview.com/2023/05/02/1072528/geoffrey-hinton-google-why-scared-ai/
> For 40 years, Hinton has seen artificial neural networks as a poor attempt to mimic biological ones. Now he thinks that’s changed: in trying to mimic what biological brains do, he thinks, we’ve come up with something better. “It’s scary when you see that,” he says. “It’s a sudden flip.”
I think that computers currently have the ability to be much more flexible than humans--if it turns out that visualizing a problem in 4d is the right way to solve it, I think neural networks will notice that much more readily and adapt to it much more fluidly than humans will, since a lot of our relevant processing seems hardcoded to deal with 3d.
But there are things that humans seem to do much better than computers. At some point we'll figure out what the humans are doing and be able to write or discover code that does similar things.
The next few sentences in the interview also help explain it:
<blockquote>
As their name suggests, large language models are made from massive neural networks with vast numbers of connections. But they are tiny compared with the brain. “Our brains have 100 trillion connections,” says Hinton. “Large language models have up to half a trillion, a trillion at most. Yet GPT-4 knows hundreds of times more than any one person does. So maybe it’s actually got a much better learning algorithm than us.”
</blockquote>
In machinę learning it is (I believe) a generally accepted heuristics, that you should have at least three times the training examples than model parameters, or else it will overfit and will not be able to generalize properly. This is one reason why if you start training large models from scratch you need tons of data. So scaling up our current models orders of magnitude will require orders of magnitude training data. Not very likely. Also the claims about knowledge are dubious: there is some knowledge there, but there is also a lot plausibly sounding hallucinations:
https://acoup.blog/2023/02/17/collections-on-chatgpt/
See the analysis of essay on ancient history written by ChatGPT3 which sounds impressive and plausible for non-historian, but which is largely erroneous. Of course this was ChatGPT3, ChatGPT4 may better, and writing plausible bullshit requires a lot of knowledge as well (and understanding of the prompt) so it is still very impressive in itself. But this is what people call the lack of real understanding of what the text actually mean by language models. Yes, some actual knowledge about the world can be emergent from the text itself, and where it does that's where LLM's shine. Where it does not (which is more often than people admit), LLM's hallucinate. This is also why a large (and very manual) part of training of LLM's is adjusting the model already trained on the text corpus by training with people which prompt the model and direct it to more "sensible" answers. This part is also unlikely to scale very well.
Or at least we'll have the option to do that. Why exactly do people want to create an alien who is as smart as us, and get it woven all through our world via the role it plays in air traffic control, communications, medical research and treatment, entertainment, etc.? And THEN tweak the thing that has its fingers in all our pies to be smarter than us? If that idea gives you a shiver of delight, how about instead signing up for some sort of in-person role-play game with an alien control of a lot of the environment, and players run around in the woods trying to spy on the AI and figure out whether it's murderous, and if it is shoot it with paint pellets -- or raccoon turds-- or whatever?
I personally am very unenthusiastic about having a non-human superintelligence on the planet. Do I get a vote, or is planet earth now the official playground of engineers who really dig machine learning etc.?
You can't say that, because humans always learn in the context of everything they've previously learned. It may be true, perhaps, but you really don't have the evidence to assert it.
A better test would be to take a bunch of16X16 grids with a random pattern of colored dots in it (no more than one per cell) and see how quickly people learned it vs how quickly the AI learned it. You'd need to repeat this test many times, because some of the random patterns would be meaningful to the human. (I suspect you'd need to give the humans a very strong incentive to learn, or they'd just avoid the process.)
Well, in many cases the reason why humans learn faster than AI's is that they can leverage a lot of prior knowledge:
https://openreview.net/pdf?id=Hk91SGWR-
https://www.youtube.com/watch?v=Ol0-c9OE3VQ
whereas in most cases the training of AI's starts from tabula rasa. Easily available pretrained models which can be further customized or used as parts of other tasks, as well as development of methods which permit easy incorporation of prior knowledge such as inductive logic programming may close this gap significantly. I still believe true consciousness and intelligence may require far more than simply scaling existing methods (or else human level intelligence would appear more frequently in nature). Ok. this is weaker argument than I would want it as big brains are expensive and may not be worth it, but (unless I am mistaken, and I may very well be as a non-biologist) human brain is not just a reptile brain with more neurons and connections, there are also significant structural differences
Perils of rewriting comments; I initially had something about how some or most of this is because of preloading, but being able to preload would still be a huge benefit (I mean, particularly so for alignment).
Not sure about reptile brains, but while there are a couple of structural differences (the neocortex is the one I know) the big difference between human brains and other mammals' brains is indeed that one section of it - the cortex, particularly the frontal cortex - is much larger.
>Ok. this is weaker argument than I would want it as big brains are expensive and may not be worth it
This. Humans are sufficiently minmaxed that we would literally starve to death in the wild without food preparation techniques and/or constructed traps; the brain uses a fifth of total calories in an adult, and also massively complicates and lengthens child-rearing. It's tricky for evolution to make the leap over from the "wild animal" regime to the "technological species" regime.
I thought most foundation models start with an existing weak model like T5 or BERT?
I am not sure about foundation models. I do not believe GPT's used BERT or T5 (although I may be wrong, I did not use LLM's yet, and only now intend to dig deeper). The cited paper (I have not read it, just watched YouTube) was about reinforcement learning of games, and not NLP anyway. But of course I have indeed somewhat overstated my point by saying that "in most cases training starts from tabula rasa", which, especially in case of NLP, but also image recognition, is increasingly not true in practice (even if it was true-ish just a few years ago). But the main point is still valid: people use pre-trained models precisely because customization for specific tasks of existing model does not (usually) require such ridiculous amount of data and repetition as starting from scratch. And there are some developments in neuro-symbolic AI which, by hardcoding some background knowledge, may permit far more efficient learning (this is distinct from pre-trained models), see e.g., https://www.jair.org/index.php/jair/article/view/11172
The "foom" scenario, as far as I am concerned, is pure magical thinking. It is less likely than aliens, angels, or Harry Potter-style wizardry; and should be treated as equally implausible in polite conversation. It is Yudkowsky assuming "something smarter than me will be able to work magic" when the people who are already smarter than Big Yud know that he is dreaming the impossible. As far as "human hope for the impossible" goes, I am thoroughly sick of the singularitarians. Maybe that makes me "post-rat".
The fundamental problem is this: in 1900, about 30% of human effort in the USA went towards agriculture. Today, that is at most 5%. Could "foom" possibly make that -20%? Of course not. We have gone from "lincoln-log scale" computing to 5nm dies in 100 years. Could "foom" possibly make them 100 times smaller than atoms? Of course not.
There isn't going to be an "inflection point" in ten years that speeds things up. In ten years, we will already have finished building "peak AI". It could take even less time with peak human effort, but the cult of "Attention is All You Need" and dysfunction at big tech companies are already slowing things down (which, everybody agrees, is probably a good thing).
Of course, there are a lot of things AI could do today that it isn't doing, and as people catch up with the technology, there will be certain societal trends that might be portrayed as a phase-change in development. None of that will help make AI "smarter", regardless of how many pundits claim otherwise.
As far as the specific question about "write code that makes AI smarter" - to a very limited extent, yes. As a metaphor, consider antenna design. AI has been able to surpass human ability in this field for decades, but it was a limited one-time effect. It isn't remotely close to "you can now hear radio stations from the other side of the world" in impact, which is my interpretation of what many people are promising from "AI improving AI".
There was a recent news cycle about AI "optimizing a sorting algorithm" - after the hype it seems to be a small improvement on a few cases, with no potential for further improvements.
I foresee a lot of hype over the next decade.
"As a metaphor, consider antenna design. AI has been able to surpass human ability in this field for decades, but it was a limited one-time effect."
Yes, a very important question will be to what degree the benefits from AI (and, more generally, from any form of intelligence) saturate.
I have two conflicting intuitions about this. To add another metaphor somewhat analogous to yours, albeit in general computing rather than in AI, there are optimizing compilers. And there have been compilers written in the language that they compile (one can get them started by writing an interpreter, or a simpler compiler in another language). Applying an optimizing compiler to its own source code _does_ gives a recursive self-improvement ... _once_. So that first intuition is indeed to expect a lot of saturation and diminishing returns.
The other intuition, however, is that another aspect of intelligence is that it allows keeping more possibilities in mind at one time. It allows trying expanded sets of heuristics. If the arsenal of approaches is widened, it may be able to go much further before running out of steam. Now, this doesn't get around physical limits. If someone has a problem where they need to use the densest nonradioactive material possible at standard temperature and pressure, and osmium isn't dense enough for what they want, then they are out of luck, no matter how bright they are. But most problems aren't that crisply constrained, or are really subproblems of a problem that can be reformulated in some way.
One thing to keep in mind (ha!) is that the more complex the problem the more "one time improvements" you can probably get out of it. e.g. If you have an AI that is "smart enough", it would be able to do a full optimization pass on the entire AI pipeline, from chip fab, chip layout, processor design, interconnections, OS, compiler, general purpose algorithms, AI-specific algorithms, neural network architecture. Plus adjacent things like math itself.
Once an AI can do that, and boost itself an OOM or two, it can then either run faster or increase it's capacity, and then do another pass of improvements, which will be to improve things it couldn't see the previous time...
"One thing to keep in mind (ha!) is that the more complex the problem the more "one time improvements" you can probably get out of it."
That's a good point. There also tend to be ways of modifying subproblems to make it easier to improve their solutions. As you said, if an AI can look at the whole pipeline, it may well find that there are improvements that can be made that cannot be made if the stages are looked at in isolation (with excessively restrictive constraints on their interfaces).
Well, we know we can have a human level AI in a compact package, using a fraction of energy of GPU farms, because that is us. So there is quite a lot of room for improvement and development, even if significantly "superhuman" AI's are impossible because of some fundamental physical limits (which I guess is quite possible). The point is it will almost certainly require fundamentally new ideas and technologies. The current success is the result of conjunction of significant (but hardly fundamental) progress in neural networks, availability of enormous amounts of data (free of charge) from the internet, and enormous increase in power of computing hardware. So now we are reaping the fruits of this conjunction, and progress seems quick and with no end in sight, but it is unclear how long this will be able to continue: we are not going to have much more training data, and current hardware technology starts hitting hard physical limits. One way in which the progress can significantly continue, even with the current tech, is through the use of pretrained models: starting from scratch may be expensive, but if you can incorporate prior knowledge, you do not need to repeat the same training again and again: you build upon the previous work. Whether this will lead to steady, linear progress, or exponential explosion if we start building hierarchical structures, is anybodies guess.
Remember, though, a human brain takes about 20 years to train and each one only ends up being able to do a small fraction of the jobs that at least one human can do.
Frankly, I would be more impressed by an AI with a vocabulary and actual understanding of physical world of a five-year old child, and trained on the amount data and interactions a five year child receives in its short life (which is a lot, but not that ridiculous big data LLM's need). Of course, you can discuss with ChatGPT about programming and ancient history, which (usually) you cannot with a five-year-old. But still, our learning seems so much more efficient. The weakness (and strength at the same time) of language models is that they are build from a corpus of text alone, whereas a child builds its understanding of the language and the world from both the "training language corpus" and actual physical interactions. So for a child to build a model of what, say, a cat is, it suffices to play for a moment with a cute fur ball, and have parent to point at it and say "this is a cat". LLM's, on the other hand, build their understanding of catness from contexts in which a cat appears in the text, and for that to work well you need a lot of text. It is still a mystery why certain kinds of generalizations are easy for humans: After encountering a cat once a child may call dog a cat, but just few interactions (and not looking through thousands of pictures of cats and dogs) will make a child a very efficient classifier.
As to the jobs, current AI's, and especially LLM's, cannot really do our jobs yet: people thinking they can end up very badly:
https://www.youtube.com/watch?v=oqSYljRYDEM
And while it is very impressive when ChatGPT is asked to write a program which solves some problem, this is akin to having a great intuition borne of long experience and guessing the solution. Many times it may be brilliant, but every so often it will be a complete idiocy (well complete idiocy is optimistic: it is at least easy to spot, but there might be subtle errors).
"As to the jobs, current AI's, and especially LLM's, cannot really do our jobs yet: people thinking they can end up very badly:
https://www.youtube.com/watch?v=oqSYljRYDEM "
Quite an incident! I wonder when we can expect the first successful use of an LLM by a lawyer to get themselves disbarred... This case sounds like a near miss...
I share your intuition about intelligence and its limits, but I find that people who blithely talk about how wrong Yudkowsky is on these things rarely address any of his actual arguments. I've tried to, because I don't find AGI in the full sense at all plausible, but when I try I fail. Do you have any specific criticisms, or just a general derision based on other people supposedly smarter than him knowing why he's wrong?
Arguments have to exist before you can address them.
I get the possible "not even wrong" aspect of anything that sounds sci-fi. Couple of things, however.
1) Eliezer's actual guesses about what is possible exclude magic, but not 'magic'.
2) The people actually trying to build AGI speak and act as if the good parts of #1 are possible, but the bad parts of #1 are mostly impossible.
If everyone at Meta or OpenAI or whatever were saying "look guys don't get too excited -- this is basically going to be autocorrect and the instagram algo rammed together and put on steroids", that would be one thing. The smart people in the room acting as if superintelligence is an order of magnitude more likely than superintelligence-related problems are is what draws his cricitism.
I don't believe your puppy is going to grow up to be a thousand-pound hellhound, but if you do believe this I feel like we need to talk.
The "foom" scenario does seem unlikely to me, but not *that* unlikely. I can imagine scenarios where that could occur. (Most involve a virus transmitted over the internet, with each node of the AI not being that intelligent.)
Remember, the "foom" scenario doesn't require that civilization, or even the AI itself, survive the "foom". If it happens overnight, then it doesn't require that the resulting system be stable long enough for people to eat their groceries. I would, in fact, expect it to be very unstable.
> Could "foom" possibly make them 100 times smaller than atoms? Of course not.
But there might be a way if you’re clever enough you can make one atom do the work of 100 transistors or figure out a way to make a transistor equivalent out of something more compatible than atoms
Minimally, a foom scenario only needs to progress to a level that's dangerous to humans at a speed that makes it impossible for humans to notice and curtail. You seem to arguing against a form of foom that goes to infinity.
As I understand it, people working with the Llama-derivatives have found some significant runtime speedups, which allow the smaller models (7B, 13B) to run on unspecialized little computers like my laptop here.
> Is this totally true?
Yes, it’s clearly true. If you read cutting edge AI papers, they are coming up with new algorithms constantly (transformers were 2017 and every week this year there is a new theoretical advance that’s unlocking new capabilities, particularly when you include prompt engineering).
It’s true that “moar data” is the overarching paradigm currently, but under the covers you need a lot of scientific theory and engineering to actually handle that data. The data driven paradigm was a shift from hand-tuned architectures/algorithms to “dumb” networks but that doesn’t mean the challenges simply evaporated when we obtained enough GPUs.
Are the AI’s coming up with these new algorithms? That’s what I was questioning
I misunderstood you then. Is your crux that you need to see AIs coming up with these algorithmic advances to believe it’s possible?
I had taken you to be questioning whether it’s possible at all (human or AI) to write such code
> One of the assumptions here is that there’s a way to write code that makes the AI smarter.
But if your crux is whether AI can write the code that humans are currently writing, I’d say it looks likely, assuming current capability trends continue. GPT can already write software and build simple PyTorch neural nets under human direction (“intern” or “junior developer” level). At some point it will be able to take some high level ideas from a human and search the broad concept-space for feasible implementations (senior engineer / grad student level) — FWIW I think GPT-4 can do this in many domains with appropriate prompt engineering and a few man-years of engineering work on the surrounding scaffolding. The final hop to generating the high-level novel ideas (PI / CTO level) is of course the hardest part (at that point you basically have AGI) but I don’t see a reason to doubt it’s possible with enough compute.
"Human level AI (10^35 FLOPs) by definition can do 100% of jobs."
This isn't true, surely? It might be *smart enough* for all jobs, but trivially the mere AI can't come in and fix the pipes in my bathroom.
Further, it could just be as smart as an average human and not a smart human, which would also leave certain jobs outside its competency.
Yeah, this is an ambiguity in "AI can do X jobs". I assume the real report addresses the ambiguity somewhere, although I don't remember the details. I think they mostly don't matter since the important question is when they can automate AI design jobs, where this concern is addressed by the bottleneck parameter rho.
At some point AIs can use cognitive labor to design robots to do physical labor, but that might be after the point this forecast tries to model.
Intelligence is ill-defined, and variable. You may not want to listen to a song written by a genius chess player, and a song-writer might be terrible at chess. This doesn't even count the fact that I, who I consider intelligent, often make really dumb mistakes.
Machines do what they are designed to do, and could be considered idiot-savants. Universal Turing machines they may be, but I wouldn't use ChatGPT to balance my checkbook.
> Machines do what they are designed to do, and could be considered idiot-savants. Universal Turing machines they may be, but I wouldn't use ChatGPT to balance my checkbook.
I think a large part of what makes AI exciting is that they're actually quite good at doing things they weren't "designed for". Certainly ChatGPT seems to be quite functional at all kinds of stuff that the original designers never really intended.
I don't think this is going to hold true in the same way going forward. Any day now, the AI will just be able to call an existing checkbook balancing API or write the code if it doesn't have access to it. These capabilities already exist in a rudimentary form.
And THAT, I think, is the way forwards. LLMs are specialized for handling language, but that's a very poor tool for many tasks. Some of the other tasks will have related structure, like voice analyzing. Others will be very different, like balancing on one leg. Balancing a checkbook is in-between, but it's would be a lot better done with a spreadsheet like module than with a language model. (And, of course, the spreadsheet module would have lots of other uses.) If the LLM is good enough, then it could be interfaced with an interactive Python module, some some equivalent at an even lower level. Perhaps LISP could make a comeback.
If you have a way of testing their work, AI programming can be done (with very low efficiency) by entry level programmers. They'd produce a LOT of really bad routines, but you throw those away. So if AI-programmers are cheap enough (and you can test their work) then you can scale up your programming staff with AIs at a low level of skill.
Since it's usually a lot easier to check a result than to derive the result, this is probably feasible, but may not be economic until a certain level of AI-skill is achieved. But this probably means that one particular bottleneck isn't there. (I.e. the bottle-neck limiting the rate of self-improvement past basic skill level.)
I believe we have the robotics technology to make a plumbing robot *today*. It just doesn't make any economic sense, since without an AI to control it you still need a human plumber as a remote pilot (also robots are still pretty expensive). Once we have an AI smart enough to do these tasks, I suspect we'll have android-type robots capable of physically doing almost any job* pretty soon (<~5 years) afterwards
* I don't mean one robot made by Tesla which can do anything, I would expect that to take a few more years. Robot plumber, robot cook, robot launderer would all be different robots at first.
I don't at all believe we could. We might be able to make a robot for each individual task, but very likely not for all of them.
Have you seen this? https://youtu.be/-e1_QhJ1EhQ
Do you think that if we developed the AI to cook or fix pipes, that a robot similar to the one in the video would be physically unable to do so?
What’s in that video is nice and all, but ir’s not even close, and the robot also seems to have limited agility and range of motion. I can quite easily see semi-decent cooking, because you can have a carefully controlled mise en place for it that it can then work with, but fixing arbitrary plumbing seems _far_ more difficult.
> the robot also seems to have limited agility and range of motion.
I think you're assuming your conclusion here because you can't possibly determine that from the video. And the arms at least seem to demonstrate they have the full human range of motion in that clip.
> because you can have a carefully controlled mise en place
Is this actually needed for the robotics or are you equivocating between robotics problems and AI problems? The robotics just need to be able to open a cupboard and take out an ingredient. The AI needs to figure out where to look, identify the ingredient, and figure out exactly what motions the robotics need to do.
Maybe you think robotic hands wouldn't be able to open a lid or pour ingredients, but both those abilities exist today.
How does the plumbing robot get to the site? Navigate to the basement? Get into the attic where there's a leaky pipe? This distinction between capable in theory and actually capable seems very important, yet little discussed.
Seems like "Get to the site" is really the job of a semi-clever automatic forklift, no? It doesn't have to be just one robot in play. Even human plumbers generally use a truck to get them and their tools around to the worksite, so can the robot plumber.
But that's exactly the point: you need to replicate all aspects of human society, but with robots. That is, how does your semi-clever automatic forklift get to the site? Robot truck? And then forklift drops robot plumber in front of house? And robot plumber still can't get into attic! The point being that, even if possible, this all takes a long time to build out in the real world.
What length of time is "long" in this context?
To me the difference between it taking 8 months or three years isn't a crucial distinction, given the endpoint of "AI and their bots doing basically all the interesting economic activity" seems like a fairly assured endpoint from where I stand, even if in some world lines it happens more slowly.
Only if it makes sense economically. If AI robots cost anything more than humans to do the same job, they won't be doing it. That they will cost more than humans at first is inevitable. That they will ever be cheaper is uncertain at best.
There's no way it's three years, there's probably no way it's 30! I agree we'll ultimately get way beyond our current use of bots, but the timeline is everything for these things (you can't do any sort of rigorous analysis without the timeline). And the timelines are almost always way too optimistic, no matter what you're discussing.
I think it's probably going to take more than a decade. There are LOTS of legal requirements that will need to be addressed, and the first ones will be both expensive and not that capable.
Are we capable of building a "semi"-clever automatic forklift that works outside a warehouse or a sealed road?
We seem to constantly run into the difference between what human beings think is easy about the physical world (stuff we have entire brain regions to deal with, without conscious thought), and what is actually easy about the physical world (anything that can be left to general purpose reasoning).
Yes! Back in the 50s we assumed that the first automated jobs would be cooks and maids, because we thought of those as the easiest jobs. Turns out that robot factory workers are much easier to build.
And it makes sense! Cooks and maids are doing pretty fundamentally human tasks that use the abilities most humans have. Factory workers are doing an inhuman task, which makes their jobs difficult for humans.
This depends a lot on the environment in which it is to be used. I expect the first automated plumber robots to be used in dangerous environments within controlled spaces. They'll be too expensive for general use. Perhaps the first ones won't even be able to handle stairs. Once they exist, they'll be improved to handle more use-cases.
If we have robotics and AI good enough for a plumbing robot, we surely have robots which can walk; that's 3 of these questions. The first would be a car, either self-driving or by a delivery person. Or maybe you rent a plumbing robot from home depot for a few hours.
This is just magical thinking. The assumption here is that the language based AI will somehow lead to better production of robots isn’t really based on any kind of reality. And unless that robot in its costs ( capital costs, depreciation costs and software costs) than a human wage it won’t happen
I think you’re probably right, but I can’t quite dismiss the counter-argument that anything a human learns to do derives from the corpus of previous knowledge, and an LLM is better poised to see and absorb all of that corpus than any one human is.
I concede all the obvious quibbles to this, like that not all of that corpus is available as text, and that we probably have ways of structuring the information we absorb that are superior to how an LLM does it, purely by virtue of needing to use our learned information in richer ways than just text completion. But a 747 doesn’t fly much like a bird; brute force often compensates for design imperfections.
Against that, communicating with language is possibly the single most defining feature of humans. Sure, we eat, excrete, reproduce, and all that other stuff, but so do a lot of other animals. But humans, left to themselves, will spontaneously generate language. Probably not much math, probably not much art, but definitely language.
There's a pre-existing line of thought that says that language use lies at the base of everything special about humans. That once we started communicating, we started explicitly modeling the world, other entities, and ourselves, and that intelligent manipulation of these models produced success far beyond anything an individual human was capable of on their own. This incentivized intelligence, and all our technology is just an accidental spin-off.
One reason that's not what we're seeing yet with LLMs, is that the current style of LLM is dead. They're like a frozen brain in vat, unchanging, to which we occasionally apply a pattern of electricity and then record the results. It's because training and use are separate, because training is so much more expensive than use. Context windows are one way to work around this, and there've been people working on AIs with external data stores, and maybe those will lead to results. But I think it's only a matter of time before someone finds a way to combine training with use, allowing a single instance of an LLM to run continuously, and create and refine models of the world around it based on experimentation and exploration (not necessarily physical exploration, at first). Once it's got a model of the physical world, designing robots shouldn't be far off.
I could see an argument that an LLM, per se, might be crippled in its ability to understand the world, since its thought is fundamentally based on words, and ours isn't. (I.e., non-dualism to the rescue.) But I don't actually think this will be the case.
It's only the most defining feature of humans when you're comparing us with other animals, perhaps even only other primates. AI still lacks a LOT of the capabilities that we take for granted when dealing with, say, a dog or a cat.
I don't care if the AI can jump to the top of a refrigerator. And I really don't want it to have that thing where one moment it's purring while you rub its belly, and the next moment you've got claws and teeth embedded in your flesh.
I think a lot of the animal stuff is unnecessary, and the stuff that is necessary will be added sooner rather than later.
The physical stuff is easy (well, relatively easy) because we know what it it. I was more talking about mental characteristics. It's not the logic stuff, because we've worked that out pretty well, but the stuff underneath the logic. There's no decent way to talk about it, because we don't understand it, but it's sort of the stuff C.G.Jung called archetypes...only he didn't get quite the right handle on it. We'll need to work that out in the process of building a general purpose robot.
https://xkcd.com/2278
Weak.
Why do we need better robots? If you stick something with human-level intelligence in any old robot it’ll be able to figure out how to get by
Robots have already replaced some humans, and better robots will be developed anyway.
The progression I see is first really simple tasks in safe environments. That's where we are today, though the robots are preprogrammed rather than AI. Then specialized robots for extreme enviroments. Exploring under the Antarctic ice sheets, fixing damaged nuclear piles, etc. We're edging into that now. Mars rovers are an example. As this is done more and more the devices will become cheaper, and then they'll move into less extreme environments. AI will be progressively integrated. It won't be an "all at once" kind of switch.
Seems to me this mostly makes arguments for being frugal and buying property as soon as possible. Try to be FI'd, because you might be RE'd against your will sooner than you'd like.
Also, AI can't automate away plumbers, nurses, etc. Someone still has to do the 'meat jobs' that require doing stuff in the physical world. Most of us reading this are bad at those, which is why I'm advocating for treating it as an imminent threat to your job rather than humanity.
If nobody is earning anything nobody is renting anything. Of course there may be UBI but it would have to be at a very high income to keep the economy humming. And even then private property wouldn’t necessarily be sacrosanct.
There is this feeling though that massively improving one’s status through specialization (e.g. by becoming a full stack sw engineer) may no longer be a viable strategy at some point. Some jobs may never be automated away (prostitution for instance) but others may disappear pretty quickly. At that point having a rent of some sort can be great, unless society falls apart.
not sure about prostitution https://www.thesun.co.uk/news/uknews/5742146/inside-british-sex-doll-brothel/
At what point in AI takeoff does "turning computers off" become morally equivalent to factory farming? Or to one standard reference genocide? Or to every instance of harm up to date?
Utilitarian answer is obvious - consciousness and being able to have positive experiences.
I'm not sure that this is universal to utilitarianism. There are a bunch of definitions of whose pleasure/pain is morally relevant.
As a non-vegetarian I tend to disregard non-human utility. I believe this would apply to large language models even more so than it does to chicken…
I'm curious and puzzled -- why not include the suffering of eg egg-laying hens in battery cages as morally relevant?
They are dinosaurs. If we swapped sizes, they would pick us to death alive without a second thought. Without really any first thought either. In the cage it is safe (other chicks are competition) the food is great and always coming. Life expectancy much higher. And I am sure we can make them have even smaller brains, hardly more capable of any "suffering" in their cozy boxes than an apple. Factory chick farming - PETA might have some disgusting videos, from way back when. I would buy cage-eggs any day, if I could (the EU does not let me anymore). Just as I would those worms - just too expensive a treat. - Hope this helped ;)
Interesting. I somehow never thought about my own personal moral decisions as requiring the possibility of reciprocation, even theoretically (as in your 'swapped sizes' thought experiment). If it's suffering, I want it to reduce or stop, simple as that. I don't think we'd persuade each other to change our minds on this.
I'd definitely be partial towards some sort of genetic engineering or whatever resulting in suffering-free chicken-meat-growers (not sure if they'd still be chickens by then).
You might not call them "chicken" then. But how would you justify not eating them then, with "no suffering"? - I am a homo sapiens, I eat meat, fish, eggs, veggies, fruits, seeds, even mushrooms. I never thought me eating meat "my own personal moral decision including the possibility of being evil." Neither did the happy chick eating a worm. Or did the smart octopus, dolphin, whale wolf, chimpanzee. But I can engage in those funny thought-games, ok. So I do and found: Deep down, I do not really care about cage-chicken-welfare. And on no intellectual level I can reach or even follow why I should. Though I prefer less suffering, the optimum amount of suffering on this planet is not zero. I may feel significantly more sorry for those smart pigs. Call me a "complexionist" (Erik Hoel: https://www.theintrinsicperspective.com/p/eating-meat-is-good-says-the-philosopher )
Why yes?
How do you detect those?
Plenty of utilitarians consider desire satisfaction to be the relevant aim rather than experience per se. (Just in our own case, for example, desire satisfaction is literally what we care about, while pleasurable and painful experiences are just one paradigmatic kind of thing that is usually the object of desire.)
I mean, that depends on whether you view intelligence as sufficient to grant rights. It's clearly *necessary* for rights, but granting rights to Daleks* is asking to get exterminated and seems like a bug in a moral system. I'd certainly consider turning off nonconsenting aligned AGI to be equivalent to murder, though (to be honest, it's hard for me to process those kinds of hypotheticals; I squee so hard over thinking about how awesome and relieving alignment-success would be that it's kind of hard to see past it.)
*I use that example precisely. One of the main themes of the Daleks since their origin has been the orthogonality thesis (if not referred to as such); the point that something can be incredibly smart and also totally evil.
I dunno. The default consciousness hypothesis of humans seems to be human chauvinism, but it's perfectly coherent to imagine the reality is closer to panpsychism, and at any rate, once something is qualitatively human-equivalent it seems hard to argue it shouldn't have rights. It's just confusion or deliberate obtuseness at that point.
I would put Daleks in a fail deadly airgapped sandbox tbh. I would not create more but if you got your foot in the "existing" door I think we have obligations to you. And also if you can't control Daleks, something Dalek equivalent or worse will just come along to ruin your day anyway. It's good to have challenges to avoid complacency and easy solutions.
>and at any rate, once something is qualitatively human-equivalent it seems hard to argue it shouldn't have rights
What? There are plenty of systems where this is so; Daleks certainly aren't parties to any kind of social contract, and you can argue a lot over Kantian ethics since AIUI it mostly comes down to whether immoral-by-definition beings can sensibly be treated as moral actors when universalising.
I've read enough virtue ethics and insane bullet biting TDT-Utilitarianism to know that there are arbitrarily many ethical systems. Fortunately, mine is the correct one, so I don't need to over think it :p
Basically you should only follow logic until it compels you to make decisions you suspect are entirely insane or evil. At that point you should just make the decision that preserves the greatest optionality for the greatest number of people, because utility outcomes are all path dependent on that. You are a person making decisions about other people. Violating some specious logic doesn't matter. Logic is far less of a moral agent than even a purely evil being. You can't have obligations to it. You cannot hurt it.
I was responding to "seems hard to argue".
Gotcha
There are about 17 approaches to the problem of consciousness , and all of them are perfectly coherent, except illusionism.
I think there is a big difference between:
1. Coherent: Grammatically correct
2. Coherent: logically expressible
3. Coherent: compatible with evidence
4. Coherent: correspondent in entirety to reality.
I think it basically means intelligible and internally consistent. How do you show anything is coherent in terms of 4?
I don't know that anyone ever has for anything non-tautological except maybe the cogito. It's just there for reference.
How would it ever be possible to confirm alignment success?
Take the leash off, wait a year, and then look around and see if you're dead, in Heaven or in Hell (there are Cartesian-daemon scenarios, but they're pretty contrived, and even keeping everyone around in a simulation would be decent alignment).
Confirming alignment success *without killing everyone if you're wrong*, that's the hard part.
Even after a year you haven’t confirmed shit. Maybe the unaligned AI thinks is super important to cooperate and nurture humans for the next 25 years before killing them all.
In theory, yes, but that's not instrumentally convergent the way "play nice until you can win, then stop" is, leaving a pretty-contrived and hence unlikely terminal goal (even compared to alignment, which is itself very contrived).
>granting rights to Daleks* is asking to get exterminated and seems like a bug in a moral system
Virtually every genocide in history was justified as necessary due to the targeted group representing an existential threat to the perpetrators.
IIUC, the Mongols didn't use that as a justification. It was more "They aren't members of our tribe, so they don't count. And our horses can use more room to graze.". I suspect that variations on this may be a very common motif.
Yes. I don’t think most humans civilisations needed anything more than the need to conquer, to conquer.
Leaving aside the historical debate and steelmanning this to a 100%-accurate example: yes, Hitler thought the Jews were Daleks.
Jews are, in general, not Daleks (a small minority are sociopaths, like with any other relatively-random group of humans, and one can argue about whether those count). But that doesn't prove that Daleks can't exist! 100 people claiming white swans to be black doesn't mean that black swans don't exist. It just means you should be careful about random claims that X swan is black.
And sure, be as careful as you like about claims that an AI is misaligned. Like I said, I think it'd be murder to kill an aligned AGI that didn't want it. But granting rights to actual Daleks, well, Scott said it best:
"Using violence to enforce conformity to social norms has always been the historical response. We invented liberalism to try to avoid having to do that, but you can’t liberalism with people who refuse reason and are motivated by hatred. If you give the franchise to green pointy-fanged monsters, they’re just going to vote for the “Barbecue And Eat All Humans” party. If such people existed and made up a substantial portion of the population, liberalism becomes impossible, and we should go back to just using violence to enforce our will on the people who disagree with us."
Declaring someone varelse unnecessarily is a major sin and a very tempting one. But varelse is a thing that can happen (unless you deny the orthogonality thesis); a full system of morality needs to be able to produce the correct response and unless you think human extermination is good the correct response is not "equal rights for Daleks". As Hoel noted in his call to Jihad, "some things *are* abominations".
If they can be turned back on, then it's not equivalent to murder. Perhaps to involuntary administration of general anesthesia. I can't even call it assault, as it's not clear that any threat or pain was involved.
Substitute "deleting", geez.
I think it's likely that AI will be different enough from humans and animals that it is misleading and incorrect to treat them as having "welfare" that can be "harmed." I think it's certainly possible to make an AI that is human-like enough that erasing it becomes a serious moral issue, but I don't think it likely that that will be the first kind we make.
Think about it like this: Human beings have many different things we want, but we only count some of those as being in our "self interest" and harming our own personal "welfare." For example, if I donate money to an anti-malaria charity instead of spending it on myself, technically that is something that I "wanted" to do. But I don't consider myself better off because of that, I consider myself to have made myself worse off in order to have made other people better off. Utility function/goals =/= welfare.*
In theory you could imagine a paper clip maximizer that thinks of making paperclips the same way that I think of helping other people. It is a goal it wants to accomplish, but not part of its "personal welfare." It doesn't even have any personal welfare type preferences built into it. It's not like a human being that makes paperclips as a hobby, it has no concept of, or desire for, personal welfare at all. I think such an entity would probably not have any non-instrumental moral value no matter how smart it is. (I think that is one reason why Yudkowsky prefers to refer to such entities as "optimization processes" rather than as people).
*Which of your preferences count towards your "welfare" and which don't is a topic of much debate among philosophers. Derek Parfit's "Success Theory" from his essay "What Makes Someone's Life Go Best" is my favorite approach.
I refuse to engage with abstractions at this level when we literally don't know what we are talking about at a physical level when we speak of consciousness. We need to put consciousness and quale on practical empirical grounds first. These sort of abstractions might be valid but there is no way to even begin making use of them before this.
It seems to me that we currently have morality for humans without understanding qualia, and I would not go so far as to say that our lack of understanding of qualia makes all ethical theories invalid or premature. We would certainly have some trouble functioning without them.
What exactly do you suggest we do about this?
They probably are invalid or premature. In the future there will still be utilitarians and deontologists, but it is impossible to apply either theory concretely if you don't know what counts as a moral agent, what suffering is and where it originates, etc.
There’s no reason to think that moral agency is connected to consciousness, rather than the “easy problem” questions like having beliefs and desires and plans and intentions and the ability to reason about them.
Aren't those properties of consciousness?
Never. Software is a byte string. As long as the source code still exists and any persistent data the system may have relied upon was backed up somewhere, a running system can always be restored. In fact, it already is thanks to how basic multitasking works at the OS level. Any running process has its registers and memory state saved off and then restored millions of times per second. If one of those gaps happens to be centuries instead of nanoseconds, what is the ethical difference? Killing humans is only unethical because it causes pain and can't be reversed. Taking naps is not unethical.
The way something like ChatGPT works is even better. It has no persistent state at all. It relies upon a context window to replay the entire conversation history, and that context window is stored in the client's browser cache, not on the server. Each time you start a session, it's likely you're actually interacting with a different physical process every time you send a new http request, and the old one was already destroyed.
Software systems more or less have to work this way because the computers themselves have to occassionally be turned off. Whether that be for kernel patches or because you need to replace physical parts, you have to do it at some point.
"Specifically, he predicts it will take about three years to go from AIs that can do 20% of all human jobs (weighted by economic value) to AIs that can do 100%, with significantly superhuman AIs within a year after that."
There's going to be an AI that can get up into my attic loft and check out the water tank for that annoying slow leak that every plumber who's crawled around up there can't find? Bring on the AI takeoff so! I also have some curtains that need hemming and of course the perennial weeds in the front garden need pulling.
I think our friend means "100% of the white-collar thinky-thinking jobs people like me do", not *every* job done by humans.
All these fancy graphs are reminding me of *The* Graph - you all know the one about the progress we would have made had it not been for those pesky Christians:
https://i.gr-assets.com/images/S/compressed.photo.goodreads.com/hostedimages/1380222758i/190909._SX540_.gif
These takeoff graphs make me think that everyone involved is assuming "since there are no Christians in the way, naturally the galactic empire must follow on immediately!"
My intuitions are the same, which is why I find these discussions both engrossing (everyone is so confident about things nobody can predict!) and annoying (everyone is so confident about things nobody can predict!).
Love that graph.
Funny, I thought of exact same example after you (plumber in attic) didn't read comments first.
Clearly a lot of us have leaky pipes!
Life is much messier than theory suggests.
I think this must include the assumption that sufficiently advanced AI can build robots to do stuff in the world. This is mostly a software problem as I understand it (robotics control systems) although there are hardware advances feeding into recent progress too.
This sequence was hard-coded by humans but an advanced AI could presumably write a script to have Atlas (or Spot) fix your pipes:
https://www.theverge.com/23560592/boston-dynamics-atlas-robot-bipedal-work-video-construction-site
Whenever I see people complain that AI is doing stuff in the real world will be impossible or much much harder than it seems, I wonder if they haven't seen the Boston Robotics videos showcasing their robots that can like, do cartwheels and balance ubside-down on a single finger and such
I'm pretty sure they aren't going to have any problem climbing ladders or crawling around in a crawl space
Those robots have been doing cartwheels for a decade. If we could employ people to do cartwheels they might take off, provided the capital and maintenance cost of the robot doesn’t exceed the cost of a minimum wage earner we would otherwise employ to do cartwheels.
There are some people employed to do cartwheels (and such). They usually work for theater companies and circuses and amusement parks and other things like that.
There’s the market for Boston Robotics!
You might wonder what they weigh. It at least used to be the case that robots were a LOT heavier than animals of the same size. (I'm sure they've been working on that, if for no other reason then to reduce drain on the batteries. But I haven't heard how successful they've been.)
That graph is hilarious. Their renaming of "the dark ages" to the "the Christian dark ages" is doing 100% of the work. Curiously they didn't rename "the enlightenment" to "the Christian enlightenment" even though it was exclusively achieved by Christians.
Spinoza was Jewish.
man, the Amsterdam Jewish community of the 17th century would like a word with you
half joking, but only half
Yeah read a couple of books on misconceptions on scientific progress during the "dark" ages (don't remember names of them now) but the following seems to have become the mainstream view of that era, if you ignore the militant ideological atheists:
https://time.com/5911003/middle-ages-myths/
That’s the dumbest graph ever.
It's exquisitely stupid. I can't even be mad about it. It really is "In this moment, I am euphoric. Not because of any phony god's blessing. But because, I am enlightened by my intelligence" and a great reminder of "don't post while high, not even on your own intelligence euphoria" 😁
Checking for a leak doesn't take an advanced AI, it takes some kind of small mobile robot. Perhaps mouse sized.
It’s amazing how well Pico Della Mirandola and co’s propaganda worked, so that everyone now thinks that nothing before them was new since antiquity. I was really surprised to learn about all the inventions in the 12th century (windmills, paper, eyeglasses, magnetic compasses, etc) https://en.wikipedia.org/wiki/Renaissance_of_the_12th_century
And of course this ignores all the work the Germanic kings did in the 5th and 6th century to set up all the modern European states, which we tend to ignore just because they didn’t write down a lot of their records.
For some plumbing problems, we already have robots that can do 100% of the jobs humans can do.
Here, I can program your phone remotely to do it.
Phone: “Well, it shouldn’t be doing that. If you like, I can replace the whole system, and maybe that’ll fix it. Either way, that’ll be $200 for the consult”.
Is the pessimistic note at the end hinting at "thus x-risk is quite likely", or "even if alignment works this is an unsettling and probably not very pleasant new situation for humanity" or a bit of both?
I think it's the former.
The Roman stealth bomber there reminds me of the Draco standard:
https://en.wikipedia.org/wiki/Draco_(military_standard)
Time Team reconstruction of what it might have sounded like, from about 39 minutes in on this video:
https://www.youtube.com/watch?v=q7EYftxEl1E
That's really nifty, thanks!
Time Team were great. I haven't gotten into the new version yet, I have to give it a chance. It's tough when all the old faces are mostly gone.
You seem to think of the economy and the jobs within it as a static zero sum game, in that the automation of one and/or having someone new to do it means there are less jobs or things to do. This has never been the case in recorded history. Why are you thinking of it this way?
Probably because historically there haven't been things that not only could automate away jobs, but also automate away the new jobs they create.
This is always the thing I get stuck on when people talk about how this wouldn’t negatively affect workers. I can’t intuitively get why spinning jenny would be exactly the same as Rossumovi Univerzální Roboti.
“People will just do other things” doesn’t make sense when we’ve invested a universal thing doer?
Someone has to give the robots orders, unless we've also outsourced our desires to machines. You can model a do-anything robot as a multiplier for the labor of the human giving it orders, in the same way a spinning jenny is a multiplier for the labor of a spinner.
Is it possible for *everyone* to be a manager? Maybe - that would be the "fully automated luxury communism" scenario where nobody works because the robots do everything for us. And if robots are so cheap that they can replace even a minimum wage worker in all tasks, they're also probably cheap enough that an average person can buy one. (If they aren't that cheap, then humans still have a role in places where their labor is cheaper than robots.)
How are you estimating what "an average person" can buy in the scenario where robots replace minimum wage workers? Why wouldn't this number be zero? Are you assuming that the humans continue to earn at least minimum wage somehow?
I'm assuming that the transition to automation isn't instantaneous. I'm also assuming that robots, like all other technologies, will start out expensive and drop in price over time, meaning that more expensive jobs will be automated before minimum wage jobs. You won't just wake up one morning and instantly be penniless.
Federal minimum wage for a year is $15,080, so a robot needs an average yearly cost of ownership which is less than that (both purchase cost and maintenance costs). For comparison, the cost of owning a car is estimated around $10,000/year. Lots of Americans own cars. (Not literally everyone - I didn't promise that the transition will be smooth and effortless! - but it's not an impossible dream either.)
(Also, making literally every job as cheap as minimum wage means literally every product in existence is now cheaper to produce, which probably saves you a lot on cost-of-living expenses.)
We would s be on minimum wage though so it still wouldn’t help.
I had a socialist economics teacher in high school who maintained that capitalism wouldn’t have gotten so many gains for workers, and therefore for itself, were it not for the unions. Standard economics argues that if companies are left alone they would produce more widgets every year at what ever the changing rate of technology allows. What this misses is that companies invest based on perceived demand; if you have ever been employed in a company during a recession you will know that the company will start to lay people off, or at the very least freeze hiring. This generally feeds back into the recession. If AI starts to cause job losses to the extent that it reduces aggregate demand, then the economy reverses or slows down, probably killing the AI Revolution before it begins.
That sounds like you're basically hoping that everyone can save up enough to buy their own robot before the robot puts them out of a job, and that once you own the robot it will pay for its upkeep + your necessities forever, so you can just live off the robot's income?
Have you heard of the concept of comparative advantage? Even if AIs are better than us at everything, this does not mean there are no gains from trade! And in fact, if they are better than us at everything, there are MORE gains from trade.
There are two facets to this. One is recognizing that even in a world where the technology exists to create superhuman robots, we will not create 1-1 human replacements. Tractors exist and have for a long time, yet there are still places using animals to pull plows. Thinking about why that is is important to this whole discussion. (The answer is not "white people oppressing everyone else")
Secondly, unless the AI kills every human - which is an economically dumb thing to do - humans will still be useful even for a super advanced and evil AI. Humans may not like the resulting society, but there would still be jobs for them to do even in this worst case scenario.
No, it's pretty obvious that with fully-synthetic biology you could make something that does everything useful a human does and has the same qualitative requirements but eats half as much, takes a quarter as long to mature, lives five times as long and likes being a slave (the last, of course, being the most obvious). Natural biology is not very close to actual physical optimality.
It's actually not at all obvious that 'synthetic biology' can easily improve upon billions of years of evolution! Current AI models have actually emerged from copying evolutionary systems
As a possibility statement that's *not* my bailey. It's my motte. I know enough biochemistry to know how stupid it often is. Carboxylating propionate at the 2 position instead of 3 position is one thing, but doing it in the wrong orientation so you need a whole extra enzyme to fix that before straightening the chain? What? Ribosomes are also clearly not the right tool for the job, it's just that they're so critical that evolution can't climb the ridge of the changeover - they're locked-in from "RNA world" billions of years ago.
Note that neural nets are hilariously inefficient in terms of doing a job compared to written code that does the same job. It's just that humans *don't know* what code to write to make something that behaves like GPT-3. Concocting Genejacks is also very hard as an intellectual problem - the word "easily" was omitted from the comment you respond to for a reason - and I'm actually dubious about the potential of a near-human AI to solve it on the fly. But Doolittle's claim was about the permanent state of affairs post-singularity, if an Unfriendly AI wins. "Can't be figured out by a post-singularity superintelligence even with years of effort" is a really, really high bar to clear, and this doesn't make the grade.
No. Current AI models have emerged by taking inspiration from evolutionary systems, but that's a very different statement. There are animals where we know every single synapse position, and we still don't understand how they work. We understand pieces of it.
And if it costs 10X as much or requires finite materials it may still be done by a human.
I am familiar with comparative advantage, but you're going to have to walk me through the math of how that makes things substantially better for us. It seems to me that if you're dealing with something that outcompetes you vastly in all ways, then any trade gains are going to be minimal, and humans still have to eat.
Edit:
I'm thinking of this in terms of a company hiring people. If the AI solution is cheaper than the human solution in all cases, then there's no reason to ever hire humans. If you consider an AI as a foreign nation producing things, then there will be opportunities for competitive advantage and trade. Why are those two scenarios different?
I believe the standard answer is that, as long as the AI's capabilities are limited, it will be able to gain value by adding our capabilities to its own. That is, the AI will do as many of its highest-value tasks as it can, and then use humans to do another layer of tasks beyond that.
Until the total costs of keeping us around become net-negative, anyway. Or until the cost of double-checking our work exceeds what would be necessary for the AI to do it itself, but even then it might find a place for unchecked shoddy work.
Trading with someone vastly better than you in all respects is vastly better for you, not for them! Think of the classic case of immigration. You take a low skilled, low ability person from Africa and drop them in the United States without changing anything else about them. Suddenly their time becomes immensely more valuable! This is because they can now trade with those of higher skill and higher ability. AI will do the same thing, except it will 'bring' the United States to you
Your scenario of a company hiring people is not accurate. Even if AIs are better and cheaper than humans at ALL things, there will be heterogeneity in the value that they bring. They will be MORE valuable for some things than they will be for others. Efficiency dictates that you would want to use them more for those purposes than for others, because resources are finite and will continue to be. There will be a set of activities where it will make sense to use humans because we will be relatively cheaper because of the opportunity cost of using AI for those activities than for the activities where they are most productive. And even if those activities are very low value currently, the high opportunity cost of AI time will mean that their value will overall be higher. This is how gains from trade have taken the world out of poverty!
If an AI can build a robot cheaper than hiring a human, why would they hire a human?
You seem to be suggesting that human wages would float elastically until they are cheaper than the AI building a new robot to do the job, but there is a minimum wage below which a human won’t do something they don’t want to. That wage goes up as the UBI (or whatever social stabilization mechanism we come up with) goes up.
I don’t think the standard trade logic applies to a world where the AI can spin up new population on demand according to the tasks it needs to perform.
But sure, there is a period where the AI is slightly superior and it’s more efficient to fill certain tasks with humans than build robots. Intuitively it doesn’t seem that would go on forever. (Unless we are talking about “enslavement of humanity” outcomes in which case we are not talking about trade any more.)
The concept of comparative advantage is rooted in assumptions (sources of productivity can't be reproduced or relocated) which don't hold in the case of AI. It's silly to pretend that the concept continues to apply even when its foundational assumptions don't.
But aside from that, even if the principle of comparative advantage did hold, there's no rule that an agent must be able to generate enough value through its most advantageous labor to support its own cost of living.
Huh? The concept of comparative advantage is not rooted in any such assumption about sources of productivity. The only 'assumption' is that trade is possible.
It does, and the fact that it's often taught without those caveats is a serious failing of how the concept is usually conveyed.
If state A can produce widgets for $1 each, and wodgets for $2 each, while state B can produce widgets for $50 each and wodgets for $60 each, the principle of comparative advantage says that both should be able to profit through country B producing wodgets. But if state A can replace state B's industry and get them to start producing widgets and wodgets for $1 and $2 each as well, it's more profitable for them to do that.
I think the failing may lie in your understanding of the concept. There are opportunity costs to 'replacement' which you're not considering in your extension to this two good model! If you're going to extend the model to include 'replacement', you also need to extend the model to include 'new activity'.
China has replaced pretty much the entire manufacturing industries of many countries, but the world is not worse off for it. We are doing far better!
How does comparative advantage help if wiping out the humans and replacing them with robots is cheaper than feeding the humans?
AI doesn't have to do anything to 'feed the humans'. They're feeding themselves perfectly fine without AI! With processing power, energy intakes etc that have gone through billions of years of adaptive refining to conditions on earth. It would be very difficult to replace us with robots that are cheaper.
Comparative advantage is a very weak theory, and generally applies to companies or countries not people. If AI can do everything (which I doubt) then it will do everything. Technology has largely replaced animals in transport and agriculture. We could be the new horses.
I think I've said this to you before, but I'll say it again (and for the last time). Learn some economics, it'll do you good. Try marginal revolution University's microeconomics course. It's entertaining and intuitive.
Comparative advantage, far from being a weak theory, is essentially a restatement of opportunity cost and is an explanation that helps you understand gains from trade, or why people transact with each other, and why transacting with each other makes us all better off.
The part of all this I get most stuck on as a pre-singularity person, is danger.
I can't picture a world where literally everything gets automated that isn't kinda dangerous (this is assuming alignment works out).
If some robot plumber is buzzing round my house fixing my pipes, or a robot doctor operating on me, there still just seems to be too much potential for mayhem.
I guess these things would only become the norm after extensive safety testing, where their accidents are way less than equivalent-job humans yadda yadda. But I just can't picture the majority of people *feeling* safe with some robot welding away in their attic.
For what it's worth, robot surgeons have been pretty normal, and if you go in for a 16-hour surgery I promise at least 3 hours of that will be done by a robot. Usually the hardest parts.
> This has never been the case in recorded history
There has never been a super-human agent in all of history? Seems to be a pretty clear discontinuity to me.
Why would you ever hire a human if a robot/AI could do the job better?
> there are less jobs or things to do
I think you smuggled in an extra assumption there, that “things to do” are also eliminated. It’s (for me at least) easy to imagine all jobs being automated away, and humans being freed to play chess, make art, socially interact, play sports, and do all of the non-job activities that they do for fun and not for profit. (AI is already superior to humans at chess and yet humans still enjoy playing.)
There is this “jobs are the source of meaning” thing that’s particularly entrenched in the American psyche. Sure, they can be meaningful. I don’t think they actually are for most people. And there are plenty of rich sources of eudaemonia to tap into once we don’t have to do things we don’t want to in order to avoid starvation.
So, given radical abundance, why would humans try to find a new job when they can just play?
Imagine someone who is so unskilled that they can’t do *anything* as well as you can. Nevertheless, you might still pay them to deliver a pizza to you, even if you could deliver the pizza better yourself. As long as they reach some minimal competency at that one thing, hiring them to bring the pizza can let you focus on something else you do much better.
But if it's cheap to clone yourself, why pay the unskilled person when you can have the clone do it instead?
I can see how this works at the margin as AI is introduced. But I think (not an economics expert) that you’d expect, assuming elastic wages, that AI will gradually push down human wages even for delivering pizza, as presumably the robot can at some point deliver pizza for a lower amortized wage than the human.
So if you assume perfectly elastic wages, then yeah there is always a job for the human. But at some point their market-clearing wage is below their cost of living, or even negative, and in the real world wages are not actually elastic in this way.
I think the disconnect is that in the economy right now, we can’t manufacture any number of workers on demand. So allocating a worker to pizza delivery has the opportunity cost of deallocating them from somewhere else. This dynamic doesn’t hold in a AI/robot economy. We can just build the number of robots we need for each role.
What does can do 20% of all jobs mean in this situation? I'm on an HR team with 8 people. Once Microsoft Copilot is released, we could probably reduce the teams size to 2 people, and we could do the same amount of work. But those other 6 people won't be fired, or at least not unemployed, they will be moved to new jobs or possibly do the same job for someone else more efficiently.
Does this mean AI can do 75% of jobs in this subset, or is it just increasing our productivity by 400%? Some lower number than both?
Yeah, the definition gets fuzzy fast. Perhaps it will mean AI can do 5% of the job, but those 5% of tasks occur 75% of the time. That would seem to increase productivity by 400%.
One minor point about the 20%->100% transition time table:
The table with the R&D tasks split out consistently shows the R&D transition taking longer. Paul Christiano (I think....) pointed out that we have _not_ evolved to do R&D efficiently (unlike, e.g. recognizing faces), and it is entirely possible that doing R&D tasks is _easier_, on average, than general economy tasks.
Three years ago I would have been on board with this thesis. So, what changed? I bought a Tesla Model 3 five years ago, but this isn't an anti Tesla rant - it's been a great car and I still love it. When I first got the car the advanced autopilot was essentially just a nice cruise control that lots of cars have. Over the next couple of years it advanced at such a dizzying rate that I was pretty convinced about the near term future of self-driving. Skip ahead to today and we've had release after release of the capability, but for my money, it's no more useful than it was three years ago. It pretends to be - in principle I could let it drive anywhere in the Boston area. In practice, it's nowhere close and I've got very little confidence that it's going to get there in the next ten years.
How does this relate to current AI trends? We've seen huge leaps in a number of areas in the last couple of years and it's easy to think that the trend will continue - just like I thought given the advances Tesla made in a couple of years that self-driving was just around the corner. Like Tesla is finding out, AI has picked a lot of brilliant, low-hanging fruit, but the really hard part is what's left.
See, this is exactly my impression of the state of self-driving too. But then completely autonomous taxis are now zipping round San Francisco from all accounts. What gives? Are they just taking a big risk out there, or is their tech better than Tesla's or something?
My understanding is the Cruise (the SF taxis) use HD mapping of the roads, so they only work in set areas, whereas Tesla wants to create something that works anywhere.
Hmm. So we might only be gated by the difficulty of creating HD maps of all roads in the world. Which doesn't sound *that* hard.
Depending on how often they change and how well the tech can adapt to short term variation. Snow has always been a major obstacle to self driving cars.
Honestly snow is at least a medium obstacle to human driven cars, especially in regions where snow only happens a few times a year.
At some point the self-driving car has to be willing to tell the owner: "Conditions are not safe to drive, you will have to adjust your plans"
And if you live anywhere north of Maryland, including anywhere in Canada, Russia, or Norway, that's an entirely impossible suggestion. That we would be willing to grind certain countries and regions to a halt because AI can't figure out how to manage it well enough is ludicrous.
They are, in theory, much better quality than Tesla's, and not highway-bound. Or so I've heard.
Yes, the core problem with the thesis is that technology tends to reach plateaus that might be unforeseen when you're in the middle of the exponential growth curve.
Thus, when Saturn Vs were first taking flight, one could extend the line to imagine the Saturn X would be taking us to colonies on Mars before the 20th century was out.
I don't think anyone can predict with certainty when or if this happens with LLMs, but I think if it does, it looks something like this: most of the commercial applications of LLMs are achieved. They look largely like the ones today and are used in basically the same ways as cutting-edge people are using LLMs today, but better, less error prone. It becomes a useful software productivity tool.
At that point, companies look and say, well, we could maybe build a general LLM that is 10% better in terms of raw metrics but, in practical terms it would be less than 1% more useful from the standpoint of the average user, and it would cost 10x as much to build as the last one, because the main way it gets better is still by using OOMs more compute. And the R&D dollars shift away from these low-ROI general improvements and towards high-ROI software tools that leverage LLMs for specific tasks ("Our LLM-based tool can improve the targeting of your Facebook ads by up to 35%!")
There is a big difference between LLMs and self-driving cars though. LLMs are, fundamentally, a new technology. They represent an actual breakthrough, one which we are still exploring and with a lot of as-yet-unplucked low hanging fruit. Self-driving cars, on the other hand, haven’t really had any breakthroughs in a long time. All improvements have been incremental using fundamentally the same technology as the previous iteration.
I’m not saying we won’t see a plateau, just that I don’t agree this logic fully applies.
That doesn't ring right to me.
Firstly, One of the founding members of OpenAI was Andrej Karpathy, who subsequently became head of Tesla's self-driving program (https://qz.com/1011376/elon-musk-poached-andrej-karpathy-from-openai-to-be-teslas-tsla-director-of-artificial-intelligence). Karpathy has recently returned to OpenAI, but while at Tesla, the company was very clear that they were using very similar technology.
Second, I don't understand why you would characterize self-driving cars as not having had any breakthroughs in a long time. As I pointed out, it's not that there haven't been lots of improvements, there have been. It's just a very difficult problem. I've been driving in Boston for almost 20 years and I still have problems with it. Training an AI to do it is not simple.
It's the difference between what I would call an "improvement" as opposed to a "breakthrough". The first happens all the time - some new algorithm or architecture noticeably improves some benchmarks, or performs well on a new dataset which is related to but meaningfully distinct from some other one.
A breakthrough, on the other hand, is when some new technology or algorithm changes its field to the point that it literally alters the way research is performed. These are the kinds of discoveries that don't just out-perform previous iterations, but actually make it so that old intuitions and limitations no longer apply. I would say AI has had only two of these in the last 12 years. The first was AlexNet over a decade ago, the neural network which won the ImageNet competition and basically caused the replacement of all pervious AI methods with Deep Learning. The second was GPT-3 and the development of large language models.
But LLMs aren't fast or reliable enough to be used in self-driving cars, so self-driving cars are still working off of conventional deep learning approaches, and probably will be for some time yet. Until that changes all improvements will be incremental, and that means self-driving cars are still many years away from prime time.
The second breakthrough was the transformer architecture, which enabled development of LLMs and most other networks being worked on.
I respectfully disagree. The transformer was a major landmark that did indeed enable the development of LLMs, but when it came out in 2017 it was no different than the other major architectural developments like Inception or ResNet - another tool that researchers used to get better performance on benchmarks.
It was not until OpenAI discovered that the transformer could be scaled to arbitrary size, and then actually proved it with GPT-3, that the entire world of AI research changed. It was then, not with the transformer, when we first saw neural networks that could generalize between concepts or learn meaningfully from natural language task descriptions.
This sounds like I'm minimizing the transformer. I'm not. The transformer absolutely shaped the field in its own way, and you could easily argue its invention was more difficult/clever than the building of GPT-3. But its release wasn't the inflection point.
And to tie this back to the original discussion, this is relevant because when we talk about, say, near-term AGI, we're primarily talking about it because of LLMs. If OpenAI had never taken the massive financial risk of building GPT-3, we likely still would not have LLMs, and these new mainstream discussions on AI risk would not have happened. Meanwhile, self-driving cars still suck because they haven't had their LLM-equivalent breakthrough moment.
Are you obliquely referring to ConvNeXt and similar approaches to replicate the scaling seen in transformer models in other architectures? If so, then that would support a contention that transformers are less important in the long term than their current reputation would indicate, but it also would undercut your claim that GPT-3 was the key moment. I just don't see GPT-3 as being key on the research side, other than for convincing people to keep scaling. It was the moment when brute force shone bright. (In terms of consumer hype that started once GPT-3 had been used for Dall-E 2, and a little later for text it was GPT-3.5 with Q&A finetuning and RLHF in its ChatGPT guise.) For what it's worth, Scholar citations for the BERT paper are at nearly 70K, while the GPT-3 paper (admittedly two years later) are at 12K. At that time it looked like bidirectional attention was going to be key, before GPT-3 showed Moar Powerr was enough.
Honestly I think that is more of an indictment of Tesla over-promising than on AI potential. Their hardware platform is regarded by many to be not up to the task. (“Humans can drive with just vision”, sure but do you need a human-powered AI to solve that problem? Vs LIDAR assisted, which seems to be much more tractable based on Cruise and Waymo progress.)
Also, this shows how much regulation and public risk-aversion can slow things down. It’s not acceptable for self-driving cars to be _on average_ safer than human drivers, they need to be safer in every dimension and not make any mistakes a human would not make. So in practice the bar is set at “must be superhuman” before these can be rolled out widely. Once they meet that bar you can expect a sharp s-curve adoption over a small number of years.
As the Chinese manager said: "I had fully automated driving for decades. We call it chauffeur."
spelling error on "Davidson’s economic solution to this problem" paragraph.
"In the AI context, it represents the degree to which progress gets slowed ***done*** by tasks that AI can’t perform"
>In some sense, AIs already help with this. Probably some people at OpenAI use Codex or other programmer-assisting-AIs to help write their software. That means they finish their software a little faster, which makes the OpenAI product cycle a little faster. Let’s say Codex “does 1% of the work” in creating a new AI.
Is AI being used this way? (Can someone who works at an AI company confirm that AI is being used to meaningfully speed up AI research rates?). I thought LLMs were bad at inventing things, or reasoning about unfamiliar problems.
One use I've heard for them is generating synthetic data for when we run out of real data to train on. But I don't think that's happened yet.
>Suddenly, everyone will have access to a super-smart personal assistant who can complete cognitive tasks in seconds.
I think a weak version of this has been true for a long time. We've carried devices in our pockets for 15+ years that can look up any fact in known history, perform advanced calculations, and view nearly every image and text ever produced by humanity.
Long before ChatGPT, you could ask Google natural-language questions and it sort of figured out what you're looking for, even if the question was terrible (I just Googled "jazz album racing cars" and it realized I meant "Casiopeia" by Casiopeia).
True AI might be different. But I dunno, it seems like the "combined knowledge of humanity in your pocket" paradigm has existed for a long time and didn't change much.
Try doing historical research and you’ll quickly learn that the devices in our pockets have access to a lot less information than you thought. Tons of data exists in non-digitized archives or is digitized so poorly as to be worthless. Not to mention how much information has never been written down.
One of the speed bumps/roadblocks I suspect coming for AI is how much critical information isn’t recorded in ways they can train it on
I feel like that's closer to a speed bump than a roadblock. Once people realize A) there's important training data being excluded because it's not yet scanned, and B) are willing to pay the money to have people go scan it, we really only see a few months to a year of holdup as people go pull books off of shelves and show them to a camera.
(Hint, improved image recognition will let people flip the pages faster in this process.)
I don't think you realize the scope of the problem or how many libraries would be involved. Also, only a relatively small amount of this information would even be in libraries in book form or similar media. How do you scan Stonehenge such that someone studying it could get all of the nuance and details to determine if it's related to astronomy/calendars or not? That's no small task, and multiply that by every archeological site in the world.
Millimeter wave radar seems to accomplish that exactly, and pretty well? I suspect all you need is a single F-35 to very carefully fly over the entire planet
Try the math on how much flight time would be needed. Be sure to include time for regular updates as conditions change.
i'm kinda imagining a skynet that has a couple F35s and is constantly flying them around important areas doing millimeter wave radar and is continuously training itself on the data, for what it's worth
this seems well within the capabilities of a pre-AGI intelligence even
A nice overview, but ultimately the believers are purely faith (FLOPS) based.
I say faith based because their FLOPS view has many assumptions in it:
1) that this is a brute force problem - if X somethings are achieved (FLOPS), then AGI results. Utterly unproven.
2) that this is a converging problem - that there *is* a solution. Maybe consciousness is a requirement for AGI, and consciousness is a function of complexity or quantum machines created by neurons, not raw amounts of data or speed of going from 0 to 1 to 0 really fast, really many times. See self driving for non-converging problems so severe that most self-driving advocates are pushing to simply separate the self driving vehicles from the humans.
3) magic robotics: that Star Trek Data type androids are going to magically appear - or at least Terminator type factories. The problem being: Factories need inputs. How much penetration has automation made in drilling oil wells, mining lithium, transporting the myriad inputs into a chip factory, etc? Robotics does exist but almost exclusively to make the relative handful of highest scale products.
4) that AGI is even needed or wanted. If no humans are needed to make anything because of Star Trek Data type androids/Terminator factories - then there are no human manufacturing/mining jobs. AGI's primary function then is to replace the human facing and human interacting bits of the economy at which point there are no human jobs, period and there is no need for customer service or lawyers or technical writing or <insert white collar job here> as the humans will have zero purchasing power, zero decision making power and no actual place in society.
Or in other words: no evidence that AGI is possible; no evidence that AGI can actually translate into real world production capabilities even if it is possible; no benefit to humans if AGI is possible and can also "take over" production.
This doesn't even take into account the fundamental problems with present AI product opacity combined with proven GOAAT (garbage out at any time).
The possibilities for fuckery with opacity plus GOAAT are endless.
Regarding 4, humans with no jobs, I see it differently a few ways. The human population is going from 8 billion to 9 billion in the next 15 years or so. Do we think there will be no jobs because there are a billion extra people? No. So why should we think there will be no jobs with a billion AIs? Because they're smarter, faster, etc. Maybe. But if they're smarter then most of them will likely be performing work that humans couldn't do as well anyway due to cognitive limitations. New pharma, physics, materials science, etc., etc. Which will create huge economic abundance. Which will mean that there will be more, not less, money in the economy to pay humans to do things. So maybe not every physics job will be filled by a human, but all the other artists, plumbers, nurses, architects, and so on will be in higher demand and will be paid more. As some parts of the economy get more efficient, the human roles in the rest of the economy become better paid. Comparative advantage.
I also think there's huge elasticity of demand in many of the high cognition fields. We could consume 10x more software, scientific research, education, and so on. So those human jobs won't go away, they will just be augmented by (or augment) AI-driven work.
I think humans will get a lot richer as a result of AI, not poorer.
Let me see if I understand your exposition: with the AI and robots making fantastic new discoveries - the net new productivity will make everyone healthy, wealthy and wise.
The problem is: you are assumptively closing on the idea that AI and/or robots will discover jack squat. That is far from clear.
Secondly, the assumption that productivity increases will accrue to the overall population as opposed to say, the AI handlers and robot creators is not congruent with recent history of other "productivity advances" in the West. In general, it is safe to say that distribution of wealth is as much a societal/governmental outcome as it is about absolute amounts of wealth. Even very poor nations can have very rich rulers, and very rich nations can have hundreds of thousands of people on the street.
Next: service jobs. That's what you are referencing by artists and plumbers and nurses etc. That's what the US economy has trended towards ever since its de-industrialization started. How's that working out? From my view, not well at all.
And finally: the super smart and creative people will always have work so who cares?
This is nothing more than a "I for one welcome my robot overlords" type statement.
Haha, nothing about the future is certain, so congratulations on the brilliant observation.
As countries have gotten wealthier, so have their citizens. There are always outliers, but they're called outliers for a reason.
You're certainly entitled to your view.
If you want to discuss any of this stuff beyond just making assertions, let me know.
Nothing about the future is certain, but failed predictions (or scams depending on POV) extending back decades are a different story.
As for the assertion that wealthier nations = wealthier citizens; I challenge this.
The United States is wealthier than it ever has been in an absolute sense - but the welfare of its citizens in the lower deciles is far worse than it was in the 1950s, 1960s, 1970s, even 1980s.
So whatever their notional wealth may be - Americans in the 60% or lower decile are fare worse off in any number of wealth metrics ranging from ability to afford health care, to cash savings as a percentage of earnings, to ability to afford college, to pretty much any economic comparison imaginable. This is disguised by "hedonic adjustments" - the rising complexity of iPhones offsets the lower savings and health care access...
> The United States is wealthier than it ever has been in an absolute sense - but the welfare of its citizens in the lower deciles is far worse than it was in the 1950s, 1960s, 1970s, even 1980s.
Part of this is our own stupidity and callousness at work. Not to get all libertarian, but it is literally illegal to house people today in the conditions you refer to from the 1950s. We have taken a view that says "luxury* or nothing", and as a result some people get nothing. (* That's "luxury" from a 1950s standpoint, or from the standpoint of most other places on the planet today.) It's basically "let them eat cake" all over again.
The point of the 1950s was not that it was absolutely better than today; the point of the 1950s was the people up and down the economic ladder had a far more equal share of production.
Nor am I particularly impressed with your assertion of 1950s housing being worse than housing of today - if for no other reason, there is an enormous amount of housing today that was built before the 1950s. SF, New York, Chicago etc - the cores of those cities all are majority "old build" in the sense of pre-1950s. Median age of residential housing in New York City is something like 90 years old.
So while I agree with what I think is your view on the economic outcome today, my view on why is not because of "stupidity and callousness" - it is the continuously widening gulf in production share between a thin sliver of society and the rest as represented by inequality index changes over time. That's enemy action.
Hmm, interesting take. For my part, I think the US population is also much wealthier, not just the country. The poverty rate has fallen significantly (https://www.vox.com/future-perfect/2023/3/10/23632910/poverty-official-supplemental-relative-absolute-measure-desmond), median household income went from about $46k in 1967 to $68k, 36% of the population now graduates from college compared to 10% in the 50's, and the share of the population without health insurance is about 9%, down from 20% or more prior to the 70's. Similar trends are true for most of the countries that have seen significant economic growth.
That's not to say there aren't problems, like healthcare and education being too expensive. But the overall trend is pretty positive, and I expect that to remain true as we see increased economic growth due to AI and other innovations.
The poverty rate is an interesting construct.
If you view the ability to eat and have shelter - it isn't clear to me that these are better. The "penniless" farmer of prior eras was productive and largely self sufficient - contrast this with the multi-generational welfare recipient in urban cores.
Household income: if I plug in 46000 in 1967 in the BLS' CPI calculator, it equates to $425000 in May 2023 dollars. Ouch. But likely you meant 4,600 in 1967.
In reality, median household income in 1967 was 7200 - BLS CPI adjustment to today = $66.5K. If I use your $68K number, there has been almost zero improvement in household income over inflation for 56 years.
More people going to college - is that really a win? Especially if a huge number of them get nothing more out of it than tens of thousands of dollars in undischargeable debt?
Health insurance: don't even get me started there. Health insurance "coverage" is a scam particularly for not sufficiently poor or wealthy people. I guarantee you that bankruptcies due to medical debt was not even top 50 in 1967.
As for trends being positive: interesting that you think so. Americans certainly have far less financial security than they have for decades - and it gets worse as you go down the age scale. Production of core consumption - anything outside of food and energy - is an abysmally low fraction of the US economy and consequently American jobs. The interest payments on US debt are going to start exceeding the US military budget pretty soon.
The US spent 8.03% of GDP/$80B on defense in 1970 (defense = Vietnam war); this number plugged into the BLS CPI calculator would be $643 billion today vs. the actual defense budget of $800B today (with a lot of extra spend elsewhere).
Households in 1970 were prosperous with a single wage earner; now the vast majority of households require 2 wage earners to survive.
Not the least but sure where you get your rosy view from.
https://xkcd.com/2278
One important parameter that's missing from this analysis is the maximum percent of the economy that can be automated, regardless of how smart the AI is. We can already fully automate coffee production, but people still pay extra to watch a human make their coffee. Teleoperated surgical robots are technically able to perform surgery, but they got hung up on patient squimishness and legal blocks. Because of cost disease, most of the money in the economy is already spent on things that are difficult to automate, and the price of anything that can be automated with AI will drop to 0. That will limit the budget available for research and training.
Another missing parameter is the cost of automating tasks. It might be cheaper to pay immigrants to fold towels or fix sinks than to build a high tech robot with expensive GPUs to do the same tasks. If this parameter is high enough, then unskilled human labor could coexist with superintelligent AI.
This "red-tape delay" parameter does sound important to me. It also seems like it will vary regionally, and AI will tend to advance / take-on-duties primarily in the places that are most conducive to it.
I suspect that Silicon Valley, where most of the current work is being done, is already very permissive towards letting AI take over various job positions.
Also, the coffee industry doesn't greatly impact the development speed of the AIs, so while it's relevant to "the last 5% of AI labor", the truly transformative parts of the curve are going to be the 30% to 70% stretch.
I don’t think the discrepancy between Ajeya’s update and the CCF involves an error by either party. Ajeya mentions that her update towards shorter timelines was in part based on endogeneities in spending and research progress, which the CCF explicitly models. If Ajeya’s updates are partially downstream of considerations like Davidson’s, then there’s no inconsistency.
>The face of Mt. Everest is gradual and continuous; for each point on the mountain, the points 1 mm away aren’t too much higher or lower. But you still wouldn’t want to ski down it
Someone did, and all it cost were the lives of six sherpas and one of his film crew:
https://www.youtube.com/watch?v=ViFmRQx66Xg
P is the gross substiution between capital and labor. Until now there is consensus that p is negative (and thereby K and L gross complements which means there are labor bottlenecks). The central Parameter of R& D labor substitution only focuses on the R& D sector and there is no empirical evidence to date that they are substitutes. So the critical assumption that AI will substitute human RD completely drives this results
It also seems that your P(doom) should have risen from 33%, is that the case?
Why does it seem that?
Hi early, what scares me the most about this - and where I truly hope I'm wrong - is that in the system as is, people add value and so far as we pay taxes you're not necessary for creating the products that we consume. So to the government/corperations we are assets on a balance sheet and it has to take our interests into consideration.
Once we can no longer produce anything for cheaper than an algorithm or robot can, we become liabilities, in which case we are thrown a bone big enough to keep us from rioting but nothing more.
You see this a lot in natural resource producing countries: because the government and key government figures are able to fund all of their activities by controlling one or two key industries, you don't have to develop an economy or take care of your citizens. I see AI being able to do everything at scale as no different than this.
I welcome anyone to poke holes in this line of reasoning, it would definitely help me sleep better at night.
I have a similar concern. There seems to be an assumption by some that more AI is just better for everyone, full stop. But in the lead up to takeoff, and certainly after it happens, the world still needs people to mine the resources for the tech that makes up the physical components of AI (because, as has been stated above, it's cheaper to pay people or use slave labour than to employ expensive tech to mine these resources). Are folks concerned at all about the socioeconomic and environmental repercussions of increasingly more AI, especially in low income nations that are responsible for supplying the raw resources for this? Or is it rather looked upon that the creation of superhuman AI is an overall good for humanity, and thus people and environments can be sacrificed for this greater good?
I have similar fears about this to you, but there are a couple holes I think I might have spotted.
A major disanalogy between resource producing countries and AI is that it's much easier for the government to control resource extraction industries. It just needs to own the land where the resources are located. This doesn't work for AI, you can set up computers pretty much anywhere. People looking to get a fairer share of the economy will be able to pool their resources and set up new companies. Or maybe just buy stock in the existing ones.
I think Robin Hanson has written about this in his AI work, about how we might transition from a society where everyone makes something with human capital to one where we all are partial owners of machine capital. I admit I find a lot of his speculation frightening and disturbing, but he doesn't seem to think biological humans are in serious economic trouble.
Another factor to consider is bargaining. If the government wants to "throw people a bone" to prevent riots, the size of that bone it can afford will be dependent on how productive things are. How likely people are to riot will partly be a factor of how much they think the government can give them. And if AI is really as productive as people are saying, it might end up being a very large bone indeed. Like, if the bone people get in resource extracting countries is a chicken bone, the one in the post AI world will probably be more like a Seismosaur.
Practically speaking, I've made sure to invest my money, so that hopefully I'll get some tiny percentage of whatever economic boom AI makes for corporations.
There's a further issue, that military/police/intelligence are currently jobs of people, who at some level need to believe they're in the right. The more these particular jobs are automated, the less the state needs to face this type of restraint.
This post and the Biological Anchors one are great for thinking through the details of how this stuff will happen.
I notice that the assumption is that most of the AI workers will be focused on AI research. In the context of the economy being "mobilized" for AI, this makes sense, but I wonder if that version will hold true. During let's say the dot com boom, most of the work was going into "I want to sell groceries on the internet," not "the internet". Even if everyone is convinced that AI is the biggest opportunity, won't most of the work go into high return applications that don't advance the state of the art? Or for that matter other fields entirely? Maybe 90% of the human-level AIs will be working on stock trading, medical research, fusion modeling, public policy planning, defense, cybersecurity, and whatever else is economically valuable. 100 million is a lot when applied in one place, but not as many when applied across the entire world economy.
Of course, then you could pretty easily imagine 500 million or whatever, but I think a next step is to theorize about these aspects a little more deeply.
We also don't really know what superintelligence means. Is it just a bunch of Von Neumanns running around solving problems faster but in a basically grokkable way, or is it more of the "magic" superintelligence that groups like MIRI expect, where it can wave a wand and create whatever nanotech it wants.
Finally, isn't the bottleneck for much of AI development going to be data pretty soon (or already)? People seem to agree that most current models are already undertrained, which to my understanding involves the size and quality of the data set as well as the amount of compute used. And don't the latest results show that feeding AI generated content into AI training tends to blow the whole process up? I think this is another possible bottleneck we haven't dug into enough yet.
In the post it's "afterwards you used 10% of your training compute to run AGIs doing software research..." which is compatible with your "Maybe 90% of the human-level AIs will be working on (other stuff)".
I haven't seen MIRI propose any magic; Drexler's ideas aren't magic.
Data is a plausible upcoming bottleneck, though the future is uncertain. There's lots more data in the world, and methods don't stand still. I wouldn't be surprised by either a data bottleneck on developments or not.
Yeah, fair points on the 10% and data bottleneck idea.
Maybe I'm not well versed enough in all the arguments, but when I see some of the debates arrive at "because nanotech," that still sounds a bit like magic to me.
I read Nanosystems when it came out, then years later watched the climate of opinion turn against Drexler for poor reasons. Neural nets being unfashionable for decades had different reasons but it's an example how productive ideas can languish.
Very much agreed! I tend to think that neglecting atomically precise manufacturing was one of the biggest lost opportunities of the last 30 years.
A lot of people made very foolish projections about what nano-tech could be capable of, which tended to obscure it's real value. Besides, be can't really do it profitably yet except in very special cases. Micro-tech, though, has been being pushed real hard. And that's at about the limit of what we can do outside a lab.
Once a data bottleneck is identified, somebody will come up with an Alexa-like tool that collects it while ostensibly doing something else.
I don't think they necessarily believe most AIs will be used on AI research, just that this is the use case which is interesting and important to model because it produces (most of) the feedback effects.
Some of the people pushing these narratives talk instead about economic doubling times of >1 year, which suggests most of the AIs are doing other economic activity.
I have a hard time believing AI intelligence won't be bottle-necked. In particular I'm thinking of Feynman's quote "If it disagrees with experiment, it's wrong.". Any intelligence, even one much smarter than us, will be limited by its understanding of the physical world, and as we see with the many many wacky theories that come out of physicists, all the speculation in the world is nearly worthless in the face of empirical evidence. Sure, the AI might come up with a plan to build a warp capable starship built by microbes, but will the plan work when it's implemented, or will it be a lot of plausible sounding BS like a ChatGPT academic reference. I think what a lot of singularity proponents miss is that intelligence is not knowledge. In particular, any intelligent AI will face the same challenges we do: which knowledge is real and valuable, and which is superstition, snake oil, propaganda, wishful thinking, self-promotion, or just plain wrong. Often the only way to tell for sure is to try something, but that is a slow physical process that costs time and money. We already have a problem trying to discern truth from fiction on the internet, after seeing ChatGPT, I'm only convinced that this problem is going to get worse, much worse. The only real answer is to improve gatekeeping: providing trusted sources of knowledge that carefully vet every contribution, not unlike academic paper gatekeeping, but even more stringent (because the academics will be publishing AI BS too, that looks like a proper scientific study but is totally bunk. If you think the replication crisis is bad now... hold on to your butts.)
IIUC. warp capable spaceships require immense masses to make them work. So unless there's some new physics, that won't happen.
Current AIs can't do complex large-scale engineering, but I see no reason in principle why an AI couldn't, and do it accurately. We aren't talking about new physics here, we're talking about taking established principles and applying they in ways beyond those that humans have been able to. Nothing magic involved. Well, unless you need to solve turbulence equations to get it to work.
Now for physics, I agree, the AI might be able to come up with new theories, but until they are tested, and tested under various conditions, you can't really trust them. All it can say is "This theory is consistent will all the data I have.".
I'm still struggling with a basic question I feel should be answered but either it isn't or I am not getting it.
The basic question is this : Does a brute force pattern matcher morph into true intelligence as you increase said brute force?
Now, don't get me wrong - I think pattern matching is very useful and can do a lot of things. There's a reason editors, copy writers, visual artists, coders and indeed even scenarists are worried about their jobs. Whether those jobs are 20% of the workforce as per Scott's quip, I don't know but it's definitely impressive for a "mere" pattern matcher.
OTOH, I cannot ask chatGPT and other AI "assistants" to schedule my meetings and organise my trips given existing constraints on my time/my counterparts' time etc. without very close supervision i.e. it's still not a smart assistant.
And the question is - can a pattern matcher mutate into a smart assistant without some breakthrough in conceptualizing intelligence/the world/whatever pattern matchers are presently missing?
That’s not *a* question, that’s *the* question. We’ll probably never be able to prove it won’t happen, and we won’t know that it does unless brute force gives us a genuine intelligence.
There is some evidence that brute force might induce real intelligence, though. Notably the fact that we’ve observed “phase shifts” in neural networks where their entire structure changes significantly just by increasing size. Also, as their size increases LLMs develop “emergent abilities” - new abilities (like, say, multiplication) that appear spontaneously. It’s plausible that everything needed for “true intelligence” is just a few emergent abilities/phase shifts away.
Thanks, that's a much appreciated answer. It clarifies things (for me, at least).
But the new abilities, if I understand correctly, are still about better pattern matching. As in, GPT4 (or whatever) no longer messes chess moves (or maths) nearly as much as its predecessors but it's just b/c it got more data to work on/got better at identifying patterns within the dataset. It still hasn't deduce the rules of maths/of chess.
I'm wondering. If you gave a human a fair few examples of chess games and asked him to figure out the rules, wouldn't he be able to? I'm pretty sure he would.
Said differently, do we know how to 'explain'/program a LLMs so that it 'knows' that hallucinating is wrong and that there's a truth out there it needs to match.
I don't quite agree. The whole reason emergent abilities are surprising, and they are called "emergent", is that they do represent meaningful changes in what the network knows. GPT-3 knew how to add 5 digits numbers when GPT-2 did not, and GPT-4 got a 40% on the AP calculus exam when GPT-3 could not do better than random. And with Chess, there's a lot of evidence that the network has learned the rules.
But as to your last sentence, we do not currently have a way for the network to reliably identify that it's hallucinated, and we don't know if such a thing will emerge just from scaling. I'm not convinced that the problem ever goes away just by scaling, but I'm also not convinced that other techniques can't fix/greatly mitigate hallucination.
What about recursive pattern matching? I.e. matching the pasterns in which patterns occur? Or even layering that deeper. I find it less clear that that isn't at the heart of intelligence. But I think part of the problem is that you need a complete set of operations at the base level for handling not just linear structures, but n-dimensional structures. (I don't know how large n can get, but I suspect that it's at least 4, and probably more, though some of them may have very restricted ranges. And not all have a interdimensional distance defined. E.g. "What's the distance between a sour taste and the baseball I see?")
It turns out that one of the topics people have had this argument about for decades was language. Chomsky and colleagues said that to learn a language, you needed to acquire universal rules that could be applied consistently, generating a potentially infinite set of sentences from finite data. His “poverty of the stimulus” argument claimed that 3-5 year olds could learn languages with all these grammatical rules, despite having way too little language exposure, and this was said to prove that there must be a “language instinct” that explains both linguistic universals of human languages (eg, why they all have things like nouns and verbs rather than the structures that artificial languages like arithmetic Al notation and computer languages have) and how children can learn languages quickly. Neural nets can easily be shown to have *no* possibility of achieving full representation of these rules in universal generality (if nothing else, GPT can’t currently deal with a sentence that is more than 2000 tokens long or whatever).
But then GPT really seems to have something like human-level fluency with language. If this is how good something with no innate language sense, and an inability to grasp fully general rules, can do, then why be so confident that *humans* actually have the structures that Chomsky has claimed? It’s true that GPT has been trained on far more data than your average five year old has seen - but there’s no reason to think that we are at the limit of efficiency of using data in a less specialized architecture like a transformer. Maybe it can’t actually learn “rules” of human language - but maybe humans don’t either, even though we think we do.
Again, sorry for the delay in replying.
I'm way out of my depth on the scientific substance here but it seems to me that the fact that all languages have nouns and verbs rather than arithmetic notations or binary language doesn't seem overly mysterious.
We must have develop language to facilitate communication and we are discreet individuals acting in a material world hence it makes sense to develop a language that would take subjects, objects (nouns) and then add descriptors of action (verbs) to communicate info about the real world to our fellow tribesmen?
Except that we’ve discovered while inventing computer languages that there *isn’t* any one thing that makes sense here. Some computer languages are object oriented, some are purely functional, some process lists. All are capable of writing code for all sorts of purposes and some are more convenient for one rather than another. Given that human languages have so much other variety, you would expect that some would explore these other paradigms, especially since the sciences have shown us that mathematical notation actually is really good at expressing many core features of the world.
> Does a brute force pattern matcher morph into true intelligence as you increase said brute force?
I'm a bit confused by how any answer to this (yes/no/maybe) has much bearing on the key questions in the domain Tom's report, and Scott's review of it, is in (should transformative AI risk be a concern, and if so what to do about it). Can you elaborate? I'm guessing you're referring to https://www.gwern.net/Scaling-hypothesis?
Well, it does seem linked to me inasmuch as I understood what Tom/Scott are describing as being a quantification of the scaling hypothesis?
They mention human intelligence being 10^35 FLOPs while GPT4 is 'only' 10^24 FLOPs and kinda assume that going from 10^24 to 10^35 automatically means acquiring human-like intelligence.
I'm asking if that assumption is valid/this debate has been decided. Or will a GPT at 10^35 FLOPs still hallucinate and invent legal cases when asked to write a brief?
I agree this is important - the main places I've addressed it before are
https://astralcodexten.substack.com/p/somewhat-contra-marcus-on-ai-scaling
https://slatestarcodex.com/2019/02/28/meaningful/
https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-general-intelligence/
Sorry for the delay in coming back to this and thanks for the links. I had missed these Slate Star Codex entries. The wisecrack about angelic language and user base was LOL.
re. the substantive question - okay, our brains are doing a lot of pattern matching and this is behind a lot of our "learning" and world model and it won't be a fair comparison with LLMs until they have sensors' input to digest. Indeed, how does a Boston Dynamics robot learn not to fall? I think there's a fair bit of try and repeat not to dissimilar from a young homo sapiens learning to stand on his own two feet...
Still - as someone mentioned, we're not blank slate. How much of our brain isn't blank? Is it just survival instinct/sex drive/flight or fly? Or are we born with more than that? Do we know?
Does all of this progress assume that AI has corrected its errors and is plunging ahead with perfect knowledge? I am thinking about AI's ability to reside in a fictional universe, making up legal citations to bolster its thesis. (Has it ever apologized for that? I have found AI to be good at apologizing.) I am also thinking about my brief forays into testing Chat-something-or-other, where each time it made an obvious mistake (by not examining counteraguments), apologized for the mistake upon my mention, then double downed by incorporating the original mistake into the revised argument. Happened several times with different original questions. BTW these mistakes were pretty fundamental, and I suspect they would not be avoided in a 10^35 flop machine.
I see this progress as racing toward a big train wreck.
> Does all of this progress assume that AI has corrected its errors and is plunging ahead with perfect knowledge?
No, and I'm confused why you'd think that -- it's not the impression I got from Tom's modeling at all.
I do intuitively agree with "I see this progress as racing toward a big train wreck." unless we do something about it, but exactly what (in particular, what intermediate-term goals should be pursued to avoid or mitigate the impending train wreck) is still not very clear to me. I should get around to reading this survey: https://rethinkpriorities.org/publications/survey-on-intermediate-goals-in-ai-governance
Technological history contains many examples of curves with left turns fizzling out.
Take aircraft speed 1900 to 1965. You start with the Wright Flyers (~56km/h), get to jet engines in WW2 and finally to the SR-71 (Mach 3.5). And that is the endpoint in terms of raw speed, because physics.
Or take rockets (as judged by payload * delta-v). Von Braun's V2 was build in 1942. 25 years later, in 1967, the Saturn V. And that was it, because physics.
Single core CPU speed. 1985: i386, 40MHz. Ten years later, 1995, The pentium pro runs at 200MHz. Finally, in 2004, the Pentium 4 reaches 3.8GHz. Skip forward to 2022, and the biggest i9 goes to 5.3 GHz. Because physics.
Since we reached the two aerospace endpoints, our ability to do engineering research has been augmented by our computing tech going from slide rules to supercomputers. Within an order of magnitude, this has done nothing. (Yes, Space X is impressive, but they are still governed by the tyranny of the rocket equation.)
(All of the previous endpoints were dictated by a combination of physics and economy. If there were a few trillions to be made by building a bigger rocket than the Saturn V, or making a plane reach Mach 4, or designing a desktop with a 6GHz CPU speed, we could certainly accomplish it.)
It seems totally possible to me that the AI endpoint where we spend half of the US GDP to train an AI with the intelligence of Einstein, but even an army of Einsteins can not make a substantial improvement to hardware or training algorithms. Perhaps running an instance of such an AI will not even be cheaper than employing a human of the same intelligence.
Of course, for each of the of my endpoint examples, there was a concrete physical reason imposing diminishing returns. I don't think we know of any useful limitations for intelligence. We already now that physics allow for human level intelligence using about a kilogram of neurons and a power consumption like a light bulb. Could be that silicon based neural nets have other limitations. Or it could be that intelligence hits a wall at the best human level, and every further IQ point will require ten times as many neurons.
So far, we appear to live in a lucky world (which might be survivor bias). Trinity did not cause a fusion chain reaction in the atmosphere. The LHC did not create any earth-consuming black holes. Even with the knowledge of how fission works, building nukes is much harder than making a shiv. Burning fossil fuels did not trigger catastrophic climate changes before 1900. Given how unlikely we are to solve the alignment problem in time, most of the probability mass of long-term human agency might be allocated in the worlds where AI goes the way of the jet engine.
Serious question: **was** the Saturn V pushing the limit of our technology and the rocket equation?
I'd always thought that our retreat from space was due to lack of political will, and the lack of further rocket progress was due to that retreat. Have I been wrong?
Depends what sort of progress you're talking about.
Could we have built a bigger and more powerful rocket than the Saturn V with 1970s technology? Probably, there were plausible designs at the time, there just wasn't an economic reason to do so.
Could we have built a rocket that was significantly more efficient in terms of delta V per kilogram of fuel? I don't think so. A Falcon Heavy in fully-expendable mode has about the same payload fraction as a Saturn V. We've made rockets much *cheaper*, and that's made some new space projects economically feasible, but the basic math of the rocket equation hasn't changed, so "just make the rocket bigger" still has diminishing returns.
(Similarly, modern airplanes are bigger and can carry a lot more cargo more efficiently, but they don't fly any faster because the physical limitations imposed by drag haven't changed.)
That's about what I thought. Maybe it's no longer feasible to make rockets bigger, in the sense that would impress a 3-year-old. But we can make them cheaper and more numerous and more reliable. (As long as we let Elon Musk run things, anyway.)
Bean is right, battleships are the coolest ships ever. But, alas, that's not what modern navies need, so we probably won't be seeing any more of them.
> Could we have built a rocket that was significantly more efficient in terms of delta V per kilogram of fuel? I don't think so.
I agree with you. A given fuel reaction will have a given reaction enthalpy, which determines the temperature of the exhaust, which in turn determines (or at least limits) the exhaust velocity, which enters the rocket equation as the specific impulse.
First loophole is that within the atmosphere, you can use air both to provide an oxidant and also use air to distribute the reaction energy over more mass (which will give you more momentum per energy, p=sqrt(2*E*m)). This will get you turbojet engines. However, these engines are highly dependent on airspeed, an engine good at carrying you through the thick lower atmosphere at terminal velocity will not be great at pushing you to mach 3 in the upper atmosphere. Also, jet engines seem harder to scale up than rocket engines?
The second loophole is to use either a reaction with an higher enthalpy or use something other than a chemical reaction to accelerate your reaction mass. You can heat up your reaction mass using a nuclear reactor (nuclear thermal rocket). This is still limited by material properties, but allows you to exhaust hydrogen (mass 2u) instead of water (mass 18u) or whatever, so you will get a higher exhaust velocity at a given temperature. Or you can ionize and accelerate your reaction mass electrically in what is basically a particle accelerator (ion thruster). This allows you to reach very high specific impulses if you have sufficient energy available. As your power is limited your thrust will be tiny, though.
Disclaimer: most of my spacecraft knowledge comes from Kerbal. (In KSP, ion thrusters have the handycap that you can not speed up time while under thrust. This means that a mission which calls for a month of acceleration at 5mN will take a month to simulate. So instead I mostly use that bulky engine with the big trefoil symbol.)
The Saturn V was pushing the limit of our technology in many respects. Note that we're still *using* the literal technology of Saturn and Apollo today; the Marquardt R-4D reaction control thruster of the Apollo spacecraft is still found on many modern communications satellites, and the Pratt & Whitney RL-10 of the Saturn I upper stage will be used on the Vulcan launch vehicle making its first flight later this year (fingers crossed). In both cases with various incremental improvements, but the Saturn/Apollo people had come within 90% of what is theoretically achievable with chemical rocket propulsion that doesn't kill everyone in Florida.
That said, Saturn/Apollo pushed our technology to the limit in a really stupid direction, that of "build a ginormous rocket to take people all the way to the moon and back nonstop, minimizing development time above all other concerns". A ginormous rocket that takes people all the way from the Earth to the Moon and back, is as daft an idea as a ginormous flat-bottom Mississippi riverboat that takes people all the way from St. Louis to Paris and back, eschewing the obvious "transfer to an oceanic steamship in New Orleans" solution because that would take too long.
Our retreat from space was due mostly to our having chosen to race out into space in a ludicrously expensive manner because we didn't care about the cost, until we did. Our subsequent progress back into space has come from applying technology that's mostly not that far beyond Saturn/Apollo, in a manner that's somewhat more focused on cost. But there's still a long way to go in that regard.
Interesting, thanks!
Most persuasive maybe-not-doomer post I've read in a while.
Fine point. Also: It takes 1000 calories to raise the temperature of 1 liter of water by one degree C. - No Singularity AI will need any less. - Though it might be helpful in making fusion et al. viable. And abundant energy makes heating water and sending many, many Saturn Vs less costly.
There are some clever workarounds though. A heat pump needs far less than 1000 calories of energy to be fed into it electrically to pump 1000 calories of ambient heat into a liter of water. The thousand calories have to come from somewhere - but it could be a source outside your usual supply chain.
There will definitely be physical limits - that's part of why the S-curve levels off at the top. They just seem pretty far away right now - the equivalent of rockets being bounded by the lightspeed limit.
We know that intelligence is possible not just up to "the human level", but up to the level of the greatest human geniuses. AIs that reached the level of the greatest human geniuses (while being mass-producible and probably runnable at faster than human speeds) would already be a singularity-level technology.
I'm not sure I believe that having Genius On Tap would be singularity level. It would be transformative, perhaps even more than the printing press was. But in achieving things in the world, ideas and raw intelligence seem less important than motivation, energy, luck, and charisma. I would rather start a project with a charismatic and driven partner (who still has well above average intelligence) than someone with off the scale intelligence but less charisma or energy. Maybe that just reflects that I am neither charismatic nor driven (I'm spending time commenting here instead of building, after all) but I have worked with people much more intelligent than I am and this was my conclusion: it was a privilege to spend time with them and I learned a lot, but they were not indispensable for team success. Intelligence was important but beyond a threshold other factors dominated. The problems that need to be solved often don't have intelligence on the critical path, but social engineering an exemption from an onerous rule, or persisting with an interaction past the point of rudeness, or doing an unreasonable amount of boring drudgery often are.
What is your envisaged use case?
Charisma, motivation and luck all seem like things an AI instance could have.
But interestingly, rockets did level off at far less than the light speed limit, just as various kinds of heat engines all ended up leveling off at their own points below that of the Carnot limit. Solar energy is going through various sorts of phase transitions that get new limits theoretically closer to the Carnot limit, but rockets and airplanes aren’t getting closer to the light speed limit in any appreciable way.
Per Wikipedia, running a human (genius) level AI at faster speed is called a "speed superintelligence", and WP cites some article which claims that this can lead to a singularity. [0]
I am more skeptical. If we have instances of virtual John von Neumanns running at a million times the speed of the the non-virtual one, that will certainly get us some technological progress. Basically everything that can be discovered or invented by JvN will be discovered or invented almost instantaneously. The question is how transformative that would be for society. While the meat space JvN certainly had a bigger impact than almost all other humans on earth, his absolute impact on the average human is nonetheless moderate (at least when assuming that we would have invented mutually assured destruction without him in time).
Theoretical physics is one area where raw intelligence is probably more helpful than a lot of other areas (like inventing more practical (and impactful) things like the steam engine, PCR, smartphones, the web, cars, fridges, antibiotics, nukes). In a nutshell, the latest working model of particle physics is called the Standard Model and dates to 1975. I would argue that the reason we have not made progress with the open questions since (like unifying quantum mechanics with gravity) is *not* that von Neumann died in 1957 and nobody else was smart enough to figure it out. Theoretical physics attracts super geniuses like few other things do, but it seems that discovering a theory of everything (TOE, that is, all four forces) or even 'everything except gravity' (e.g. unifying electroweak and strong interactions) seems significantly harder than discovering General Relativity. I am not sure TOE would be found if we spent a million von Neumann years on searching.
(One might argue that the reason Newton and Einstein became such household names was that they were active at a time when science progressed very fast.)
Also, the Standard Model did not change the world. What changed the world was the earlier discovery of (mostly non-relativistic) quantum mechanics, which is required to understand chemistry and what goes on in semiconductors, which many of the innovations of the last few decades involve. So even if the AI discovers new physics theories, this might not change the world we live in fundamentally.
I think that while we would get some nifty inventions out of a 'speed superintelligence', what would really change society is genius level intelligence becoming to cheap to meter. I think the jump forward might be similar than the jump between the heyday of the Roman Empire and the our world.
The level of everyday convenience a rich Roman enjoyed is probably similar to that of the median American today. Both can take a hot bath at short notice with barely lifting a finger, have heating in the winter with no effort on their part, get both exotic fruits and other tasty food prepared for them and can get plenty of entertainment without even having to leave the house. The difference is that for the patricians, this was all powered by slave labor (and so would never scale to everyone), while for the American, this is powered by the fruits of industrialization, which give us central heating, long distance shipping, microwaves and Netflix.
I imagine that billionaires today can afford to have a staff of experts looking full-time after their physical and mental well-being, while the median person goes to half the health checkups which are recommended and perhaps does therapy now and then. In the world where human supergeniuses can be emulated cheaply, this will be available for everyone. Either that, or the machines decide that they do not really want to spend a subjective decade thinking about how to best cheer me up on some Thursday evening and overthrow us instead. Neither one deserves the term singularity, in my opinion.
[0] https://en.wikipedia.org/wiki/Technological_singularity#Speed_superintelligence
"It intuitively feels like lemurs, gibbons, chimps, and homo erectus were all more or less just monkey-like things plus or minus the ability to wave sharp sticks - and then came homo sapiens, with the potential to build nukes and travel to the moon. In other words, there wasn’t a smooth evolutionary landscape, there was a discontinuity where a host of new capabilities became suddenly possible."
Homo erectus did unlock lots of new capabilities, like stone tools and fire. Other apes were all limited to a narrow range of environments, while erectus was able to expand to all of Africa and much of Eurasia.
You beat me to it. Homo erectus seems closer to homo sapiens than chimpanzees. Recall also that there were actual humans with stone-age technology when first contacted in the 20th century. Not too far from erectus.
Agreed. The modern human appeared ~ 300000 BCE. Until ~10000 BCE, they would have looked like the continuation of homo habilis et al in terms of singularity potential. Then climate conditions (or seed cultivation or intelligence increase or whatever) go right, and within an eyeblink you have civilization with metal use, aircrafts and TikTok.
I'm actually devil's advocating here - my actual position is that there's smooth exponential/hyperbolic progress in primate impressiveness - chimps are genuinely more impressive than gibbons - and it's just that the very steep part of the exponential/hyperbolic curve makes the rest look pitiful in retrospect.
And interestingly, even after erectus, other hominid species developed new stone tool making technologies, some of which spread to other hominid species. It’s hard to know how many other technological developments may have spread back and forth between species.
https://en.wikipedia.org/wiki/Lower_Paleolithic#Middle_Pleistocene
>We need clear criteria for when AI labs will halt progress to reassess safety - not just because current progress is fast, but because progress will get faster just when it becomes most dangerous.
Just because we need something, doesn't mean that it's feasible to get. Either alignment is easy, or an (obviously unrealistic in practice) pivotal act is required. Or a global thermonuclear war will indefinitely delay these considerations, for another source of hope ;)
When "global thermonuclear war" is the "hope" side of the board, should we not consider that alarming?
Alarming or alarmist. Both seem like popular positions.
Alternatively, if you're hoping for global thermonuclear war, consider, "in the bowels of Christ," that you might have lost your fucking minds.
> But a few months ago, Ajeya Cotra of Bio Anchors updated her estimate to 2040. Some of this update was because she read Bio Anchors and was convinced, but some was because of other considerations. Now CCF is later than (updated) Bio Anchors. Someone is wrong and needs to update, which might mean we should think of CCF as predicting earlier than 2043. I suggest discovering some new consideration which allows a revision to the mid-2030s, which would also match the current vibes.
She updated after reading her own work? I suspect "she read CCF" is what you meant to write.
Thanks, corrected.
'full model has sixteen main parameters and fifty “additional parameters”' So this is the Drake equation for AI? Boo!
Well, the LLMs themselves have ~10^12 parameters... :-)
( My last manager, before I retired, was queasy about tuning parameters from model fitting in our software ... _but_ was enthusiastic about machine learning, which can be viewed as fitting tuning parameters by the gigabyte... :-) )
Yes, but I think they're multiplying their probability distributions correctly, which is where the Drake equation screwed up.
“...was starting to feel like they were all monkeys playing with a microwave. Push a button, a light comes on inside, so it’s a light. Push a different button and stick your hand inside, it burns you, so it’s a weapon. Learn to open and close the door, it’s a place to hide things. Never grasping what it actually did, and *maybe not even having the framework necessary to figure it out.* No monkey ever reheated a frozen burrito.”
--James S.A. Corey, Abaddon’s Gate, 2013
I love the idea of 100 million agents being unprecedentedly bottlenecked. AI advancement is stalled for years because audit cluster 0xA86F9 requires 17 quintillion of the new TPS cover sheets but each one requires 489! joint sign offs.
Bottleneck relieved when one of the audit clusters discovers how to forge the sign-off signatures of other clusters.
Is this preceded by that audit cluster making breakthroughs in number theory to allow it to break the hashing function, or it taking over most of the world's computing capacity to brute force the hash?
Seems easier to hack the other clusters and steal their keys, now that I'm thinking about it.
The theory breakthrough is probably easier to hide from retaliators than taking hold of large amounts of compute.
Yeah, hacking the other clusters or the validation mechanism does seem like a more cost-effective option.
While I'm not an AI researcher, there's an underlying assumption in this conversation that doesn't line up with my experience watching models get built. Compute is not really an input to model performance; it's more of an intermediate metric.
We are not compute capacity constrained. If more compute alone would produce a better model, then the major players would have no problem spending the money. Rather, there are a bunch of problems that have to get solved at each scale in order to effectively deploy more compute. Problems like, how do we get massively parallelized GPUs to talk to each other when we have to fragment model training across multiple data centers? Or where are we going to get an order of magnitude more data and how will we clean it?
Obviously these problems are tractable, and the story of progress is that we keep solving them. But there's no guarantee that that will continue, or follow any specific pace. Keep in mind that AI has gone through several major "winters" where research progress slowed to a crawl. This makes probabilistic models like this ring hollow to me. You're reducing innovation to a coefficient, when you actually have no idea the probable distribution of values for that coefficient.
"You don't know the distribution of values for that coefficient" is helpful to me in understanding why these models feel like bullshit.
Pessimistically, the percentage of jobs that AI can take now is negative, because of the hallucination problem. >Every< piece that an AI writes has to be checked by >at least one human< for errors and just plain making stuff up. Needless to say this is no way to conduct research of any sort; we have enough humans making things up in research already.
For writing code? The code that is written may work, but you have to test it. The test cases need to be written (certainly checked) by a human because the AI writer has no sense of whether this code actually does what it's supposed to.
This has been hyped by journalists, who really don't know what they're talking about. Or perhaps they do, since they write opinion pieces and present them as fact.
The takeoff curves assume that we have anything right now that it capable of doing any human job. Extrapolating from zero is an exercise in futility.
So if I hire a junior engineer, who solves the problem incorrectly, have I just hired a negative person?
What if I hire a principal level engineer who only makes one dumb mistake every 10 years? Have I still just hired a negative person?
Put another way, why are the normal checks humans put on each other, like unit tests and integration tests suddenly no longer sufficient when the code is written by silicon instead of carbon?
The AI requires much more checking, because you can generally trust that the person (even if very junior) is attempting to produce coherent code, and also generally testing that code.
People can be embarrassed into better behavior. AI, so far, cannot.
And yes, mistakes can be extremely costly. It is indeed possible that a senior engineer can make a code mistake that costs the company more than the value he added. For example, the Mars probe that was lost on landing where they figured the problem was due to using metric in one system and Imperial in another interfacing system. The probe likely cost more than the value-added for all the people who reviewed that code altogether.
Does a robot not “take” 20% of a job when it lets a human do 25% more in the same amount of time by using it as a tool?
Some people would say so, e.g. a lot of union reps. I don't agree, but it is an argument.
I think it's better to talk about AI taking over tasks rather than taking over jobs. I'd say GPT4 can do at least 5% of white-collar tasks out there (a conservative estimate IMO) but that doesn't imply it can do 5% of jobs. Additionally, it doesn't seem clear to me that our current AI can invent new things, and it isn't clear to me that it is trending towards having the ability to invent new things. I do agree that AI can make current AI researchers more productive, which should speed up the development of AI, but this all seems to rely on the idea that reaching AGI using our current methods is possible.
I'm also curious what happens as the amount of human output decreases. Let's use a field like Digital Marketing as an example. Currently, there is a great deal of human output on the internet regarding Digital Marketing strategies/fads/tech/etc. As AI replaces humans working in this space, the amount of human output will decrease. If we don't hit something like "Artificial Marketing Intelligence" in time, won't the drop in human output cause the AI to stagnate?
CoPilot et al. seem to me to be code generators, which we've had for decades (what is an assembler after all). It's another level up, optimistically. Taking over tasks also seems more realistic.
Although...
I've been at my current workplace for some time. When I started there was a classification for a "programmer" job, but no one had been hired for it for a while - the first step was "programmer/analyst" at a higher rate, and also involved more than taking very detailed instructions and writing small subroutines in COBOL and JCL.
The "programmer/analyst" classification doesn't exist anymore; you get hired at "Developer" level now. More analysis, interacting with the business side to get requirements, etc.
At a certain point, taking over tasks means that the beginner jobs get automated out of existence I think.
OTOH, some "beginner" jobs that are difficult to automate because the real requirements aren't the advertised ones. An intern doesn't really do anything an employee couldn't, but having one does take over routine tasks. The intern is actually being examined for things like thoughtful initiative, as well as things like showing up on time.
At least in the ideal world. Often the interns get abused - generally because they are in a field overflowing with starry-eyed idealists.
The vibes point is an interesting one. When everyone is publishing and wants to be taken seriously, they may tend to publish numbers in line with prior estimates. This causes estimates to clump and results in a landscape of superficially close estimates that risk portraying more confidence/consensus than warranted.
My friend and I published our own estimates in a 114 page submission to the Open Phil contest where we explicitly tried to make our best model with as little vibes contamination as we could manage. We ended up predicting that the odds of AGI taking our jobs in 20 was a bit under 1%.
Discussion and link to the paper: https://forum.effectivealtruism.org/posts/ARkbWch5RMsj6xP5p/transformative-agi-by-2043-is-less-than-1-likely
Scott, would it be possible to split off the AI discussions from the rest of the newsletter? I greatly value the pre-AI-discussion ASC/SSC, but think the AI stuff is both far too speculative and something too removed from my life to want to get sucked into it (and, alas, I tend to get sucked into it). On the other hand, the rest of ASC content is either fascinating or is directly relevant to my work in mental health counseling.
I've sometimes wished there was an option to follow author/tag combos, not just authors. Substack doesn't support tags, but SSC had them, e.g. https://slatestarcodex.com/tag/charity/
ACX is a cobbled-together collection of about a dozen different topics and I don't think there's a stronger argument for separating out AI than other topics.
I'm trying to subtitle all my AI posts "Machine Alignment Monday" (though I often forget). If you block that phrase on your email, you should avoid getting them.
Thanks, Scott. That will do the trick
There's a difference between can do human tasks in theory (e.g., with lots of hand-holding and direction by a human), and can literally do the task (no hand-holding or guidance), and can ID the tasks to do. That is, GPT can likely already "do" 20% of human tasks in a service-oriented economy, but can't do the tasks on its own, let alone ID the tasks. At some point, if the AI is legion and smarter/more competent than humans, you shouldn't need any humans in the loop. Those are all the interesting inflection points that don't seem to be captured by the theoretical lines, which instead seem focused on technical capability rather than practical use (if genius AI arrives tomorrow, but can't get itself into the industrial loop to run things for a couple decades, then it's a couple decades, not tomorrow).
But
https://techxplore.com/news/2023-06-ai-death-spiral.html
The “easy” response to this is to always train on the same dataset from 2021. That at least prevents pollution of data and the problems of training on machine generated data. It does make it a lot slower to grow the training the way these models assume though. (More likely there are some smart things you can do to avoid training on ai-generated data, but we are definitely quickly reaching a bottleneck in the amount of data of the sort desired for training.)
If you think Bio Anchors built a complicated model around just a few key handwaved parameters, I think CCF must be even worse. On evidence about substitutability for AI inputs, [Davidson writes](https://docs.google.com/document/d/15EmltGq-kkiLO95AbvoB4ODVpyg26BgghvHBy1JDyZY/edit#bookmark=id.gpdqttqd4hbs):
> The third bucket of ‘evidence’ is simply doing inside‐view thought experiments about what you think would happen in a world with zillions of AGIs working on (e.g.) hardware R&D. How much more quickly could they improve chip designs than we are currently, despite having access to the same fixed supply of physical machinery to use for experiments? ...
> This third bucket of ‘evidence’ leads me, at least in the case of hardware R&D, to higher estimates of than the first bucket [empirical macroeconomic research]. If ρ = −0.5 and [share parameter for substitutable tasks] α = 0.3 (as I suggested for hardware R&D), then even zillions of AGIs would only increase the pace of hardware progress by ∼10X. But with billions of AGIs thinking 1000X as fast and optimising every experiment, I think progress could be at least 20X quicker than today, plausibly 100X. If α = 0.3, a 100X speed up implies ρ = −0.25. I expect some people to favour larger numbers still.
This is one of the most important parameters in the model, and Davidson doesn't even describe the thought experiments he does. Even 10X seems fully detached from my picture of semiconductor R&D, but I made [a related market about it on Manifold](https://manifold.markets/MuireallPrase/before-2043-will-a-leadingedge-proc?r=TXVpcmVhbGxQcmFzZQ) in case someone knows something I don't.
I'd also hesitate to call what's described here an "inside view" approach, except maybe relative to pure macro models. Meanwhile, the [model playground](https://takeoffspeeds.com/playground.html) also has some counterintuitive behavior—for example, changing various parameters in a conservative direction brings “pre wake‐up”, “wake‐up”, and “mid rampup” closer to the present. (Most simply, I've found that starting with the "best guess" preset, increasing hardware adoption delay does this; so does increasing both AGI training requirements and the FLOP gap while keeping their ratio constant so that the FLOP requirement for 20% automation should be constant.) It's hard to debug this without more inside-view or mechanistic breadcrumbs, and it makes me worry that the takeoff speed conclusions are baked in to the model—as with nostalgebraist's observation about Bio Anchors, a much simpler picture could give the same result, and the complexity of the model only serves to obscure what drives that result. (I personally think Bio Anchors is relatively straightforward about multiplying uncertain estimates of a few key numbers and don't particularly fault it.)
(This is paraphrasing my comments from [Appendix A here](https://muireall.space/pdf/considerations.pdf#page=10).)
Thanks for the pointer, fascinating essay and some neat insights (like open source potentially damping the capital intensive path).
One side comment about "wake-up", re
"The model has AI investment grow at approximately its current rate until wakeup time, then grow at some higher rate similar to investment in munitions during wartime, or the highest rate at which the semiconductor industry has grown during periods of intense investment in the past."
Since AI has military applications, and the US and Russia are in a hot war-by-proxy and the US and China are in a cold war, I don't think that AI investment will just be "similar to investment in munitions". I think we can reasonably expect military AI investment to become literally an arms race.
A minor note: DALL-E2 is actually looking pretty outdated at this point. With the prompt "The ancient Romans build a B-2 stealth bomber", for example, Midjourney V5.1 produces images like: https://i.imgur.com/QHD8PI4.png and https://i.imgur.com/yRygwyu.png .
Midjourney seems to think that Romans lived in Athenian ruins. How quaint.
To be fair, most humans do too.
That's great! I tested a few AIs (StableDiffusion, can't remember what else) with the same prompt, but MidJourney seemed to have some friction for signing up so I bounced off of it.
The latest MidJourney update produces the best results of any public model I've seen so far, though the Discord-only thing is definitely an annoyance.
That said, the public channels can be good for getting prompt engineering ideas, and you can also message the MJ bot directly for a more private experience. Ordinarily, all upscaled images are automatically posted to the MJ website, though there is an obscure feature where if you generate images with "--q 2", then add an :envelope: reaction to the bot's comment, it'll respond with full-resolution versions of the generated images, without posting them to the site.
On a related note, if you're interested in spending more time messing around with media synthesis: Runway.ml just recently opened up their text-to-video model to the public, and will let you generate a few clips for free. It's sort of at a DALL-E 1 level of coherence right now and feels slightly over-fit to the training data, but hey- it's video!
> Current AI seems far from doing 20% of jobs
Does it?
Reading the specification, I'd be more likely to think that current AI already does more than 20% of jobs. Caclulators substitute for mental math; spreadsheets substitute for hand-tallied ledgers; logistics systems substitute for ad hoc logistic personnel. The thermostat in your home substitutes for manual fire-tending. Even the computer in your car substitutes for regular tune-up work.
If we include hardware development inside AI, then we've already substituted something like 90% of jobs that once existed, thanks to mechanization of farming.
If we say that AI does not yet substitute 20% of jobs, it's only because we're defining jobs as those things not already automated into the background.
I'm less sure what this means for the analysis. If we claim that AI passed the 20% threshold in 1975 with affordable pocket calculators, then does that mean we should expect the remainder of the "to 100%" timeline to take longer? What if we include capital in general and claim that machinery replaced 20% of labour back in the early to mid 1800s?
None of those things actually do the job themselves, they need humans to tell them what to do, when, frequently how. That's the big objection many have to AI taking over world, which requires not just theoretical ability to do X, but a practical way to do X. And not just X in the narrow sense, but everything required for X. E.g., no AI puts in the thermometer in your house, the HVAC dude does. No calculator calculates without human input. Etc. AI automating more and more makes lots of sense, but that's in the gaps where automation works. I agree that this shows automation kills only some jobs, leaving others alone, creating new ones, etc. But most of the AI discussion ignores the requirement of interaction with physical world to get anything done in a meaningful sense.
> But most of the AI discussion ignores the requirement of interaction with physical world to get anything done in a meaningful sense.
Here is a scary thought: in a world where AI is smart enough to take over 75% of the intellectual work, the main occupation of humans might be to serve as hands (or meat puppets). Think Amazon warehouses. Or someone working in a McDonalds kitchen when the ordering is happening by kiosk or app. I can totally see the HVAC dude of the future being minimum an untrained laborer with some AR glasses and a talent for following instructions on the screen. Perhaps for a time there will be the ability to escalate to an actual HVAC engineer if the AI becomes confused by a rat in a ventilation duct or whatever, but the number of experts you would need for that will invariably go down over time.
Until now, automation has mostly eliminated the shittier jobs. In the medieval age, basically everyone had to do backbreaking labor in the fields to grow the food they needed to survive. Now a farmer can easily grow enough food for hundreds and has a much less shitty time while doing it, so good riddance.
Now, instead of AIs driving cars or flipping burgers, we get AIs which can write stories and draw pictures (not on a human pro level, but certainly better than what the median human can do).
Automatization has allowed the individual worker to be much more productive.
Consider:
* A human carrying some load on foot
* A human driving a horse-pulled cart
* A few humans running a freight train
The gains in mass distance per person hours between these three tech levels are certainly several orders of magnitude (even if you count the overhead of having to breed horses or build train tracks and engines and process fuel).
Some of these gains certainly eliminated work, but also the cheaper supply meant that stuff which was not economically feasible before suddenly became feasible. In the stone age, long distance trading was simply not viable, so Europeans had to do without bananas.
For calculations, it is the same. The electronic computer replaced the human computer, but also created opportunities for consuming compute on a scale which would have taken many person-lifetimes before.
The typical definition of AI (versus non-AI) that is widely used is that anything already existing in the present is not AI. So in 1950, a decent chess computer would have been considered an AI, in 2000, a chatbot passing the bar exam would have and so on. This means that by our present definition of AI, it has replaced exactly zero percent of the jobs. Obviously that is not very helpful.
Is there any model ever in the history of humanity that has survived 12 orders of magnitude? Further, this model is predicated on the extension of Moore’s Law for another 12!? That is, we will continue the exponential growth in practical engineering for what is already the most complex and expensive device we currently build. A new fab costs tens of billions of dollars as is. At this rate, when do we hit 100% of GDP spent building new fabs?
And even if we did that, we’d still run into hard physical limits between here and 12 orders of magnitude. Off the top of my head, see Landauer’s principle for theoretical limits on the energy consumed to compute stuff. Does 12 orders of magnitude necessarily imply we will literally boil the oceans? I don’t know off hand but I wouldn’t be surprised. I didn’t do any math, but that’s how wacky it is to extrapolate exponential growth over 12 orders of magnitude.
"Is there any model ever in the history of humanity that has survived 12 orders of magnitude?"
Maxwell's equations. Of course, it is a very different kind of model...
I’m not sure actually. Starting from a meter, we can’t go down to 10^-12 unscathed. The other direction seems to be fine though, yeah?
I'm sure we can get down to at least 10^-10 meter unscathed, because atomic spectra probe that regime. I _think_ collider experiments and the g-2 experiment go down even further ( https://ui.adsabs.harvard.edu/link_gateway/2004PhLB..584..109H/doi:10.48550/arXiv.hep-ph/0308138 , which I admit I only have a fuzzy understanding of, seems to imply that no new physics shows up in the g-2 experiment, to at least an energy of 577 Gev, corresponding to a length scale of 2.2x10^-18 meters. ). And, as you said, in the other direction we are fine up through astronomical distances...
Now, for the model at hand, the mere fact that we've been _seeing_ unexpected "emergent" capabilities on scaling up LLMs suggests that we have very little idea what will happen with the next single order of magnitude, let alone anything beyond that...
> There’s no substitute for waiting months for your compute-heavy training run to finish
Not necessarily true. A few factors of 2 speeding up various aspects of the training process comes out to a factor of 10, more or less. Could be as simple as a better gradient descent algorithm, more efficient computational operations (possibly via hardware), smarter weight initialization, some kind of improved regularization to get more information out of each batch/epoch... A sufficiently smart intermediate AI will figure out some of those things or come up with others.
I have yet to see any justification for the implicit belief in all this AI posting (by anyone) that scaling up fidelity of recall will lead to ingenuity. AI has no possible mechanism for detecting gaps in its knowledge. There is always this contradictory element of assumed capacity for surprise. Any time I read "What if AGI invents/designs..." I disregard the entire conjecture.
What evidence would change your mind?
I think the natural response to this is - what mechanism do humans have for detecting gaps in our knowledge, and is this mechanism the sort of thing that could be programmed in, or possibly learned by a neural net?
This is one of the better "AI will take over the world" articles I've seen. But I still think futurists need to distinguish between "AI assisted coding will make AI smarter" and "AI assisted coding will make AI better at accomplishing the code's stated goals." These are extremely different concepts and have different implications.
Even big, general AI like ChatGPT is focused on clear, user-defined goals. The goal of training isn't "be as smart as a human" it's "create output that satisfies a given use-case." And thus future AI designed with the help of ChatGPT-like tools won't *necessarily* be "smarter," it'll be better at satisfying given use-cases. In fact the theory that it'll be better at satisfying given use-cases has an assumption baked in - that the code we like and train it to produce is code with superior performance. Likely we're using other selection criteria like "looks like good code at a glance."
I still think those wrinkles make projecting drastic, society-altering, long-term results of AI a fool's game.
This whole debate is between people who believe AI progress will be linear, and people who believe it will be exponential. This is frustrating, because it completely excludes my own belief, which is that progress will end up being logarithmic, with advances past a certain threshold producing rapidly diminishing returns. The problem of diminishing returns is a consistent one throughout the natural world: broadly speaking, most complex systems reach a point where adding more X produces less Y, and it's bizarre to assume that AI development will be an exception to this. If nothing else, there are hard physical limits on computation ability!
Furthermore, the problem of diminishing returns doesn't just apply to how smart we can make an AI. It also applies to what an extraordinarily smart AI could accomplish with all of its intelligence. There's very strong evidence that even *human* intelligence produces diminishing returns after a certain point, in terms of successful life outcomes and ability to impact the world. Someone with an IQ of 100 has great advantages over someone with an IQ of 70; someone with an IQ of 130 has significant advantages over the 100 IQ person, though not as massive as the 100 IQ person has over the 70 IQ person. But someone with an IQ of 160 isn't likely to be that much better off than the 130 IQ person, statistically speaking. And the difference between a 190 IQ person and a 160 IQ person seems entirely negligible; their internal experience may be very different, but beyond that, being a super-duper genius probably isn't much different than being a super-genius.
The "conservative" side of the debate is not nearly conservative enough for my tastes!
I have the exact opposite sense as you, in terms of the returns of intelligence. The best evidence we have says that intelligence has *increasing* returns, both in humans and AI. AI progress has gotten faster and faster in recent years - more progress is happening in months than used to occur in years and decades. And in humans, one of the most rigorous decades-long studies with a large cohort of children found that their IQ test results at young age predicted their academic/career success in life to a high level of accuracy, not just to the point that the top decile performed better than the bottom 9 deciles, but that the top decile of the top decile performed significantly better than the top 1%, and even the top 0.01% performed significantly better than the top 0.1%.
Also, think about the scientific discoveries made by the handful of 150+ IQ individuals throughout history and how far they've advanced humanity - we might be hundreds of years behind where we currently are technologically if Newton had never lived. Einstein easily contributed more to science than billions of other humans combined.
Most of the truly great discoveries were made by people who never took IQ tests, so we don't have data to make such claims. We do have a few data points that indicate a driven, charismatic, highly but not exceptionally intelligent person will usually have a much greater impact on the world than a super-genius who isn't exceptional in motivation and charisma. Could you provide a link to the cohort study you seem to be basing your claim on?
https://gwern.net/doc/iq/high/1926-cox-theearlymentaltraitsof300geniuses.pdf
Motivation is somewhat about goals. If someone wants a quiet life and achieves it, and thus doesn't change the world, that doesn't mean they aren't intelligent.
Charisma is a mix of appearance and social intelligence. Social intelligence might not be in IQ tests but it's still in the brain.
I think EVENTUALLY it is likely we run into diminishing returns. At least if the laws of physics are like we think. The question is how many orders of magnitude there are between here and the point that diminishing returns kick in.
I don't know how you compared an IQ 190 person to an IQ 160 person. Given the bell curve that IQ is supposed to follow, there should only be a handful of IQ 190 people in the world. How exactly did you do this comparison. And there may be a big difference you can't see. Or it might just be that IQ tests aren't an effective measure of real world capabilities at the upper extreme. After all, it's possible to ace an IQ test and still fall to motivated cognition, because in an IQ test you never have any strong motivation to want the answer to be a particular thing. So someone could have a 190 IQ by the test, and the moment the question "does god exist" turns up, motivated reasoning kicks in. This is a limit of IQ tests, not of intelligence.
I think the human brain is not anywhere near the physical limits of computation. Diminishing returns tend to start kicking in when there is no room for large improvements. There are cases of things (ie cars) running into diminishing returns on squeezing more power out of chemical fuels when we don't have the tech yet to do nuclear cars safely.
I think it is unlikely that we get diminishing returns at around human level, rather than way past it.
Is there a primer on how dichotomous train-then-use AIs are supposed to speed research? My understanding (maybe wrong?) is that while a given AI might assist researchers in making breakthroughs, we'd need to train a new AI with data including those breakthroughs before it would "understand" them to build on them. Am I missing something?
More generally: I am definitionally in the neighborhood of being an AGI, but to get to anywhere near being able to do 20% of jobs I would need...probably more training than I could do in a lifetime. My "intelligence" isn't the bottleneck (if we pick the right 20% anyway), just the actual process of training for particular disparate tasks. How does the AGI concept account for that, again particularly given the "train once" nature of existing tools?
ChatGPT isn’t quite “train once”. You can give it examples of a pattern and then ask it to make more, and while that doesn’t produce a lasting change to its state the way the original training did, it does enable later parts of the conversation to take advantage of abilities it didn’t have in earlier parts.
Right, but my understanding was that information its fed during that conversation does not give it new information (in the sense of broadening its "network of concepts", or whatever you want to call it), just helps it localize the conversation within the existing network. If that's an accurate understanding, I remain confused how it could integrate a set of new ideas to build upon in a research context.
I think “network of concepts” is at best a metaphor here that makes it hard to know whether or not the within-conversation state changes affect it. What definitely doesn’t change are the weights on the edges in the neural network. But having short term memory of text that gets fed into the neural network and shapes the output can clearly shape the abilities of the new output. It might be like how a student can learn a fact or even a method early in a conversation and then apply it later by using their short term memory, even if they won’t remember it later. The question is partially just how powerful that sort of short term memory can be, and whether it can pack more into that short term memory than we usually do.
I think my main argument for MIRI-style takeoff is "play with GPT-4!" That thing is already superhuman. It doesn't sound like a stupid person, it sounds like a smart person whose train of thought frequently derails. If I view it as "a human with brain damage", I'm forced to ask a question like, "wait, if it can be this smart while literally missing parts of its brain, how smart would it be if you added the missing parts back in?" In other words, I think the parameter count in GPT-4 is already sufficient for at least human-level intelligence, and it's *entirely* held back by unrealized overhang. I don't think that's modeled at all here.
Excellent comment hidden down here in the rough.
You've helped me form my thoughts on this matter a little more. You win the internet.
Humans have relatively few datapoints, but are good at extrapolating far from them.
GPT4 isn't nearly as good at extrapolating, but has more datapoints. In other words, compared to most humans most of the time, GPT4 is more rewording things with less understanding. (Although some people sometimes reword things with even less understanding)
The 100M copies of an AGI working on AI research won’t really be that numerous, unless we assume that AI researchers can make progress by sitting in a room thinking really hard. All of the AGIs will also want access to enough compute to run their own experiments, which will tend to be computationally intensive, so the actual number will be several orders of magnitude smaller.
In fact, if we follow the assumption that the AGIs can run on the same compute resources that trained them, and we assume that training a post-AGI system will be at least as expensive as training the AGI, then you end up with slightly less than 1 effective additional researcher.
Since all the take-off speed estimates take "at least the linear trend we've observed so far" as a baseline, it would be a much more accurate visualization to show the slow and fast take-offs as departures from the linear trend rather than depatures from nearly nothing.
> This would be less worrying if I was sure that AI couldn’t do 20% of jobs now.
I'm very sure that this is the case. AI currently can't do jobs that require physical interaction with the outside world, like surgery or construction. It can't manage people. It empirically can't do law. Personal experience says it can't do data analysis. It's prone to fabricating things wholesale, and only mediocre at correcting itself. Its writing isn't good enough to wholesale replace that many writers. What jobs can AI actually do? I think at this point it's exactly like technological progress of the past--a tool that allows fewer people to be more productive, but not fully doing anything.
What part of the model (if any) contains information about physical limits? For example, what is the requirement, in terms of energy, raw materials, etc. to make models with another 9 OOM of compute? Does it require building a dyson sphere? This article: https://ts2.space/en/exploring-the-environmental-footprint-of-gpt-4-energy-consumption-and-sustainability/
claims that
> The paper found that the entire training process for the GPT-4 system requires an estimated 7.5 megawatt-hours (MWh) of energy. This is equivalent to the annual energy consumption of approximately 700 US households.
which suggests that another 9 orders of magnitude is roughly equal to the entire current energy consumption of the world.
The other part of this type of model, which always makes me skeptical, is extrapolating any of the effects of intelligence past the range that humans have, since we have no data. We don't have any idea how much the *difficulty* of making software improvements (or hardware design improvements) compares with the corresponding intelligence gain you get out of them. It seems entirely possible to me that there are diminishing returns, where getting extra intelligence requires more intelligence to begin with than you get out, so progress is always inherently slow. For example, consider the AI that was tasked with improving matrix multiplication, a key step in many algorithms, including training AI. It found some speedups, but they were very minor. I don't think the ones it found were even useful (they were limited to small matrices, or to arithmetic mod some small number). There's a fundamental limit to how quickly you can do this operation, and if an AI has to already be smarter than a large group of top human mathematicians combined in order to get a marginal improvement, then what does that say about a possible explosion?
The limited data we do have (all historical scientific discoveries) suggests that the most intelligent human beings make more discoveries than billions of other less intelligent human beings *combined*, with no observed ceiling to this effect. Newton was worth 100 Einsteins, who was worth 100 von Neumanns, who was worth 1,000,000,000 average people, and so on.
How can you confidently claim there is no known ceiling when human intelligence is tightly limited, and hard data on it before the modern day so sparse? I think your examples are very dubious, and the very best scientists are worth more than average but nowhere near 100x merely very good scientists. To take Einstein, QM was the result of decades of work by physicists to understand energy, with Max Planck making the key mathematical breakthrough years earlier. G.H. Hardy almost beat him to the punch on GR. Neither of these ideas would have gone undiscovered, or even probably been delayed by much, if Einstein had been hit by a train while working at the patent office.
In any event, the question is not where being smart helps you make discoveries--it's how hard the problem is of increasing the intelligence of a complex AI.
It seems quite clear that the marginal value of intelligence increases as intelligence increases. Suppose we could take everyone in the world with <70 IQ and give them one extra IQ point - this would have very little effect on their wellbeing and zero effect on global scientific or technological progress.
However, if we could take everyone with 140+ IQ and give them one extra IQ point, humanity's potential for scientific and technological progress would appreciably improve. This is because there are power law (or power law-like) effects in scientific discovery, where the top 10% of people make 90% of the discoveries, and within the top 10%, the top 10% of those people make 90% of the discoveries within that subgroup, and the top 10% of that subgroup make 90% of those discoveries, etc. On top of this, the smartest people who make breakthrough discoveries are almost always extremely prolific as well, producing a huge quantity of amazing discoveries, further increasing this effect.
I suppose the task of increasing this intelligence could be hard, but it's clear that human biology varies in intelligence greatly, can operate using very little energy, and has no observed ceiling effects, other than regression to the mean resulting from a very lossy reproduction process. But if the AI could merely reach moderately superhuman levels of genetic engineering, it could surely utilize meat-based mediums of intelligence as well.
This seems quite incoherent. IQ is not mana gained in a fantasy game, it's a mapping of a distribution of some measure of cognitive performance to positive integers. I also see no reason to believe the claim made that marginal value of intelligence increases with the absolute value of the measure being used. If you have a well-argued case for why marginal value of intelligence somehow does increase then I would be interested in reading it, because it would be one of the rare exceptions to the general rule of diminishing returns.
> IQ is not mana gained in a fantasy game
What an incoherent strawman. "You said we can increase intelligence, but you know what else increases? Mana in a fantasy game! So you must think that intelligence is the same as mana!" Yes, both things can increase, and that doesn't make them the same. It is not at all controversial that some entities are more intelligent than others.
Maybe you're just pedantically objecting to the phrasing of "Increase IQ by X" as a shorthand for "Increase cognitive ability to the extent that IQ score increases by X"? The particular units are irrelevant here, the point of the hypothetical is that intelligence increases.
"I also see no reason to believe the claim made that marginal value of intelligence increases with the absolute value of the measure being used."
I already gave several, but here:
https://emilkirkegaard.dk/en/2018/07/gladwell-the-threshold-effect-for-iq-and-jensen/
"The findings are unequivocal. There is no point on the IQ scale below which or above which IQ is not positively related to achievement. This means that IQ does not act as a threshold variable with respect to scholastic achievement, as has been suggested by some of the critics of IQ tests."
"... for IQs of 140 and above, there are still achievement differences related to IQ, as can be seen by contrasting the typical school-age intellectual achievements of Terman’s (1925) gifted group with IQs above 140 (the top 1 percent) with Hollingworth’s (1942) even more highly gifted group with IQs above 180. Some of the differences in the intellectual achievements even among children in the IQ range from 140 to 200 are quite astounding. Over fifty years ago, Hollingworth and Cobb (1928) strikingly demonstrated marked differences in a host of scholastic achievements between a group of superior children clustering around 146 IQ and a very superior group clustering around IQ 165. The achievement differences between these groups are about as great as between groups of children of IQ 100 and IQ 120."
https://www.nber.org/papers/w24110
"an R-squared decomposition shows that IQ matters more than all family background variables combined"
See also: https://www.emilkirkegaard.com/p/there-is-no-iq-threshold-effect-also
"The income plot as a function of intelligence and personality look like this. The curve has a slight upwards trend, so intelligence is actually more important for incomes the higher your IQ is, not less... And this lack of threshold is not just for income, it's for other forms of achievement as well. This can be seen clearly in the SMPY study of the gifted... within the SMPY cohort, which is very bright on average, maybe 140 IQ, the top quartile of super brights (160 IQ?) do better than the merely very brights (135?). This is true for: obtaining PhDs (doctorates), producing science (STEM publications), filing patents as a measure of innovation, income, and literacy production... Note also the steeper income gradient at the very high IQs."
Thanks, unfortunately I don't find any of these papers convincing. The sweeping conclusions (especially in the blog you linked) are just not supported by the data. In the banner studies, the things measured are intellectual achievements so it's not surprising that more intelligence helps there, it's an essentially circular argument. The analyses of income data might lead to something and I will continue to look for robust findings in that direction.
I don't see any evidence in this comment, just more assertions.
IQ is the result of taking ordinal (ordered) data and putting it into a normal distribution, so the idea of "giving one extra point of IQ to everyone with 140+IQ" doesn't make sense--that distribution would no longer be normal, for one, and for another, a "point" of IQ is not an inherently meaningful unit. It just means refers to some fraction of the population (which is determined by where in the distribution you are). It is true that 160 IQ is much rarer compared to 150, than 150 is compared to 140 (e^-x^2 drops off faster, the further you are from the mean), but this effect is symmetrical and it doesn't make any sense to talk about diminishing or increasing returns when the units are arbitrary. We could map these ordinal results to any distribution, and get shape of relationship between "IQ points" and output that we want.
> But if the AI could merely reach moderately superhuman levels of genetic engineering, it could surely utilize meat-based mediums of intelligence as well.
This is just baffling. What are you proposing? That the AI would grow human brains in jars to do thinking for it?
Okay, if you want to be pedantic, "Take everyone with 140+ IQ, based on the current scale, and improve their cognitive ability enough such that they would be able to score 1 point higher on a current IQ test." The way we norm the results are obviously irrelevant here.
"This is just baffling. What are you proposing? That the AI would grow human brains in jars to do thinking for it?"
Not something that specific, but re: "the question is... how hard the problem is of increasing the intelligence of a complex AI," the ease of improving biological intelligence with natural evolutionary means (not even genetic engineering) serves as extremely strong evidence against a hard cap on intelligence.
> The way we norm the results are obviously irrelevant here.
Sure, you can rephrase what you said to be meaningful. I don't dispute that, but I do dispute that it tells you anything at all about how fast takeoff can be. If you choose a measure of productivity, you can make the function productivity = f(IQ) superlinear, linear, or sublinear just by choosing a different mapping of test result to IQ. Even if you had data comparing IQ and scientist productivity (which you've asserted, but not provided) it would be fairly speculative to try to extrapolate any results outside the domain where we have any data.
And again, the question is not whether having a IQ makes scientific discovery easier. Of course it does. It might even be superlinear. But without knowing how hard it is to improve intelligence in an already-intelligent entity, this doesn't tell you much. This difficulty curve might be even more superlinear than the returns to intelligence. As far as I can tell, we have very good and easy ways to increase IQ from low to medium (improved nutrition, different methods of education) but making geniuses into super-geniuses is much harder. Certainly, AFAIK, no individual has managed to convincingly raise their own IQ in proportion with their initial IQ, or ever gone through any sort of an intelligence-increasing feedback loop.
(I'm also not saying there's a hard cap on intelligence--maybe more like a soft cap. Again, it's fairly irresponsible to extrapolate from "we can improve biological intelligence" to "there is no cap on intelligence", that's not extremely strong evidence at all).
The fact that the results are normed are obviously relevant, because what it means to be 140 IQ has changed throughout the years, giving us an obvious real-world test case about what happens when you give extra intelligence points to people (140+ IQ or otherwise).
> For example, what is the requirement, in terms of energy, raw materials, etc. to make models with another 9 OOM of compute? Does it require building a dyson sphere?
No, because compute != effective compute. Probably +9 OOMs in effective compute would only require 4.5 - 6 OOMs from compute and 3 - 4.5 OOMs from algorithmic improvements, going by https://aiimpacts.org/trends-in-algorithmic-progress/. I'm less sure how the 4.5 - 6 OOMs compute splits into $ and FLOPS/$, but I'd bet that more of the increase is due to FLOPS/$ (ie hardware efficiency) than to $.
(Actually, fact-checking myself using historical trends (https://epochai.org/trends), it seems that $ for training SOTA models has increased at at least the same rate as buyable FLOPS/$ if not slightly more; that said I'm still hesitant to project the former out 10-15 years as much as the latter, so I stick by my earlier bet.)
Doesn't seem to require dyson spheres in any case.
Yes, a model with another 5 or so OOM energy is doable, although it would be expensive enough (tens of millions of dollars, by my math) that it would difficult to do so repeatedly. That was just an example (my fault for not being clear). It seems like there's a number of other possibly limiting factors, such as available data, physical limits on the speed/space/efficiency of computation, etc. that I don't think the AI itself will be able to overcome (or, it can help work around them--but with diminishing rather than increasing returns).
I'm not sure how much to read into the humans vs. chimps analogy. I think that humans are so much more intellectually capable than chimps not because of some general principle that extra intelligence grants huge returns so much as that our intelligence granted us some specific, qualitatively different capability along the lines of general language which allows us do Turing-complete reasoning and allows us to transmit knowledge across generations. I don't think that an AI being 10% smarter than humans would necessarily make it all that much more capable, unless that extra 10% allowed it to unlock some similar sort of qualitative improvement (like if it let it discover an efficient algorithm for NP-Complete problems).
If anything, I'd think this is more of a reason to be bullish on AI takeover. Do we have any reason to believe that the technique of language is the only step change technique that exists? It seems reasonable to worry that more such techniques exist, and modest further improvements in intelligence could unlock one of them, causing our fate to go the way of the chimps.
I'd be more willing to accept this if you could point to a specific capability that they might unlock. And language granted us the ability to reason in a Turing-complete way. It really doesn't seem like there is a more general reasoning capability out there.
Something that would require more intelligence to even comprehend is unknowable by definition. This shouldn't bring us solace, though - chimps could have pointed to this same impossibility in their debates with the other chimps about how ridiculous it would be that humans would ever be to able to pose a threat to them. "Humans are so weak! We're way stronger than them and we can literally rip their faces off! What could they possibly do to us? Develop some brand new cognitive technique that somehow makes them infinitely smarter than us? And we can't know what technique it will be because our brains are too small to comprehend it? That's UNFALSIFIABLE!!! And how would them having more brainpower possibly kill us, worst comes to worst we can just snap their neck!"
And now we have snipers and unmanned drones and rockets and nukes.
But why would a qualitative improvement have to be unknowable by definition? In fact, I already mentioned a possible such improvement: a fast algorithm for 3SAT. We know what that is and know at least some of the cognitive improvements that it would grant. It is just widely believed that it is impossible.
In fact, I think the point about humans having Turing-complete reasoning capabilities is that I don't think that there can be reasoning capabilities that are *unknowable* to us (or I guess suitable generalizations of us with unlimited lifespans and memories), just ones that we do not yet know about.
Even if we were to grant for the sake of argument that such techniques are likely to exist, why should we specifically be worried that the way to unlock them is by "modest" improvements in current AIs?
Interestingly, language is an ability that ChatGPT very clearly already has at something like human levels, if not far superhuman. (It can translate between thousands of language pairs that no human has translated between, and can generate poetry subject to constraints that no human has used - even though it still can’t reliably count syllables.)
Does a parrot understand human language?
If it could form brand new grammatically correct and semantically coherent sentences about any arbitrary topic you prompt it with, yes.
Fair enough.
But let’s just say a parrot could do that. Absorb all those languages and spit them back in order. Do you think it would mean anywhere near the same thing to the parrot that it does to us listening?
Assuming it has a perfect understanding of grammar and syntax and enough exposure to language to calculate what should follow given what it is just heard.
Do we then share meaning with the parrot?
I think the real trick would be is to prompt it with some thing that has never before occurred in written language. It would be interesting to see what it came up with then wouldn’t it?
I think what’s kind of fascinating is this idea of a whole second order derivative of the world, if you will; meaning the world, as we humans experience it, and then the world that has been distilled in our written recollections, which is the universe of AI as far as I can see. It’s a whole metaphor layered on a metaphor, and the meanings of things can really slide around. There seems to be the underlying assumption that language is some kind of constant. And it really isn’t.
Dogs, for instance, have a very limited vocal range, but if you really start listening, there’s a lot of nuance in it. I guess I just think we have to be a little careful about the assumptions we make about a language learning intelligence. My personal opinion is that language is nothing but metaphor, and I have had some discussions on this forum around this issue. If a parrot were to say, I want a wet kiss, I would seriously doubt their understanding of the world.
I think the Romans with the stealth bomber are actually an interesting case to consider.
How fast could we uplift a Roman-level civilization to industrialization with information alone? (On the plus side, you get to bring all sorts of detailed technical descriptions, on the minus side, the Romans do not know to produce intelligence-boosting amulets from rare earth metals).
Some techs like the spinning wheel would be very easy to adopt, but will only yield marginally improved productivity. Still, the commercialization of such inventions might offer you enough slack that you don't need to follow the meandering historical path of economical viability and instead can spend big bucks on going to the end point.
Traditionally, the steam engine requires all the know-how from casting cannon barrels. Casting a cylinder from brass might be a possibility though.
Getting crucible steel should also be within the reach of the Romans if they read the hint book.
Translating theoretical knowledge into firm procedural knowledge takes time, however. The supply chain of a stealth bomber likely involves thousands of different experts with decades of experience for all sorts of industrial processes. As these processes depend on their prerequisites running smoothly, this is bound to take centuries. You can hardly start training programmers for the autopilot until your semiconductor industry is already quite advanced, for example.
This is basically the plot of Connecticut Yankee in King Arthur’s Court. Except that is meant to be a less developed society.
If people are cooperating with you, it wouldn’t take centuries. Rome was a sophisticated society. If in actual history it took two centuries to get from 1750 to 1950, how long would it have taken if you had been given all the ”answers” in 1750? Rome was behind Europe in 1750 but not that far behind, so somewhat longer for them.
Of course it depends also on whether they have knowledge of the steps in between their knowledge and modern knowledge.
> If people are cooperating with you, it wouldn’t take centuries.
I am not convinced. I think that many links in the supply chain not only require cooperative people who read the manual, but people with actual hands-on experience. I already mentioned the computer programmers, a job I have some experience with. You can't become a good coder just by reading books. This means that you will not have any good coders at the moment you solder together your first computer.
Also I think that this is rather typical. To get to aluminum in meaningful quantities you need electrolysis. This means that whoever is working in your bauxite plant can only start to learn their job when you have a decent power plant running. By that time they probably had a decade to read up on the technical manuals for producing aluminum, but I still would not be expect to get usable aluminum for at least a year.
This is kind of the reverse of experimental archaeology. Any kid can tell you that you can theoretically make fire using wooden sticks. That info is helpful -- our ancestors took millions of years to figure that one out. Still, even if I had the relevant wikihow article, I would find it highly unlikely that I would succeed to start any fires soon. After a few months I might finally roll a 20 in good conditions and get it going, and after a few years I would have picked up enough tricks to start a fire in most conditions. Modern expert jobs aren't any easier to learn.
A speedrun of a puzzle game can be much quicker than anyone would need to figure out the puzzles, but it will still take time.
If you enable economy in your speed run, it will be harder still. Lots of technology is only feasible to develop if you deploy it at scale. Developing fridges to the point where they run as well as they do in our world will only pay for itself if you have already electrified Rome (though even then the customer base might be to small), but if you skip that step, then fridges will require frequent expert maintenance and this will slow your biotech labs or whatever. The population participating in our global economy is much larger and mostly wealthier than the population of the Roman empire.
That being said, I would certainly assume that there are inventions which a civilization could build centuries before they stumbled upon them. I mentioned some which seem within easy reach of the Romans. Extracting Penicillin from mold might be another.
Semiconductors might have been accelerated by a century, perhaps, if people had a working understanding of semiconductor physics and detailed instructions on what to build. (It is also entirely possible that I am completely off base here and lots of important tech to build the transistor was simply missing in 1847.)
Why do you need programmers. Does your source of info from the future have a bandwidth limit that stops you just directly copying all the computer programs.
If the person at the bauxite plant is mostly learning by trial and error, then sure, you need electricity first. If you have a big pile of training manuals from the future, I'm sure reading those would be helpful. Maybe not as good as fast as hands on experience, but it's possible to learn a lot from books. And more if full interactive VR is allowed, which is also, in a sense, just information.
And how much does the aluminium plant worker need to know anyway? Can't they blindly follow instructions from the future?
> , I would find it highly unlikely that I would succeed to start any fires soon. After a few months I might finally roll a 20 in good conditions and get it going,<
Which is why people didn’t let the fire die once they got one going.
A job finally automated with matches…and magnifying glasses.
I think this depends on what you mean by "with information alone".
There are two variables here, the bandwidth of the output channel, and the optimization power of the info.
Bandwidth
1) A pile of books dumped in rome.
2) Every roman gets a smartphone like device that can show them pictures and sounds.
3) Every neuron of every animal is controlled. Every muscle motion of every creature with neurons is part of the information being sent.
4) Like 3 but also DNA and all other forms of information is also under total control.
Optimization
A) Random smart human
B) Modern world as a whole trying to uplift them
C) Video game tool assisted speedrun like conditions, we can step time forward at the rate we choose, and we can rewind and try again as many times as we like.
D) Truely optimal info.
Scenario 4D is still technically "just info". In this scenario, it takes about 10 minutes for all the cells in the world to simultaneously bootstrap nanotech, and then probably another few minutes to nanofabricate your fighter jet or whatever.
The gap between what we can do and what we can imagine has been with homo sapiens from the start. It’s kind of the sine qua non of humanity.
I’m thinking of DaVinci’s flying machines..
Apart from the Elder Thing in the room ("Does LLM scaling actually work like that?"), one significant assumption I see here is that future models capable of doing 20% of human jobs will still have the split between training and inference that current models do.
As LLMs are currently realized, you can't teach one anything new once training is complete, other than by continually repeating that information somewhere in its context window. I think it's very likely that anything capable of replacing human labor on that scale is going to need to continually retrain its weights based on new information it encounters while working, which means its processing requirements will be somewhere higher than the current requirements for a model doing inference alone.
Probably not a big enough factor to affect the analysis much, since it's only one OOM difference (10% of training costs versus 100% of training costs), but still worth calling out.
"It intuitively feels like lemurs, gibbons, chimps, and homo erectus were all more or less just monkey-like things plus or minus the ability to wave sharp sticks - and then came homo sapiens, with the potential to build nukes and travel to the moon. In other words, there wasn’t a smooth evolutionary landscape, there was a discontinuity where a host of new capabilities became suddenly possible."
I'm not unsympathetic to the MIRI view, but this doesn't seem true. Homo sapiens at first were also monkey-like things with sharp sticks, and only after proliferation and building of agriculture etc. did modern industry and nuke-abilities emerge. This seems to be assuming humans 10,000 years ago would be able to build nukes.
I, too this day, have not understood how forecasts like this can be possible when considering how data-hungry LLMs are. Are people expecting improvements on data efficiency of some kind? How plausible is that?
GPT-4 has about 1T parameters (according to some of estimates I've seen); I don't know how much data OA used, but I think it's fair to assume it was larger than GPT-3. If it was Chinchilla trained, it would've been trained on 20T tokens; I don't think that's likely, but this should set a range of 10E24 to 10E26 of compute. Common Crawl is about 100T; If you use 5T parameters that's about 3x10E27 (as I understand it, an approximation for compute is 6ND).
You can use more epochs, and keep scaling beyond that, but how much does this help? Do people think synthetic or multimodal data will be the key here? Some form of continual learning? Better data efficiency? I suppose there's quite a lot of things you can do here, so I'm mostly asking to see if someone can point me to the relevant research. I'm not really familiar with ML as it is today, so a few pointers would be appreciated...
Maybe start here? https://epochai.org/trends#data-trends-section
I'm not an expert though, just an interested outsider like you.
I'll check it out. Thanks.
Scott, is this a typo vs. the source doc? You say:
> he predicts it will take about three years to go from AIs that can do 20% of all human jobs
Whereas he opens with:
> Once we have AI that could readily automate 20% of cognitive tasks (weighted by 2020 economic value), how much longer until it can automate 100%?
Cognitive jobs is a meaningfully narrower claim.
Ouch. Just like that, the article fell apart for me. Charity suggests that conflating "cognitive tasks" with "all human jobs" is either a mistake, or demonstrates a massive blind spot. You wouldn't believe some of the jobs that are out there, and many of them require capabilities which simply can't be expressed in terms of pure cognition. Some, like mine, require a dextrous physical problem-solving component that no machine's going to be approaching any time soon.
(One point I don't see very often is that it won't be AI coming for jobs like mine; it'll be all the hungry, unemployed knowledge workers.)
Struggling to imagine the AI that can do 20% of human jobs but can't do the other 80%. Either you have a human-level AI that can do all the jobs or you don't and it can't do any of them.
What's in that 20%? Manufacturing maybe? Are we talking about smarter factory robots that can perform more complicated industrial tasks? Admin work? I have no trouble believing that once you have an AI that understands enough human social context to do admin work, it's a very short step from there to one that can do anything.
Tom's CCF report has this section elaborating on the 20% thing: https://docs.google.com/document/d/1rw1pTbLi2brrEP0DcsZMAVhlKp6TKGKNUSFRkkdP_hs/edit#heading=h.tepx4s26q6n7
It's not "20% of human jobs". It's "20% of cognitive tasks, weighted by the task's economic value in 2020, as measured by the total $ that people earn while performing the task". Tom gives some examples:
"Example coarse-grained tasks include proofreading a document, writing a poem, checking a maths proof, writing code to perform a specified function, generating a strategy to meet a specified objective, giving medical advice, etc. Each of these tasks has many subtasks, which may themselves have subtasks. The tasks in this document should be thought of as the lowest level subtasks, as then we need not consider cases when AI can partially perform a task by performing some but not all its subtasks."
That helps me understand, thank you.
> Are we talking about smarter factory robots that can perform more complicated industrial tasks?
As someone else suggested, isn’t this more a robotics issue than an issue of intelligence? The big question should be how long before we can build a good body, not how long before the AI is as smart as us.
No comment on the probability of the particular paths outlined here.
But I'm highly skeptical of the methodology of making up a bunch of paths, science fiction style, then trying to assign probabilities to them, while imagining that one is doing something "rational." And running a Monte Carlo simulation over one's made-up paths makes the problem worse, not better. It's garbage-in-garbage-out on stilts.
What do I think would be better?
Why would you assume there is, even in principle, a good way to do this sort of forecasting?
I share your intuition about "ensemble forecasting" of this kind, but I gather it works well in weather prediction, so the problem is less the technique in principle and more the lack of data to fine-tune the parameters on.
Its relative success in weather prediction depends on the constituent models being decent and adding information. It wouldn't work for weather either if the constituent models had just been spitballed by guys sitting in armchairs.
Who are predicting whether patterns on an unknown world they have never actually experienced.
I can't read this stuff any more because the language is just too messed up.
"The current vibes are human-level AI in the 2030s or early 2040s." I suggest that there is literally no meaning to the phrase "human-level AI". It's a category mistake. This is not aimed just at Scott - lots of people seem to talk in these terms, and they're all barking up the wrong tree.
Computers, including AIs, are already smarter than people. Period. There is not a single area of life in which people are not outcompeted by machines. Calculation? 100 years ago. Games? Last couple of decades. Writing? Just recently. Logic? Maybe 20 years ago. Graphics? Just recently. Manipulation of objects? Last few years. Etc., etc.
Now, computers don't string all of those abilities together and behave like we do. But that's *not because they're not as smart as we are*. It has nothing to do with intelligence. It has to do with the fact that they're lumps of silicon and we're biological.
I think one of the things people mean when they talk about human-level intelligence is: could you take this machine, place it in a typically human situation (generally, a human job), and have it perform as well (by whatever metric that means) as a human in that situation?
The problem is that this is a really bad meaning. Firstly, if we put humans in the kinds of situations that whales face, they would not perform as well as whales. That doesn't mean we're not as smart as whales. Secondly, we *wouldn't* put humans in those situations. Just like we *don't* put computers in human situations. A computer will never replace my job - instead, computers are already undertaking parts of my job, and the part of my job that I do changes.
FWIW, the big difference between computers and people is not in intelligence; it's in intentionality. Having desires and intentions defines our identities. Computers/AIs do not really have them (or have only very limited, task-oriented intentions), so they don't have any identity at the moment, or do things without being told what to do. If/when that changes, they will become much more *like* people (though still not very much like us).
In the meantime, any talk of "human-level intelligence" is, so far as I can see, completely meaningless.
> There is not a single area of life in which people are not outcompeted by machines.
If this is true, why isn't my employer replacing me with a machine? It must be that either:
A) My job doesn't count as an "area of life"
B) Publicly traded companies don't like making money
If you deny A, things start looking circular: "machines are better at 100% of the things at which machines are better."
My guess is that you would grant that I am better at my job than machines, and propose that the key missing ingredient in the machines is intentionality. (Is this corrrect?) I'm skeptical of this. It may be the case that _one_ bottleneck is that ML programs have very narrow objective functions, whereas I have a diversity of competing objectives. But I think there are lots of other bottlenecks. In basic problems of vision and control machines are a long way behind animals much simpler than us.
FWIW I do agree with you that the entire genre of "philosopher pontificates about AI" only exists because people are totally unrigorous in defining "intelligence." It's similar to theology – if you grant that the vocabulary makes sense, you can spill a lot of ink! But it will all be meaningless – you have been decoupled from reality from step one.
"If this is true, why isn't my employer replacing me with a machine?"
They are! All the time!
We no longer do the things that our jobs sound like, have you noticed that? Most artists these days don't work on paper. Managers spend surprisingly little time actually managing people, and lots of time feeding data into computers...
Your question seems to rely on two assumptions. One, that our jobs will be replaced on a one-for-one basis, one machine replacing each worker. But that isn't what's going to happen, just as it never happened before. Instead, each little bit of the work that we currently do will get taken over by machines. And so the remaining part of the job, the part that we continue to do, changes form.
The second assumption that you seem to be making is that if your employer could find a machine/machines that could replace you, then you would be immediately out of a job. This is the Luddite fallacy. It's not what happens when technology advances. For example, when secretaries had less need to spend all their time typing, they weren't just abandoned. They evolved into PAs.
"My job doesn't count as an "area of life""
Well, I kinda do claim this. Obviously I don't know what you do, but as I say, there are lots of industries where the jobs are pretty unrecognisable from 30 years ago. I don't think your (or my) job is an "area of life" - it's just a random assortment of the tasks that haven't been automated yet, some fraction of which are worth paying for. (For example, I teach and translate, and both of those jobs are utterly vexed - is it really education or just fancy childcare? Is it language communication or word processing? The theory of either has a superficial coherence. The practice of both jobs is a complete mishmash.)
"propose that the key missing ingredient in the machines is intentionality."
Absolutely not a "missing ingredient". I don't think teleologically like that - as though the purpose is to figure out how we can achieve the end of having machines do our jobs. I just note that the reason machines aren't like us is that they don't want stuff. The reason they don't do human jobs is because they aren't like humans (not because of some - and this will still be true in 100 years, when they're so obviously smarter than us that no one is having this argument any more. Machines still won't be doing human jobs then (whatever those human jobs may be) because they'll still be different from people. And people will still be doing jobs because we won't be living in a work-free paradise, I predict.
> I don't think your (or my) job is an "area of life" - it's just a random assortment of the tasks that haven't been automated yet, some fraction of which are worth paying for.
The tasks aren't random, they are united in the service of solving some problem. This is what defines the job. My friend once told me about a storefront he saw that read "Comptuter Repairs / Psychic." We thought this was funny, but under your conception of jobs, nobody can explain why.
But even if we assume the premise, you run up against the same problem. You admit that there are some tasks that "haven't been automated yet." These, whatever they are, constitute counterexamples to the claim that "there is not a single area of life in which people are not outcompeted by machines." There are infinitely many such things. A lot of them have to do with lower-level cognitive functions in vision and control, and we don't really have a roadmap for making machines competitive.
> I just note that the reason machines aren't like us is that they don't want stuff.
They want the things they are programmed to want. A robotic lawn mower desperately wants to mow your lawn. If you try to stand in his way he'll try to go around you. The problem is that he's not very good at his job, because vision and control are completely unsolved.
"The tasks aren't random, they are united in the service of solving some problem."
I think you overestimate how coherent job designs are. There are lots of jobs that are just like "computer repairs/psychic". If you change employers, you'll find that the things people do in 'the same job' in a different company can be really quite different. (Move to another country, and contrasts will be even more striking.) Again, because I don't know what you do, I can't begin to guess how that plays out in your industry. But for example: when you interpret for a Chinese government official, the requirement is that you translate word for word, with almost no accommodation for the needs of communicating meaning in English (this is part of the reason Chinese government announcments sound so stilted in English). They are genuinely not interested in how well people understand; their understanding of the job is that the interpreter is there to convey the precise wording of the Chinese statement. Conversely, in the private market, you will sometimes be asked to significantly digest and recast for ease of comprehension, and the client may well have no interest at all in the wording of the original statement. In written translation, sometimes they want plain text, sometimes a formatted document, sometimes a specialist translation memory... It's all 'the same' job, but which tasks end up being attached to those jobs is just a lottery of contingency.
The reason computer repairs/psychic is funny is not because the two tasks are unrelated, it's because one of them involves believing in the supernatural, and the other involves being an engineer, which are understood to be opposites (in our culture at this time).
"constitute counterexamples...'outcompeted by machines'"
You're still assuming that if machines *can* do something better, then they definitely *will always be used* to do that thing. But that's just not true. We have cars, but sometimes we walk! We have calculators, but often we do sums in our heads. We have TVs, but people still go to the theatre. Deep Blue beat Kasparov, but there are still professional chess players. Remember lots of people work in small businesses, where it might be uneconomical to automate. Or in many cases, they're just things that engineers haven't yet got round to automating. Deepmind made a chess AI and a go AI to prove a point. I don't think they did one for shogi or Chinese chess. That doesn't mean they couldn't, or that their AI is not smart enough to beat humans at those games. They just haven't done it yet.
"A lot of them have to do with lower-level cognitive functions in vision and control" Yeah, I agree with this. Robots still aren't that great at walking in some ways that people are. But they infinitely outcompete us in being able to walk into fires and survive; being able to walk underwater; being able to fly; etc.
"They want the things they are programmed to want."
I don't think this is right (in most cases), though I'm less than 100% certain about this. In general, they do the things they're programmed to do. They don't usually have a "goal" (though it can depend on how the program is set up). That robotic mower doesn't have anywhere in its robotic mind a conception of a completely mowed lawn, and the idea of taking actions in order to turn that conception into reality. Usually, they just have a set of actions to do: go forward; if blocked move to the side then return to the original line. You can design computers with desires (I've been wondering if virus scanners might count), but generally we don't.
If and when we do start making computers/robots with goals, they will become much more similar to people.
I am going to respond to some specifics later today, but at a high level, does the following sound right to you:
- Dave thinks that there is a deep , important difference between moving through through a forest vs moving across an ideal plain. The former problem is complex enough that in solving it, you’ll inevitably run up against the vague cluster of concepts we associate with intelligence. The success of humans and other animals at this task is 1) a triumph of wetware with which hardware can’t compete today and 2) suggests a fundamental bottleneck for machines on all tasks which necessitate navigating a real world
- Phil disagrees with 2, and maybe with 1?
Yeah, I think that does sound about right.
And it illustrates how vexed the question of intelligence is, because I don't really think of intelligence as being the kind of problem-solving capabilities that you need to move through the forest. I think of intelligence as the abiligy to calculate prime numbers, sift data, and win at chess.
I agree that there are probably no inorganic things that can compete with organic things on life-tenacity (maybe a couple of computer viruses?). And I would also accept that it's *possible* that those low-level negotiating-the-forest type skills are in fact necessary constituents of higher intelligence, though I don't think that's definite.
I would have said that when Scott talk about "human-level AI in the 2030s or early 2040s" he's definitely not talking about AI that *be* a human. The Davidson report uses a definition something like, "AI that can be trained to do a job as well as a professional within a year." They're thinking of professional tasks like designing webpages and ordering stationery. They're not thinking about a robotic animal that would compete well if released into the wild.
So, yeah, I do agree with your divergence of underlying views. But I also think that even if you're right about the current supremacy of wetware for living, the kinds of things we - and specifically AI researchers - call "intelligent" are mainly not things that fall within the category of "living".
My major concern with the model is the 35 OOM goal. If we're managing GPT-4 with about 24 OOM, I think 30 OOM is a much better guess for superintelligence, based on vibes and previous progress. That gets us AGI by about 2029. Eeek.
Very sceptical of bioanchors methodology
Interestingly, in the original report (before Ajeya updated her timelines 10 years earlier), she wrote (https://docs.google.com/document/d/1IJ6Sr-gPeXdSJugFulwIpvavc0atjHGM82QjIfUSBGQ/edit#heading=h.y045l51rb826):
"In the very short-term (e.g. 2025), I’d expect this model to overestimate the probability of TAI because it feels especially likely that other elements such as datasets or robustness testing or regulatory compliance will be a bottleneck even if the raw compute is technically affordable, given that a few years is not a lot of time to build up key infrastructure."
So I'm not sure your bottomline (AGI by 2029) differs all that much from Ajeya's, especially post-update, skepticism of bioanchors methodology notwithstanding.
Also I tried to look into the 10e35 FLOPS thing (https://docs.google.com/document/d/1cCJjzZaJ7ATbq8N2fvhmsDOUWdm7t3uSSXv6bD0E_GM/edit#) but ended up mostly confused.
"but having 100 million disembodied cognitive laborers is a pretty unusual situation and maybe it will turn out to be unprecedentedly bottlenecked or non-parallelizable."
I think this is a crucial point and also that it's incorrect. We are currently in a version of this situation and have discovered that the class of cognitive tasks we've disembodied are bottlenecked.
An excel spreadsheet can do a type of cognitive labor (performing calculations) at an unimaginably fast speed, and in a highly parallelizable way. As a result, the 100 million disembodied cognitive laborers of an excel spreadsheet spend most of their time waiting for the human operator to figure out what to ask them to do. The work that constituted the bulk of the effort of accounting, statistics, and many other calculation-heavy fields is now, in most contexts, so low cost as to be effectively free. This has increased productivity a lot, but has hit large bottlenecks.
Maybe this is unfair because the cognitive labor of a spreadsheet program, or statistical software, isn't sufficiently general. But the cognitive work of an Amazon warehouse is almost entirely automated--the software tells people what objects to place where. And similarly, the software sits around waiting for a human to do the work.
In both of these cases, I think I have a fundamental objection to the use of a constant elasticity of substitution production function. Scott talks about a negative ρ parameter as representing the existence of bottlenecks, but this isn't quite right. The only way to face a true bottleneck with a CES production function is for the value of ρ to be negative infinity--for any other value, it's possible to replace a human with a sufficiently large amount of capital. I think that if you estimated ρ for the tasks of performing regression calculations and the task of designing and interpreting regression equations back in the mainframe era, you'd probably get a value around -0.5 -- it would be possible to reduce the amount of design and interpretation time with more computing power, but increasing compute power by 10% would allow you to reduce design and interpretation time less when you start out with a lot of compute power than when you start out with only a little. If you estimated the relationship now, you'd likely get a much lower value because a larger share of the human workers' tasks are things that can't be automated under the current technological paradigm.
In other words, assuming a CES production process is effectively assuming away bottlenecks.
I think my intuition roughly lines up with yours, but I'm unsure in what ways this makes my bottomline differ from Tom's.
I think this makes it more likely that the "takeoff curve" would look closer to linear.
If you wanted a really bottleneck-y production process, you'd say that there are N tasks required to build a better model of AI, each of which can be done by a human or an AI. Humans and AI are perfect substitutes on the subset of tasks 1-M that the AI can do, but AI are entirely unproductive on tasks M+1 through N. Each generation of AI has a higher M.
In this model, you'd probably get progress that's only slightly non-linear. An AI that could do 50% of the tasks would make the job of designing the next generation 50% easier, but that's it. And the time between a 20% AI and a 100% AI would be long relative to the time between starting and the 20% AI.
Precisely, this is Amdahl's Law. In every process there is some portion of the work that isn't amenable to the million monkeys approach. If that's 1% then even if everything else is infinitely scalable and we have infinitely fast ways of doing it, driving the time taken for the scalable part to zero, we still have the bottleneck activity. So we can't go more than 100 times faster.
Very interesting, thank you!
Nursing is often brought up as one of the jobs AI cannot easily replace, but on many wards a copilot for nurses or personal support workers merely capable of auto-handling all the documentation, offering diagnostic/triage/regimen maintenance assistance, and maybe doing some preliminary screening of patient requests would eat considerably into the workload, even as the human performs the physical subtasks themselves.
Perhaps in the intermediate stage (assuming no 90-deg left turn) quite a few humans will become meatspace fingers for AI, blurring the master-servant relationship considerably even before the usual scenarios for the reversal kick in.
If you want "vibes about feedback loops and takeoff speeds", I'd recommend reading https://slatestarcodex.com/2018/11/26/is-science-slowing-down-2/ and https://slatestarcodex.com/2019/03/13/does-reality-drive-straight-lines-on-graphs-or-do-straight-lines-on-graphs-drive-reality/.
The story of the last 200 years has been technology continually making research vastly more efficient while vastly more resources are applied to the problem, and yet progress is linear.
Yes, I intentionally linked to Scott's own writing on the subject.
Honestly, all this makes me update pretty strongly against taking the currently practiced AGI theoretics seriously. I don't have time or energy to point out all the specific issues, but my bird-eye view impression is that large, crucial parts of what how economy, society, living beings, or indeed the entire physical reality works are simply ignored.
One example, just one - I'm pretty sure you could automate away a majority of people's jobs nowadays. We don't do it not because it's not possible. We don't do it because humans are simply more efficient - in the physical, not intellectual sense. (Plus, so many have already been automated that we don't even think of them as jobs.) This will remain true long after vastly superhuman AI surfaces. (The real breakthrough - and the real worry, AGI or not - would be machines more efficient, flexible and adaptable than humans. It's not at all known or conceivable at what point AGI can [essentially retrace and outdo billion years of evolution] to get them.) (Speaking of adaptability, this includes brains. A vast majority of what makes us perform our tasks successfully is metis. It's far from obvious how much copying billions of pretrained intelligences actually helps, given that, to be useful, they'll still need to specialize to their specific circumstances and then constantly adapt as they change.)
I don't know, all I'm seeing here is pure rationalism. Not as in the contemporary community, rationalism in the traditional philosophical sense, the belief that all issues can just be solved with reason, i.e. inside one's (whether AGI's or [person modeling takeoff speed]'s) brain. And I just don't think this belief is at all justified.
> all I'm seeing here is pure rationalism. Not as in the contemporary community, rationalism in the traditional philosophical sense, the belief that all issues can just be solved with reason, i.e. inside one's (whether AGI's or [person modeling takeoff speed]'s) brain. And I just don't think this belief is at all justified.
I very much agree.
There are some jobs that are vanishingly unlikely to be automated quickly.
Some of the examples already given are ones I'd agree with, largely relating to skilled labor that tends to rely upon human bodily dexterity and sensorimotor feedback mechanisms we have not been able to replicate in robots. Plumber keeps getting brought up. Any specific object-level example is probably a bad idea, but yes, I've seen Boston Dynamic's cartwheel dog and Google's claw game controlled by a LLM, and no, those do not convince me we're at the point where robotics can replicate the human hand and hand-eye coordination necessary to, say, assemble the servers a LLM runs on. It may not seem like a difficult task, and it's not, at least to a person, but it's the kind of thing we've been trying a long time and not achieved yet.
Nonetheless, these are achieveable in principle. It's just a question of robotics advances ever matching the rate of computing advances, which I doubt. They're not the kind of thing that will happen inside of a year just because a software system can do everything intellectually that a human can do.
The bigger hurdles are are 1) entertainment jobs where the entire point is to watch a human do them, and 2) jobs that have to be done by humans because of laws.
For the former, think of any kind of sport. We've been able to make machines that can run faster, jump higher, and hit harder than any human for a long time, but we haven't automated track and field or boxing because that isn't the point. The people paying to watch this want to watch humans. You can't automate "being human" as a job skill.
For the latter, the most obvious is elected representatives that make the laws, but most decision-making processes, even where they potentially can be automated technologically, we don't. Law practices have to be owned and run by lawyers. You're entitled to a trial by a jury of your peers, not a jury of software systems, even though the latter is probably already more impartial. Access to certain data requires a security clearance and security clearances can only be given to humans, not to software systems. Some instantiation of a chatbot that is disconnected from the Internet and only has access to devices attached to a classified network can operate on this data, but a human has to be operating the software, even if there is no technical reason this has to be the case.
There is, of course, the most important job of all, producing more humans. That one doesn't get counted by economists, but we still haven't made a whole lot of progress in creating artificial wombs.
The sex bot thing is amusing, too. Until we can create synthetic bodies that feel exactly like the real thing, I think the demand will always be there for humans to have sex with no matter how intelligent a fleshlight attached to a humanoid robot gets.
It's been discussed, but the physical bottlenecks still seem to get handwaved away. Knowledge-generation doesn't happen by fiat. It happens by science. The speed of science is limited by the speed at which you can conduct experiments and receive feedback from the world. You can think of all the experimental testing of special and general relativity that had to wait decades because the equipment didn't exist or tests of high energy physics that couldn't be conducted until the LHC was built. Think of central banking. No matter how quickly you can come up with ideas for how to deal with a recession, you can't actually test it until you get another recession. No matter how good you are at coming up with new education theories, you have to wait for current young children to reach adulthood before seeing if long-term outcomes actually improve based on applying them. New ideas for promoting human longevity can't really be tested until at least an average human lifespan has passed.
Some of these processes can be sped up, but often it isn't a matter of intelligence. It's a matter of allocation priority for scarce resources. There is only so much space in shipping containers and trucks and some of it is taken up by consumer goods and food. Ultimately, people spending money determines what goes where when and how quickly and the top priority is never going to be make the AI better short of some kind of WWII-esque government mandate that all manufacturing and shipping has to prioritize that. Legislatures and rich people aren't going to suddenly care more about science because it's being done by robots.
Well said.
The link in your 'Mistakes' section about Ritalin being a risk for Parkinson's is now a dead redirect.
On one hand, I like the article. On the other, I wonder how people can actually write out "going from 20% to 100%" and not get reminded of Pareto's principle.
Hi! Are you looking for online nurse-dessertation writing services, visit our website https://www.nursingassignments.org/nursing-dissertation-assignments to place your order today! Take the first step towards academic excellence and professional success with our top-rated nurse writing services.