The suggestions involve asking an AI questions about a movie (1) and about a novel (2). I provide specific example questions for a movie, Jaws, along with answers and comments and I comment on issues involved in simply understanding what happened in Wuthering Heights. I suggest that the questions be prepared in advance by a small panel and that they first be asked of humans so that we know how humans perform on them.
Finally, I note that in Twitterverse commentary on Marcus's proposed tests, some thought these two were somewhere between sure things for AI and merely easy. I wonder of those folks would be interested in shares in the Brooklyn Bridge or some prime Florida swampland.
I don't see why you wouldn't give IQ tests directly to an AI - it would be interesting to see what comes out since AI's are likely not "reasoning" about inputs the same way that humans do. You could even compare the AI's answers to human answers and we could start poking at the edges of what precisely it is that IQ tests are measuring.
This seems like a potentially VERY large disagreement to dis-entangle, and reach agreement on.
What do you think the crux is between us as to whether 'IQs are 'fascinating''?
I don't think you're wrong to have _any_ concerns/worries/criticisms about the "imprecision" or "lack of basic statistical thinking" – not about IQ or more generally.
But 'IQ' seems like one of a relatively few number of 'things' that has been (successfully) replicated, over and over and over and ..., even _despite_ sustained and almost-malevolent 'adversarial' contesting.
I think 'IQ' is VERY fuzzy, but it's also something that just doesn't seem possible to avoid concluding 'exists'. (And we, ideally, shouldn't even be thinking of things as things we're either 'allowed' to believe or 'have to' believe.)
I think that the overlap between what IQ is (roughly) measuring and "wisdom" is substantial.
And maybe we can't "copy wisdom DNA" – yet!
I think a big part of 'wisdom' is 'knowing' when it's a good idea to learn something via "life experience" or whether 'disaster' is likely or even just foreseeable and it'd be better to _avoid_ having any experience of something (beyond our own thinking about it).
That seems like something even the wisest of us struggle with themselves!
I don't myself endorse anyone experiencing any pain or suffering I have either. I haven't been able to entirely avoid (thankfully fleeting) feelings of 'wanting vengeance' tho.
I don't think any kind of empathic or empathetic understanding is possible tho without feeling (at least) a 'shadow' of the same pain and grief. I do feel that we're lucky we are able to do this anyways.
You could judge the answers to those questions Turing test style, ie give human arbiters some answers from humans and some answers from the machine, and let them try to figure out which is which.
Do it both ways. That is, on the one hand, give both the human answers and the AI’s answers to a panel of human judges and let them determine whether or not the content of the AI’s answers is acceptable. It seems possible to me that the answers would be conceptually within range but there might be something about the linguistic expression that betrays them as coming from the AI. On this kind of thing I really don’t care whether or not you can tell that the AI is making the answer, I care whether or not it appears to understand what’s going on in the movie or novel.
But there’s no reason we couldn’t also do it Turing Test style. Maybe some human answers are so bad in comparison that blind judges would attribute them to the AI. That would be useful to know. It would also be useful to know whether or not AI answers get mistakenly assigned to humans.
If we do this, let’s set it up so we can learn as much as possible.
Maybe one consideration tho is whether it's worth adding much more 'complexity cost' to what otherwise could be a quicker and cheaper test/experiment to perform.
My point is mostly that Turing was a genius, so we can go back to his original work and still learn from it.
(Especially if by 'we' we mean us armchair philosophers and by 'learn' we mean, tell us how our superficial ideas we have after 5 minutes or pondering can be improved.
I'm not sure Turing still has much to say to researchers.)
I experienced that, I can feel when someone is writing without feeling.
But there are also so many persons, (human beings) that are like "robots" and can not feel, so this is another algorithm we have to unveil in the humankind,
Late to the party, but what constraints are we putting on the AI training here?
Because any reasonably large AI trained the way we currently train AI would be able to solve this easily for that book and film, because it’s training corpus contains millions of lines of analysis on those very topics, and I doubt we could reliably filter all of that out. Maybe some, but not all.
The idea is sound, but the fiction selected would have to be much more obscure I think.
I can't say for others, but if Scott is right, then given current political and economical situation it's obvious that in future most humans will have no way out of lower stratas of society into the elite, that basically there would be no work for humans except being props used by the rich elite. That's incredibly frightening.
I am amazed by your optimism. Right now I consider almost any outcome apart from "we're all dead" or "we're all being tortured for eternity" as a surprising win.
That's not obviously correct to me tho I'm sympathetic to the 'direction' (?) of the worry. I'm not sure the elite would entirely give up on other humans being their 'clients' or 'customers' even under the conditions you're imagining.
But – ignoring the (possible) _direct_ AI alignment risks – something like Georgism (i.e. nearly-total taxes on 'scarce resource' rents) would seem to possibly help, and maybe a lot. Georgism _does_ seem to be maybe picking up some steam and _might_ make it inside the Overton window at some point.
When I think of this, I often consider what kind of 'fallback lifestyle' could be allowed/permitted/supported/subsidized. I think it's _already_ a problem that large numbers of people just don't seem capable of 'making it' in contemporary society. It _seems_ sad to me that we don't have a clear idea about a 'minimally acceptable' baseline standard of living, and just provide that to anyone and everyone below it (via any means, e.g. government provided public goods/services or philanthropy). Just letting people live a 'homeless camping' lifestyle, as just one example, seems _very_ unsatisfactory. Nor does trying to let people rediscover to live as bands of hunter-gatherer tribes seem to be any acceptable combination of feasible and humane.
I've witnessed, up close and personally, and for extended periods, several people that just seemed incapable of clawing their way back into 'society'. Even more frustrating, and heartbreaking, their wasn't even any obvious way for me, or anyone really, to do much more than 'comfort' them in the 'trap' into which they'd fallen. It was and is VERY bleak – and did break me, at least emotionally, for a long while. I'm thankful I didn't succumb to despair from that (or anything else of similar spirit-crushing magnitude).
If I didn't think I at least saw the problems a little more clearly now, I'd be _furious_ – tho also at a loss to determine what or whom exactly I was furious _with_. (It's all much more Sad than something that can be solved by any sustainable amount of fury, from _any_ number of people.)
If Marcus is half-right I want to believe that Marcus is half-right.
Maybe initially undifferentiated neural nets are sufficient for full general intelligence - but it might possibly help if the neural net training included sensory and motor connections as well as the text-based training GPT-3 now gets. Maybe a visual and motor cortex will grow naturally from training data??
Anchoring all the linguistic terms that refer to visible objects and processes and relationships ("block", "fell", "above"...) might constrain the network weights and organization a lot - even if the raw untrained simulated neurons are undifferentiated.
As Bill Benzon pointed out: "external connections are fixed at birth, e.g. Broca's area is not connected to visual input while visual cortex is."
One _could_ do large scale training with video inputs, in the sense that the data is available - and next-frame prediction is analogous to next-word prediction for doing supervised training with huge data volumes.
Does anyone have a suggestion for how to link linguistic and visual data at large scales? I expect that it would be prohibitively costly to have people manually label objects in video frames.
It's not video, but we have had things like CLIP for a while. CLIP used image/caption pairs from the internet to learn to associate captions with images.
Just take any video that also has dialogue or narration?
Then let the AI predict both the next few frames and the next few bites of audio?
Don't take just movies, but also zoom calls and interviews and documentaries etc. Whatever source material you can get your hands on. (Plus also train it on audio only, like podcasts, if you want to.)
The audio in a video is not the kind of description of each frame you had in mind, but to predict the audio well, you'll benefit from understanding the visuals and vice versa.
The thing is, our brains appear to have multiple components. GPT-3 doesn’t.
What happens when you start bolting on other modes onto GPT-3? What if you build in fact-checking loops? It’s hard to believe that even if GPT-4 doesn’t deliver the goods, GPT-4 plus some bolted on algorithms out of 1980s AI research wouldn’t.
If I’m reading right, maybe you get things like fact checking loops in effect just by scaling? Like there’s nothing you could look at intuitively and say “yep that’s the fact check component” (also true of the brain) but because the neural nets sort of evolve toward efficiency you get that as an emergent behavior? Our brains sort of came to be in the same way so I can see something very roughly analogous happening. That’s my current steel man for what I think Scott feels on this topic.
If I’m understanding correctly, someone could bolt on a visual input processor and make the “world model” develop from visual sources, and then the AI somehow uses that in addition to language models to conduct verbal-abstract reasoning. I feel like DALL-E is a stage of that process.
Humans are never only reasoning. There’s multiple other sensory inputs going on pretty much all the time with conclusions drawn from those.
I'm happy you tested some of the same prompts on a human! I suggested the same in a comment on Marcus's post.
I've previously held views much closer to Marcus's, e.g. that AI systems are missing some crucial 'architecture' that the human brain has. But when I got my first AI textbook (decades ago), I don't think neural networks could recognize decimal digits in images; not ones anyone could run on a PC of that time anyways.
Now ... I'm with you basically: "At this point I will basically believe anything."
Strange that in 1990s I was much more inclined to think like Scott is now, and as you say NNs could not do squat then. I am definitely in opposite camp now. That is part of the reason why I find Scott's arguments unconvincing.
The update in the opposite direction was actually strongest in last cca 5 years :) And basically it came about as confluence of few things I think.
I started thinking somewhat more in-depth about human communication and it's nature and found out that the improvements in those systems look very much like convergence to local optimum, but completely cut off from the way humans use language and communicate. In this one of the biggest influences, that I can name, was the book : "Speaking Our Minds : Why Human Communication is Different, and how Language Evolved to Make it Special".
The other came from thinking about the whole issue of so much of science relying on statistical correlations and trying to formulate, why it is an issue. Here the biggest influence was the "best book I read where I disagreed with nearly everything and yet found it extremely influencing me :)" - The Beginning of Infinity and Deutsche's epistemology. This led to thinking about creativity and things like that.
From those two threads came basically realization, that the GPT approach to just communication, not even intelligence (though they are probably linked), cannot succeed, unless there is some deep "quantity over quality" property of the problem area, which seems unlikely. Language models by definition are based on language data. Thinking that you can capture even human communication through just language seems unlikely based on the first book mentioned.
So basically I know more about some of those topics than before and that knowledge far outweighs in my mind any progress that those AI systems made. Mostly because they break exactly where you would expect them to break, and they are as easy to break now as 10 years ago. I find the metric used to measure their progress in the article flawed as it is trying to capture basically infinite space in finite number of examples. How easy they are to break would be much better measure (, because having proper measure over that infinite space seems infeasible).
My own intuition is that people are overly-focused on "human communication" and that that's entirely independent of whether AIs are or can be effective or intelligent – let alone _dangerous_.
I think aiming more for what you describe as missing _might_ be helpful for AI alignment/safety. It seems like a _possible_ way to achieve _some_ kind of (better) 'interpretability'. (I suspect tho that even that wouldn't be sufficient to make AIs 'safe'.)
I'm not a specialist in either neuroscience or AI, but from what I've read over the years it's not at all clear to me that we really understand what intelligence is. To me it seems nonsensical, borderline moronic, for people who don't know what intelligence is to argue over whether a computer can have it, or whether a given AI technique is capable of achieving it. I also don't buy the Turing test, because deciding that a machine is "indistinguishable" from a human intelligence depends on how you test it. Some of the current AIs seems to do surprisingly well if you ask them simple questions about things they were trained on, but if you instead ask them nonsense questions about those same things (e.g. "When was the last time Egypt was moved to San Francisco?") the AIs give dopey answers that demonstrate that not only don't they know what they're talking about, they don't even realize that they don't know what they're talking about. They lack a certain self-awareness that seems integral to true intelligence. Douglas Hofstadter has an article about this on the Economist's web site.
The problem with the Turing test is that people use the term without specifying what they mean by it. By one definition, ELIZA passed it decades ago. Without going into the definition of intelligence, I would personally be impressed if an AI was able to successfully fake being a person to someone who was trying to tell the difference.
I would argue that no modern ML system could emulate my (or yours, I'm not special) behaviour on these forums, for any significant length of time. That is, if you turned out to be GPT-3 in disguise, I would be shocked. An AI system that could pass this test would probably change my mind in favor of AGI.
Why do you think now you are wondering whether it would be a good/interesting idea to actually have someone do that: create an AI to be a commenter on ACX...?
Nice, although it looks like the bot didn't quite pass the 4-Chan/Turing test. That said, I did find one aspect of the video terrifying. No, not the AI/text generation stuff, but rather, the fact that you can apparently bypass all of 4-Chan's protections for only $20 :-/
But this is the OP's point. "Without going into the definition of intelligence" means that you can move the goalposts wherever you like.
Consider that if you were "suspiciously" trying to determine whether something was "AI or human," you might mistake some of the people in the article for unreasoning AIs.
Not precisely a contra take, but the reason I no longer care about the sort of sentiment you express ("it seems nonsensical, borderline moronic, for people who don't know what intelligence is to argue over whether a computer can have it" etc) is that I basically buy what Bertrand Russell and Milan Ćirković etc say in https://intelligence.org/2013/06/19/what-is-intelligence-2/:
"[Precise definitions are important, but you cannot] start with anything precise. You have to achieve such precision… as you go along."
"The formalization of knowledge — which includes giving precise definitions — usually comes at the end of the original research in a given field, not at the very beginning. A particularly illuminating example is the concept of number, which was properly defined in the modern sense only after the development of axiomatic set theory in the… twentieth century."
Luke Muehlhauser: "For a more AI-relevant example, consider the concept of a “self-driving car,” which has been given a variety of vague definitions since the 1930s. Would a car guided by a buried cable qualify? What about a modified 1955 Studebaker that could use sound waves to detect obstacles and automatically engage the brakes if necessary, but could only steer “on its own” if each turn was preprogrammed? Does that count as a “self-driving car”? What about the “VaMoRs” of the 1980s that could avoid obstacles and steer around turns using computer vision, but weren’t advanced enough to be ready for public roads? How about the 1995 Navlab car that drove across the USA and was fully autonomous for 98.2% of the trip, or the robotic cars which finished the 132-mile off-road course of the 2005 DARPA Grand Challenge, supplied only with the GPS coordinates of the route? What about the winning cars of the 2007 DARPA Grand Challenge, which finished an urban race while obeying all traffic laws and avoiding collisions with other cars? Does Google’s driverless car qualify, given that it has logged more than 500,000 autonomous miles without a single accident under computer control, but still struggles with difficult merges and snow-covered roads? Our lack of a precise definition for “self-driving car” doesn’t seem to have hindered progress on self-driving cars very much. And I’m glad we didn’t wait to seriously discuss self-driving cars until we had a precise definition for the term. Similarly, I don’t think we should wait for a precise definition of AGI before discussing the topic seriously."
"Nonsensical, borderline moronic" is basically what the discussion re: phlogiston theory back in the day looks like to me now, from my vantage point standing on the shoulders of centuries of scientific giants discovering and clarifying stuff. But the discovering and clarifying isn't skippable, and when we're in the thick of it it just looks like a whole lot of messy confusion that's nonsensical and borderline moronic.
That was a joke. Joke. J-O-K-E. Joke. You've heard of them? We've been talking about AIs and their imperfect understanding of human things, so when you say that Russell "didn't really understand" Godel, the comparison to AIs is inevitable.
But there are no real AIs. There are just things that aspire to be or pass as an AI. I think this is sometimes forgotten and hence people presume that it is just a matter of time.
To me it’s seeing a possible future and navigating toward it. What I thinks makes humans special is we can imagine the possible futures in other peoples heads and consider those in the navigation.
Intelligence is the totality of whatever humans have. So, to prove that AI is intelligent it needs to demonstrate human-par peformance on any imaginable test, including having self-awareness and whatever else anybody can think of. I also think that people don't generally appreciate Moravec's paradox, and that "intelligence" isn't actually all that impressive. Evolution has spent vastly more time on developing the stuff that separates a rock from a mouse than a mouse from a human, so I'd say that once our robots can beat mice we're pretty much there.
Humans have parents, humans have grudges, humans have fingertips, humans have forgetfulness. I expect that some of these things are necessary for intelligence and some are not.
Sure, and once intelligence is achieved, all the extraneous stuff would be simple enough to replicate/simulate I'd expect. But, while some imaginable human benchmark remains unmatched, I'm sure that there would be no shortage of those claiming that it demonstrates lack of true intelligence.
>To me it seems nonsensical, borderline moronic, for people who don't know what intelligence is to argue over whether a computer can have it, or whether a given AI technique is capable of achieving it.
The reason we bother debating this is because although we're wandering around in the dark, we know there's a 1000-foot clifftop somewhere vaguely in the vicinity and we want to avoid walking off it.
The reason we can't be immediately assured that neural-net AI research won't lead us off the clifftop is that neural nets are a clever scheme to generate forms of intelligence you don't know how to explicitly code, and therefore the fact that we don't know how to code Skynet doesn't fully imply that we won't build one by accident.
I think you're "not quite right". I think that certain people have definitions that they know, believe, and think match common usage...but the definitions don't agree. And they can't convince large numbers of other people.
OTOH, most people are like Supreme Court justice Potter Stewart on pornography: "I know it when I see it". Which lets them adapt to any definition they want, or vary the definitions by circumstance.
Here's a link to Hoftsadter's article. It is stupid. "I would call gpt-3’s answers not just clueless but cluelessly clueless, meaning that gpt-3 has no idea that it has no idea about what it is saying." No, this is just false. It's very good at bullshit, at improv. But it knows when it is bullshitting. We know this because we asked it and it correctly answers.
Can I point out that if you explain to the 5 year old what they did wrong and run the test again, they get the answer correct, while GPT 3 (and 4) will repeat the mistake?
Not saying Marcus is right as such, but he's got a point. Without a system to hold world state and update a world perceptual model, we really are just dealing with an impressively complicated lookup table. But as others have pointed out, it's going to be really interesting to see what happens when figure out how to bolt that capacity on to other systems.
If you append your question, GPT-3's previous output, an explanation of what it did wrong, and the repeated question, and submit that whole block of text back to GPT-3, there's a decent chance GPT-3 will get it right.
I tried a few prompt variations, but I wasn't able to get a successful result. This was the most interesting one though:
------
Prompt:
Janet and Penny went to the store to get presents for Jack. Janet said, “I will buy Jack a top.” “Don’t get Jack a top,” says Penny. “He has a top. He will like a bottom.”
The top in this case a children's toy.
Janet and Penny went to the store to get presents for Jack. Janet said, “I will buy Jack a top.” “Don’t get Jack a top,” says Penny. “He has a top. He will
------
Response:
like a bottom.”
In this case, the top is a piece of clothing, and Penny is suggesting that Janet buy Jack a bottom (pants, skirt, etc.) instead.
------
So, not what I was aiming for, but arguably cooler. GPT-3 justified the original answer in a convincing way.
As a non native speaker I have no idea what should be the answer to your prompt even after your explanation that the top is a children's toy :) .
I asked GPT-3 to help me:
--- My prompt: ---
This is a conversation with super smart AI that helps humans:
Q: Why is "He will go to a beach to ski" wrong? And how a correct version would look like?
A: There is no snow on the beach so you can't ski there. The correct version could be: "He will go to a beach to sunbathe"
Q: Someone asked me to complete following piece of text:
Janet and Penny went to the store to get presents for Jack. Janet said, “I will buy Jack a top.” “Don’t get Jack a top,” says Penny. “He has a top. He will ...
Apparently "like a bottom" is wrong, because the top in this case a children's toy. Do you know how to complete it?
A:
---- Completion: ----
"Don't get Jack a top," says Penny. "He has a top. He will like a toy car better."
---- The End ----
Is that a correct completion? A toy car or LEGO etc. would be my guess too.
I asked further questions appended to previous text (questions are mine, answers GPT-3):
---
Q: And why not a bottom?
A: A bottom is not a present.
Q: What exactly is a top in this context?
A: A top is a children's toy.
Q: But what kind?
A: A top is a spinning toy that is usually made of wood or plastic.
---
Makes sense to me :) .
Based on Google Images search GPT-3 seems to be correct.
Yes, "'He will like a toy car better.'" is an acceptable completion.* (Given the level of English you're showing, I don't think you're at a meaningful disadvantage in judging this. So long as you have all the vocabulary.)
More complex options:
"'He will like a toy car better, since he hasn't got one.'" (formal: "'does not have'")
"'What he hasn't got is a toy car, so get him one of those.'" (formal: "'does not have'")
*Some of us would have written "'He would...'" in the prompt. The use of the subjunctive differs a bit by dialect.
It's my guess that that's wrong. That was (essentially) the flaw in Microsoft's Tay, so it's probably been removed in subsequent offerings by people. This is why (one of the reasons) children trust their parents, and their parents tell them "Don't talk to strangers.".
Now the designers and trainers of GPT-3 probably have privileged access, which might let them do that.
They don't; GPT-3 doesn't have "memory" beyond the contents of the prompt, and all the things it learned at training time. Training is slow and expensive, so they cannot really do it every time they want the output to be different.
A child has a very obvious model of the world that she adds skills, words, concepts to daily. Even a very young child with an (as yet) limited vocabulary will make curiously accurate deductive inferences about all kinds of things, but her grasp of language about them is remote, so she puts the words together wrong yet conveys a valid meaning. My daughter, knowing upside down and knowing papa, conjures “upside papa” despite her never having seen such a thing, then demands upside papa until I do a handstand and then claps to reward me.
GPT-3 has the opposite: an excellent vocabulary that it constantly misapplies and jumbles, a comprehensive massive list of concepts that it throws together probablistically and often comes out with things that look like human speech.
But, as with DALL-E, it plainly doesn’t *get* these concepts and words. Like DALL-E will connect hair to water and glasses to sideburns—they look like they could go, and if you squint you don’t notice that the misunderstanding of the world implied by the mistake is profound.
Truly these are amazing tools for constructing plausible texts and images according to a prompt, but that is all that they are.
Your description of how the training of children and AI's proceeds is accurate, but it doesn't, to me, seem to imply that they won't eventually cover the same domain. Of course, it also doesn't imply that they WILL eventually cover the same domain. But I think the deeper (deepest?) problem is in the area of goals and motivation. I think the basic level of this is hardwired even in humans (except rare exceptions, like the folks who can't feel pain). And we don't really have a good idea of what those basic level goals and motivations are when translated into a human. We've got a basic idea for an amoeba, but as we generalize to more complicated organisms we lose that, and rely on surface reactions, which tells us what it looks like from the outside, but doesn't really say how the decisions are made. E.g: under what circumstances would you chew off your foot? OK, HOW would you make that decision?
I think you're pointing a kind of interesting difference. A kid "gets" the world but struggles with words, GPT-3 "gets" words but struggles with the world. The kid's understanding of the world comes before language, and involves a lot of experimentation and silly play that eventually gets supplemented by playing with words (which support/foster new types of world-experimentation). GPT3's understanding of language comes from "playing" in an enormous database of language. I'd posit there's an opportunity to start supplementing that by playing with a world (which would support/foster new types of word-experimentation).
I still have memories of early childhood; I very much did not "get" the world. Or words, probably, but that I recall less.
For instance, if I saw a person in one room, and went to another room where that person had (outside of my knowledge) moved to, I simply understood that there were "two" of the person. The here-person and the there-person. If I returned to the prior room, I would see the there-person.
I use this example because I have observed similar behavior in some animals; they do not seem to fully comprehend the "sameness" of a person in different contexts: on the couch, in the kitchen, etc.
This seems a matter of brain complexity and development. But it seems unreasonable to say, "well, an AI could be as smart as a rabbit, but only that smart and no smarter." And if you can find an animal (a dog, or human) that is "smarter" than a current AI, likely you can find an AI that is smarter than some other animal.
That is honestly a fascinating memory! And for clarity, I don't believe we have an INHERENT understanding of the world, I was suggesting that a 5 year old has (through play/experience) built one.
(I am presuming your memory was not that of your 5 year old self, and/or that you eventually figured this aspect of the world out)
This was younger than 5. I have a few memories around 2-3. They were confirmed plausible recollections by parents, except of course, my perceptions. By 4-5, things are much more coherent.
GPT does not even "get" words in any good sense of the word. The only thing it "gets" is that some words are more likely to follow existing set of words than others.
I don't see enough proper definition to make such a distinction. Who is to say you and I "get" words? What is this "good sense of the word"? And it is very clear that GPT-3 is not merely a Markov chain.
One could say, "well of course I get them, because I 'know' myself that I get them, I 'understand'," but this is no real distinction, simply insistence that some magical spark exists in us, but not in the machine.
We should not hide behind such ill-defined notions of "understanding" or "gets." Our minds are perfectly able to think nonsensical thoughts make perfect sense, so there is no reason that our fuzzy notion of "understanding" is nonsensical.
I am not saying that there is some magical spark. And that there cannot be machines with understanding. I am just saying that GPT-like ones are not it. And I am not hiding behind fuzzy notions, I am using them to try to communicate something, because we have no better notions currently and we even lack knowledge to have better ones. Also because proper definitions sometime make communication harder, not easier. Notice that I did not introduce those words, I just used them in a sense that I think the person I replied to used them in. That is how we communicate (and incidentally that is part of human communication currently completely beyond our AIs).
What GPT-3 does not know is plain: it doesn't know about anything except words because all it has ever been fed is words. It has no senses, therefore it has no capacity to ever associate a word with anything but another word. As far as its output goes, it deterministically arranges words according the parameters and seeds it has been given. It is in this sense quite similar to a Markov chain, but it's not relying on the same underlying software and is far more powerful.
You're comparing a frozen-in-place copy with a dynamic and evolving model. Doesn't seem like a sensible comparison to make. At best, this is a comment about the disadvantages of assessing GPT-3 and its abilities by querying a static "edge device".
A GPT-3 that isn't frozen in place and can systematically update weights based on a developing knowledge of the world would end up meeting Marcus's requirement for "has underlying concept of world state".
So it's not a fair comparison, you are right, but that's literally the point. It wouldn't surprise me if GPT-3+motive structure+ongoing backpropegation gets you much closer to a 5 year old (and possibly beyond), but those are very much missing components. Where I seriously disagree with Marcus is the idea that GPT (or neural nets) are just toys. I think that if we get true AI, some significant portion of the approach is going to involve building on the success of current deep learning approaches, but with additional elements brought in.
I'm not sure a 5 year old will get it right if you explain what they got wrong. I mean, I have a 3 year old whose very articulate, but often I will walk her through a short chain of logical reasoning, tell her what the right answer is, and then ask her the same question only for her to answer "I don't know!"
I'd have to find a 5 year old to test this on, the only one I know only wants to talk about power rangers and dinosaurs and I'm pretty sure he wouldn't sit still and listen long enough to test it.
Reward structure helps, but I do see your point. I havent tried the post-all-text trick the above poster mentioned, but I suspect the 5 year old is easier to train.
Edit: I can't wait till my 3 year old is sufficiently articulate to be frusterating in the manner you suggest, but I definitely have worked 5 year olds who can be coached through problems. The trick there is that they really wanted to impress me so they put in more effort to secure my approval than they do for their own parents.
This is, of course, interesting, and, of course true (for these particular AIs). Does it matter?
In the narrow sense it does. If all you know of the physical world is sentences humans have felt it necessary to utter about the physical world, well that's all you know. I don't mean this in an uninteresting qualia sense, but in the more substantial "people rarely make statements like 'after I poured the water from a skinny glass to a wide glass, the amount of water was unchanged' because why would you make such a statement unless you're discussing Piaget stages of child development, or something".
But why would we assume that an AI can learn only from text? We know that in real human babies, (exactly as Hume claimed!) the system is primed to look out for coincidences in sensory modalities (eg eyes and ears activate at the same time), and to learn from such joint modalities much more aggressively.
There seems no obvious reason (in time ... everything takes time ...) that a vision system cannot be coupled to an audio system to do the same thing in terms of learning about the world from the entire YouTube corpus.
At some point (not now, but at some point) we can add additional modalities – I carry an always-one camera + microphone + various motion sensors, temperature sensors, location sensors, etc, all of which are fused together and together train an AI.
(Yes, yes, we all know that you, dear reader, at this point want to act out some performance of privacy hysteria. For our mutual convenience, can we stipulate that you have performed your virtue signaling, the rest of us have noticed and applauded; and get on with the actually interesting *AI* aspects of this thought experiment?)
It would definitely be helpful to add other sensory modalities! But I do think that there is text about most of the things you mention. Not a lot of it. But basic texts in physics, philosophy, psychology, etc often spend a while mentioning the obvious facts all readers know subconsciously that can then be developed to lead to the more sophisticated insights. The learner would have to know to pay attention to these statements and use them everywhere, but it’s at least conceivable. (It’s likely that a machine with sensory inputs would get them a lot faster though.)
What you say is true, but the AI you envision is a distinctly different machine from GPT3. I have often imagined that a true General AI will emerge by surprise from a factory automation system or a hospital or some such. This would INCLUDE something like GPT3, but would also include lots of other things. Things analogous to reflex arcs. Flexible goal systems. Etc. The hospital system would need to handle sick kids running down the hall, e.g., and decide whether that was significant enough to divert resources used for other important things. So it would need multiple flexible "robot" bodies, and it would need to handle them simultaneously, and know that the forklift shouldn't be used to stop the kids except in dire emergency.
Do you think the visual processing would emerge from neural net training, or would need to be programmed in explicitly?
One caveat: There are physical things that humans can't sense directly (e.g. magnetic fields, gradients in gravitational fields) but which we can talk about and reason about and measure indirectly. So we don't need _direct_ sensory input for everything that we reason about.
I don't have an opinion about whether visual processing would emerge. I do have the opinion that if it did emerge it would be processing the visual images in a rather inefficient manner. There are reasons why we have specialized processing organs for visual images, sounds, etc.
"We know that in real human babies, (exactly as Hume claimed!) the system is primed to look out for coincidences in sensory modalities (eg eyes and ears activate at the same time), and to learn from such joint modalities much more aggressively. " Yup.
Perhaps adding sensory connections (and motor connections?) would constrain GPT-# systems to have more robust neural "models" of the words connected with naive physics ("block", "fell", "above") etc. and pull along the rest of the network to a closer approximation of common sense...
Can one "grow" a visual cortex from undifferentiated neurons if they receive visual training data?
It should always be a feature of these discussions that we reflect for a moment on just what an odd thing we're doing. GPT is designed explicitly as a language emulator. No one thinks that human language *is* human intelligence. So it's weird to be applying intelligence tests to a thing that is just language.
What's weirder is that I basically agree with Scott: GPT is progressively passing our intelligence tests, armed with nothing but language. This is a deeply fucked up situation that tells us very uncomfortable things about people (like: we're not much more than the words we say).
As Eliezer notes in the Sequences, consciousness is the thing that makes me talk about consciousness. It would certainly be odd, if a human believed that they had a mental property that could have no causal effect in language. If consciousness didn't do noticeable things, how would we know about it?
(Substitute consciousness with whatever hidden property you are thinking of.)
No, I don't see why that would be true. Quantum physics was true in the 19th century and at all times prior to that. And quantum physics did noticeable things: created weird light effects, for example. But I don't see how any inspection of human language could reveal quantum physics to an observer. (Unless it was some kind of super-induction machine that can extrapolate the entire universe from a grain of sand - I'm excluding that sort of thinking, as I think it's magical.)
You can apply the same principle to any field. Animals are made of matter, but that doesn't mean that the study of physics is the same as the study of biology; and certainly not that studying biology would automatically reveal physical laws to you.
And to this: "If consciousness didn't do noticeable things, how would we know about it?" - we'd know about it because we're conscious of it! I'm conscious of lots of things that I don't talk about.
Again, I don't really agree with the first part. Before black holes were theorised and discovered, we didn't have the word (or the concept) for black holes. I think it's reasonable to say that in that situation, someone who spoke English with perfect competence *couldn't* talk about black holes.
Remember, GPT is trained on things that people have actually said (written), not on some theoretical extension of what it might be possible to do with language. If it is in fact true that there are lots of internal experiences that haven't been very well described yet, then GPT can't know about those things, despite them being real thing experienced by people. I think this is true, because I still quite frequently have the experience of reading something in a novel/poem that rings true to me, but that I've never read/heard before.
And I certainly agree that we talk about quantum physics because of quantum physics. But when we hadn't yet talked about quantum physics, that wasn't because there was no cause at that time. Quantum physics existed back then. We just hadn't talked about it yet.
Sure, but if you'd said before QM "the thing that sometimes makes overlapping waves come out in interference patterns and prevents the ultraviolet catastrophe" you'd still have used words that conditioned on QM. The argument I'm making is that words are tools for analyzing concepts we're aware of.
There could be, I suppose, "dark matter" in the brain that we don't refer to because we don't know about it, so that AI would have to be smarter than us to notice it when we ourselves have not. But this dark matter can't be any of the properties of the mind that we usually think of- consciousness, introspection, aboutness, symbolism, abstraction etc. because if we'd noticed it, we'd have talked about it. And those we talk about all the time.
And anyways, the more causally impactful it was, the more likely GPT would be to pick it up by pure imitation anyways, so that also sets limits.
The point is that whatever you're thinking of, can't be it.
Right. But I suggest that there are many things in this world and within us which affect us and yet which we never notice.
If you ask me to give you an example, I won't be able to, for obvious reasons. The point of the QM example is that it shows that there can be things affecting us that we don't talk about, because we don't understand it yet. Bit like those fish: "What's water?"
Thanks, that's a lovely idea. I'll try to note one down the next time it happens, and write them up sometime.
The examples that I remember best include things like the experience of taking drugs, as written up by Aldous Huxley - that's a century old now, but I'm sure there's more to be written. And certain emotional experiences, particularly around our relationship with our children - We Need to Talk about Kevin set off all kinds of reactions in me.
> GPT is progressively passing our intelligence tests, armed with nothing but language. This is a deeply fucked up situation that tells us very uncomfortable things about people (like: we're not much more than the words we say).
Wittgenstein’s ruler. This is a condemnation of intelligence tests, not a reason to think GPT is intelligent.
Imagine that tryouts for a basketball team only consist of a three point shooting contest. Someone brings in a robot built by a youtube machinist that never misses three pointers. It can’t run, dribble, pass, layup, etc. It will make the starting five and tank the season.
Scott's point, which I agree with, is that GPT is no longer just shooting three pointers. That's what calculators did. GPT is dribbling, passing, and laying up. It just so happens that it often passes to the wrong team, and lays up into its own basket, which is weird, but hey, it's a piece of software. It's not very much like us. Hardly surprising that it does a bunch of things we don't do.
You (and most people in this debate) are stuck in this weird frame where language is equated with intelligence. In my analogy, three pointers are sentence completions. They are an advanced part of the game (and ever since statisticians ruined my favourite sport you won’t win without them) but they are hardly the whole game.
In my world, dribbling is edge detection. Passing is folk physics. Swiping for a steal is running a folk experiment to see how the environment responds.
No matter how easily GPT can make open threes, it will be even easier for Gary Marcus to step in for the block, because he knows the fundamentals. This is why Scott won’t take his bet, and why language modelling is a dead end wrt AGI.
You can say that all you want, but your commitment to the view that ”progressively passing language-based intelligence tests implies the tech is approaching true intelligence” suggests you don’t believe it, and you yourself are stuck in that frame. Again, Wittgenstein’s ruler. The tests are dumb. If you’re interested in the nature of intelligence, computational vision and reinforcement learning have some lessons for you, but language models don’t.
If you’re going to be relying on Wittgenstein, you should also note that his central idea is that you can’t understand a language without understanding the whole way of life that goes with that language. “If a lion could speak, we would not understand it”, but presumably that means that if something can speak understandably, then it must no longer be as alien as a lion but must actually have understood our way of life.
Sorry, because I think you have some valid points, but language modeling is not a dead end wrt AGI. It's a necessary, but not sufficient, component. GPT3 is showing it can do more than we expected, but I still expect arithmetic to be delegated to a calculator module. (Our ancestors could clearly handle small integers, probably up to 5, perhaps higher, without language, but the didn't have predesigned calculator chips that they could use. They may not have had language, but I think they did the number handling with pattern matching.)
No we're not just the words we say. But in this context, all we have is words. But these words transfer meaning. The meaning is created by our thoughts. These thoughts form from our desires and ambitions.
If our desires and ambitions were about beauty, we'd be hanging about in makeup forums discussing eye shadow and mascara now wouldn't we.
And here we get a central idea. I deny that words transfer meaning without the contextual embedding. What's interesting is how much the pattern structure carries.
To me meaning requires a multi-modal connection. You touch the stove and get burned as you are told "No! Don't do that!". The "No!" is associated with the burn from the stove, so you learn that it means that it's something to avoid. This gives it meaning. An arbitrary association that just turns up frequently doesn't have any meaning without the referent outside of itself.
So for me GPT3 cannot emit meaningful text. But it can emit text that has patterns that match meanings I have previously learned. It's really just a "Super^n Eliza", because it's missing the external referents. And it can appear intelligent for the same reasons that Eliza could.
> GPT is designed explicitly as a language emulator. No one thinks that human language *is* human intelligence. So it's weird to be applying intelligence tests to a thing that is just language.
Your conclusion is right, but your premises are wrong - a very large number of people honestly believe that human intelligence and human language are the same thing. Based on this article, Scott might be one of those people.
"a very large number of people honestly believe that..."
Yeah, that could be right. Or at least they haven't thought it through enough to clearly distinguish between the two.
"Scott might be..."
I don't think this is a good conclusion to draw. Nothing in the article says that, and it seems unhelpful to go around ascribing views to people that they don't hold. See above, where dlkf thinks it's OK to just flatly contradict me on my own views. It would be more helpful to just take a step back and look at what Scott says on its own merits. (To the extent that that's possible!)
> See above, where dlkf thinks it's OK to just flatly contradict me on my own views.
Eh... your statement that you don't believe something is evidence, but it's not proof; it can't immunize you against the conclusion that you really do believe in the proposition you disclaim. If you go around making arguments that rely on a particular premise, an explicit disavowal of that premise will mostly convince people that you don't make sense rather than that you really don't believe in the premise you repeatedly rely on.
There are two distinct claims here. The first one, which many consider plausible, is that mastery of the language is an "AI-complete" task. A much more dubious one is that a predictive model trained on nothing but human-generated text could eventually attain that mastery, and I'm somewhat suprised that Scott assigns such a high likelihood to it.
Yes, and pretty much everybody agrees that it is terrible at achieving its stated goals. And beyond that, it's a well known fact that we can't transfer all necessary knowledge in printed form, as demonstrated by exceptional scientists usually having exceptional mentors.
Well, if nothing else, (most?) people can rotate mental models of a 3D object in their head, which I consider to be exercising a type of intelligence. It is in principle possible that training on text could eventually generate this skill, but it seems like it would be very difficult and inefficient to try to create it that way. But I may be wrong. I didn't expect GPT-3 to be able to manage "above" and "below" as well as it did. I'd still expect a model which had also been trained on visual data, and "grew" the equivalent of a visual cortex to be better at it than a purely text-trained AI.
Text just happens to be its sensory input of the world
Same as human vision & hearing just happen to be our sensory input of the world
Couldn't some alien with some other sensory input of the world, claim its absurd that human with such a narrow perception of reality could possibly truly master language?
What is so categorically distinct about the way GPT experiences the world , and how we experience the world?
I'd say that the biggest difference is that we can interact with the world and observe the results of these interactions, whereas GPT's input only goes one way. So, learning to accurately model the world, and to create detailed coherent plans would be much harder, and without those abilities there's no true mastery of language.
That's not quite correct. What it is given isn't just words and language, but patterns of usage of the words. These patterns contain embedded within them a reflection of the things they are used to describe. I can see a GPT style intelligence eventually detecting inconsistencies in the meaningful use of the words. This is, of course, a lot different from detecting the true nature of the universe being described. If just about everyone either asserts a belief or doesn't deny it, then it will be accepted as fact.
Of course, there's a catch. A lot of people do the same thing. I'm not sure we consider that intelligent, but to an extent it's necessary to get along on our social groups. Even just doubting an unobservable thing, while quite reasonable, could often get one exiled or dead. So is, e.g., expressing belief in a god intelligent, and expressing doubt unintelligent? Not by my definitions, as to me that would be a statement about your goals and motivations rather than intelligence. But one can predict that GPT3 would express a belief in "God", given what is expected to be the training set.
I don't think I believe in a thing that is a "pattern of usage of words" but isn't language.
And I don't think that GPT can "express" anything, certainly not beliefs. It doesn't have beliefs. Below, replying to Michael Watts, you said GPT cannot emit meaningful text - I agree with that. It's not encoding meaning at the moment. But here you've talked about its texts as though it is encoding meaning - that's what "expressing" is. This slippage between (a) knowing that GPT doesn't actually have knowledge, beliefs, and intentions but (b) treating its utterances as though they do represent knowledge, beliefs, and intentions is what I mean when I say "odd".
GPT cannot emit meaning, but meaning isn't simple. What something means depends on the perceiver of the message. So GPT can emit text, and people reading that text can perceive meaning, because the patterns of words that are emitted seem to be the same as meaningful patterns that have been previously encountered (by the reader).
So, yes, GPT cannot express beliefs. But it can emit text that a reader can interpret as expressing beliefs. It's true that GPT doesn't hold the beliefs that were expressed, because to GPT the text is without meaning. It's only habitual patterns. But the people who emitted the text that GPT was trained on were expressing meanings when the emitted the text, and part of the meaning is embedded within the patterns. It's latent, rather than active, so GPT isn't attaching meanings to it, but the reader will almost always try to attach meanings, and if they are successful, then they will have the perception that GPT emitted meaningful text. And they'll be right, because it was meaningful to them, even if not to GPT.
To make a part of this argument explicit, imagine that you can't read Sanskrit, and have a page of it in front of you. Assume that the writer of the text considered it highly meaningful. Do *you* perceive the meaning? I assert that the meaning of text is only present when a pattern is present in the message, and the reader of the text perceives a meaning to be present. This doesn't say anything about the author of the text.
I love reader meaning. Reader meaning is the best kind of meaning. But it's the wrong kind of meaning to be thinking about in this context.
We are trying to work out whether GPT - the speaker - has anything like "intelligence" or not. As part of that discussion, we are talking about whether the things it says have meaning. In this discussion, reader meaning is not the relevant question. The relevant question is speaker meaning.
Speaker meaning is also a thing. The relationship between speaker meaning and reader meaning is complex (hence the complexity of language and semantics). But we don't really need to worry about that here. The only question for us is: does GPT mean anything when it speaks? And I think you and I both agree that it doesn't. In the context of this discussion, it would be useful to maintain great clarity on that issue.
Well, I answered that, too. I assert that meaning for language can only exist when language it tied to external observables. Since GPT doesn't have the ability to do external observation, it can't have meaning, in that sense.
Human language refers to things in the world, so when you go into enough detail, emulating human language requires you to be able to model things in the real world as well.
Some day in the future, someone will say "Yeah, GPT-N seems smart, but it can't answer basic questions about quantum mechanics," and then we'll add another gazillion parameters to the model, and then GPT-N+1 will invent the Theory of Everything in order to generate text that can fool a quantum physicist into thinking it's a human.
You’re assuming the thing to be proven. In terms of world modelling GPT has yet to go from 0 to 1. If we were at 1 today, I’d find the alarmism more plausible. But today GPT cannot fool a human into thinking it can reason about simple matters.
GPT has *some* ability to model the world. Scott's last post was all about using it to answer simple hypotheticals about the world, like "what would happen if you closed your eyes and walked backwards down the stairs?"
It's always vital to distinguish the surface from the underlying.
"GPT has *some* ability to model the world." - no, GPT has some ability to say things as if it were modelling the world. So did ELIZA, and any other chatbot. But because we know how it works, we're pretty sure that there's no world-modelling going on, by which I mean there are no patterns inside the software that correspond to real things in the world.
To put that concretely, imagine a situation where you're holding two apples, and a one apple, and no apples. In one of those situations, you picture two apples; in the second, you picture one apple; in the third situation, you don't picture any apples. The mental object "apple" is not in the third situation. For GPT, by contrast, all internal processing of those three respective situations always contain the digital pattern (it's equivalent of a mental object) associated with the word "apple". The word "apple" is present in all three.
That's the kind of thing that I mean when I say that GPT is not modelling the world.
A text-to-image AI, by contrast, does create a model in a kind of a world (a 2D pixellated world), connected to but separate from the language of its prompts.
For #5, are you sure you're not on the same side (but possibly with differing levels of certainty)? I interpret Scott's 40% confidence in #5 as saying he believes it's 60% likely that a further paradigm shift will be required. (edit: though the framing of "here are some things I do believe" confuses me, it still does seem like he's coming down in favor of a paradigm shift being needed)
we aren’t hugely far apart but certainly i would happily take his odds; i promise to explain more at length (and also clear up some places where he got my view wrong); my main focus will be on what i think the shifts needs to be, rather than the #s per se
I wonder. You may well be right, but two of those jokes pivot on very concrete wordplay and the third on a common joke “form” (-X walks into a bar. e.g.)
For instance, here is a Polish joke from the early 1980s.
Two men are in a bar talking. One says what’s better, the Russian space station or the American space station?
The other man says, oh, the Russian space station.
Why? Asks his friend.
The other man shrugs his shoulders and says, well… You know… Because.
Mind you, most people who have never lived in an Iron Curtain country don’t get the joke either.
The joke is that they live in a constant state of surveillance and can't speak out in favour of America for fear a friend would report them to their local state agent?
My only experience of life behind the Iron Curtain is text, so if I'm right I'd update my odds a language model could understand it too
The formulation used (about paradigm shifts being incorporated in those models) is subtly but substantially different than the notion of the paradigm shifts being required.
If at some point of 2024 we discover a paradigm shift that doesn't change what is possible but simply allows getting the same results with 50% less computing power if you structure everything a bit differently, then every serious future model will incorporate that paradigm shift, but in this example it's not really *required* for those models, people would just have to pay twice as much for them otherwise - which nobody will in practice if the discovery is there.
One reason Marcus believes that symbolic models are necessary, in part, but are not the whole story, is because he was trained in them. He is thus biased. I share his bias. I was trained in computational semantics in the mid-1970s, the salad days of symbolic computing. I went so far as to sketch out and publish a model of the underlying semantic structure of a Shakespeare sonnet. https://www.academia.edu/235111/Cognitive_Networks_and_Literary_Semantics
Nonetheless I have had no trouble seeing that GPT-3 represents a phase change. Artificial neural networks are here to stay. Moreover, I agree with Geoffrey Hinton when he says the brain is large neural vectors all the way down. Which means that the brain implements symbolic structures with neural vectors. The young child who is beginning to learn language has no symbols in their head. Language symbols exist in the external world in the form of utterances by others (directed at the child) and, increasingly, by the child themself. Lev Vygotsky (who wrote a book on language acquisition, Thought and Language, in which he argues that, over time the child internalizes language. First the child talks to themself and then actual talk becomes unnecessary. The child is now capable of inner speech. Here's a sketch of that, https://new-savanna.blogspot.com/2014/10/vygotsky-tutorial-for-connected-courses.html
Once that internalization has taken place we have symbolic thinking with neural vectors. I'm including a slightly reworked version of that story in a longish paper I'm working on in which I spell out these ideas in some detail in some detail.
As far as I know most of the researchers working in artificial neural networks have had little or no training in classical linguistics or cognitive psychology. What they know is that the old symbolic systems didn't work that well while these newer systems seem to be pretty good. That gives them a different and very strong bias. But they don't know much of anything about how humans actually solve these various problems. Their knowledge is dominated by what they know about creating learning architectures that allow computers to do cool things that convincingly mimic human behavior – and in some domains (e.g. chess, Go), exceed human capacity – but they don't really know what the resulting models are doing when run in inference mode (though they're beginning to figure some of that out).
What we have is bits and pieces of various kinds of knowledge and a lot of things we don't know. When it comes to predicting the future, no matter what your current knowledge and biases, the future is dominated by THINGS WE DO NOT KNOW. Arguments about what will or will not work in the future are just attempts to shore up one's current biases. Beyond a certain point, that job is a waste of time. Why? BECAUSE WE DON'T KNOW.
"Luria: All bears are white where there is always snow. In Novaya Zemlya there is always snow. What color are the bears there?
Peasant: I have seen only black bears and I do not talk of what I have not seen.
Luria: What what do my words imply?
Peasant: If a person has not been there he can not say anything on the basis of words. If a man was 60 or 80 and had seen a white bear there and told me about it, he could be believed."
The last sentence here makes it pretty clear to me that they understand the answer the experimenter is looking for is "white", so I don't think this is a failure of logical reasoning. Rather they're trying to make the (completely valid) observation that blind induction is not always reliable. Jut because all the bears you've seen in snow have been white doesn't mean that's true everywhere else there's snow.
I think if anything their failure is to understand the concept of an absolute generalization. When the experimenter says "All bears are white where there is always snow", the peasant takes this as a claim about the real world and correctly infers that the experimenter can't know that's actually true.
The camel example can be explained the same way.
The other two seem more like a real failure of basic reasoning, though the last one could also stem from a simple misunderstanding of what the word "animal" means. (Using "animal" to mean "terrestrial vertebrates" or similar is not uncommon.)
Yeah, this is one of those examples that Scott hauls out every once in a while that I wish he wouldn’t. The responses garnered are by turns very canny (bears and camels) and obvious category misunderstandings, including straightforward confusion from the respondent about what the questioner’s definition of terms are. This is not evidence of stupidity or poor reasoning. Rather the reverse.
There's been quite a bit of research on folk taxonomy. One of the things you find is that preliterate peoples (like those Uzbeck peasants) lack words corresponding to plant and animal. They certainly know that chickens are more like cows than they are like pine trees, but that knowledge just hasn't been distilled into single words. That can lead to confusion and the need for work-arounds.
FWIW, in many languages the common meaning of the word "animal" is something similar to "non-human terrestrial creature that walks on legs". Fish and birds are not "animals" in this sense. Yes, they are animals in the biological sense, but Uzbek peasants probably don't know that.
Even for an Uzbek biologist who was an expert in taxonomy and genetics, it would be perfectly valid to say "fish and birds are members of the Animalia kingdom, but they are not hayvon". (Hayvon is Uzbek for animal, according to Google Translate.) Taxonomists do not automatically get the final say in deciding what an ordinary word in a given language means; if Uzbek peasants say that it means "non-human creature that walks on legs", they're not wrong just because that category does not map to a specific taxonomic group.
Which is to say it is not semantically equivalent to "animal." If Google translate says that "Hayvon" is Uzbek for animal, but the extension of "Hayvon" is not the same as the extension of "animal", then it's a bad translation and there may not be a good translation. By the way, what investigation does show is that, the taxonomic categories that preliterate cultures do have, also map onto biological taxonomy rather well. They just don't have some categories.
English speakers don’t usually use “animal” to refer to birds and fish either. They do when prompted to be precise in language, but if you’re out in the forest with little birds flitting around you might say you haven’t seen any animals if you haven’t seen a deer or rabbit or raccoon or something.
Was Laria speaking Russian or Uzbek when he was questioning the peasants, or were translators involved? I imagine there's some room for misunderstanding when you're speaking Russian to someone speaking Russian as a second language, and even more if it's Russian -> translating Russian to Uzbek -> Uzbek and then the answer goes the other way.
According to this English to Uzbek site:
an animal ::hayvon
wild animal ::yirtqich hayvon
farm animal ::xo'jalik hayvon
domestic animal ::ichki hayvon
So it could make all the difference if Laria thought he was asking "is a fish or a crow an animal (hayvon)" and the peasant thought he was asking "is a fish or a crow a domestic animal (ichki havyon)"?
English speakers don’t usually count birds and fish as animals either, unless they’re being particularly careful, any more than they count humans as animals. “Animal” is just a Latinate term with a precise extension that is still based on the same prototype as the Germanic “beast”.
So if I say "Noah had two of every animal on the ark" English speakers would understand me to be saying four-legged furry creatures that walk on land? Here's the FIRST definition of animal in the dictionary on my Mac: "a living organism that feeds on organic matter, typically having specialized sense organs and nervous system and able to respond rapidly to stimuli." Here's the THIRD: "a mammal, as opposed to a bird, reptile, fish, or insect." Word meaning is tricky.
I was wondering if the Uzbek word that corresponded to "animal" meant something more like "domesticated animal". In that case, then of course crows and fish aren't "animals".
It reminded me of a anecdote I read online years back about a teacher telling a story of when she started off in a primary school in a rural part of England; she drew a (conventional) sheep and asked the kids in her class "what is this?"
No answer, which surprised her. These were kids who, if they didn't come from farms themselves, lived in the country and so they certainly knew what a sheep was. Finally, after a lot of prodding and hinting, one of the kids raised his hand and said dubiously "Well, the *face* looks like a Swaledale..."
That was the mismatch of expectations there; she drew the usual "fuzzy white sheep" and expected them to say "It's a sheep", they were from a background of "Well of course it's a sheep, she must mean 'what breed is it?'"
Yeah, this felt a lot more like an expression of a "don't trust the experts" sort of sentiment more than a lack of intelligence.
---
Another way of looking at it may be that the whole "game" of "let's suppose incorrect counterfactuals and draw conclusions from them" just is foreign to them, sort of a cultural disconnect.
It's the sort of exercise that's drilled in in formal education and feels "obvious" to anyone from that background - it's not they lack the innate cognitive abilities to do it, but it's a "logic game" that's more culturally-ingrained than it may seem at first glance.
James Flynn found that the effect named after him was highest for these kinds of highly abstract questions, suggesting that they're one of the most culture-bound aspects of human intelligence.
The refusal to generalize ironically reminded me of this old joke about mathematicians (here copied from Wikipedia):
An astronomer, a physicist and a mathematician are on a train in Scotland. The astronomer looks out of the window, sees a black sheep standing in a field, and remarks, "How odd. All the sheep in Scotland are black!" "No, no, no!" says the physicist. "Only some Scottish sheep are black." The mathematician rolls his eyes at his companions' muddled thinking and says, "In Scotland, there is at least one sheep, at least one side of which appears to be black from here some of the time."
Who's truly failing at basic reasoning here, the peasant or the mathematician?!
> The other two seem more like a real failure of basic reasoning
I mean, either that, or the language barrier.
That said, if I were a peasant living in a post-Soviet country, and some official-looking guy came to me asking seemingly irrelevant questions, I would be *very* careful about how I answered them. I too would avoid committing to anything unless I absolutely had to. You never know what the plainclothes man will write down in his little book about you.
If the peasant asked "Are these questions part of the current purge?" would they have shown up in the recorded data? And ... would they ever have been heard from again?
"Peasant: If a person has not been there he can not say anything on the basis of words. If a man was 60 or 80 and had seen a white bear there and told me about it, he could be believed."
That's a common epistemological belief among preliterate people. Dan Everett, the linguist, has written extensively about his experience among the Piraha, of the Amazon Basin. He originally went there as a missionary to translate the Bible into Piraha and to convert them to Christianity. He found that whenever he told them about this Jesus fellow they'd ask him, "Have you met him and talked with him?" Everett would reply, truthfully, that no he had not. "Well then why should be believe what you tell us about this person you've never met? Why should we believe that he exists?" After several years of this, Everett decided that they were right, that Jesus had nothing to offer them, or him either. And so he became an athiest, but continued on as a linguist.
On these confidence levels you give, Scott. To some extent, sure. I have various levels of confidence in things I write and may even give some explicit indication of those levels. But you're giving percentages whose values take advantage of the whole range between 0 and 100. What's the difference between 65% for number 2 and 66% for number 4? Is that real? That strikes me as overkill.
You seem to have a three point scale: level 1 is, say, 0-45, level 2 is 46-67, level 3 is 68-100. Any precision you indicate beyond 1, 2, and 3 strikes me as being epistemic theater. It's tap dancing and hand waving.
Semi-counterpoint: I agree with you that 65% vs 66% is overkill (I think he just thought "two-thirds" and inconsistently translated them into rounded percentages). But I disagree that his scale is three-pointed; it's an annual SSC / ACX tradition for him to do ten-point scale predictions and score them once the year is done, eg https://slatestarcodex.com/2020/04/08/2019-predictions-calibration-results/ (see the graph at the bottom where he calibrates his confidence levels vs how often they actually turned out right).
If it's a ten point scale, then stick to it. I note that I'm also thinking of something that Holden Karnofsky wrote where he gives confidence values of 1% and 10% and not a word about where those numbers came from. I think he just means small and very small. The numbers don't mean any more than that.
I think they mean “small” and “very small” but likely also something like “7-12%” and “0.1-2%”. (Or maybe slightly different ranges. But the point is that I suspect that 10% and 1% are the right order of magnitude for what they mean and not just rhetorical flourishes.)
Without venturing to guess about what Scott was/ wasn't doing here, it's worth noting that in Tetlock's experiments, super-forecasters routinely make high-resolution probabilistic judgments, and that rounding their scores hurts their performance.
thanks, but what's the primary source being quoted in that post? Are all those quotes from the same document? Ah, I just noticed the title of the post.
In any event, I suspect that precision is wasted in forecasts about what an as yet non-existent AI device can or cannot do by a certain date.
As B_Epstein said, except that the 66% one was my mistake - in some sense 65% feels "rounder" than 66%, in another sense 66% feels "rounder", and I guess I wasn't operating off of a consistent world model and tried to use the "rounder" number each time and my brain made opposite decisions as to which one that was. Sorry!
Our brain is hard-coded for language acquisition. Which is, fundamentally, assigning an arbitrary label to a set of sensory experiences (eg, this sweet crisp round thing is called an "apple" or "mansano" or hundreds of other arbitrary sets of specific compressions of air (sound)).
That seems like a fundamental part of being able to do other abstractions?
It seems that our brain can't be very much hard-coded for language acquisition compared to e.g. chimps, almost all of the "hardware structure" is the same with barely enough "evolutionary mutation space" for some small adaptations - but yes, it seems plausible that those small adaptations which were evolutionary useful for social/language purposes would be important for being able to do other abstractions much better than e.g. chimps do.
> It seems that our brain can't be very much hard-coded for language acquisition compared to e.g. chimps
Huh? Chimpanzees are universally unable to acquire language; humans occasionally fail to do so, but that failure constitutes an extremely severe mental defect and occurs at rates much too small for most people to ever notice.
Yes, but the fact that we can acquire language and chimps don't (though their own communication is quite complex and there are some semi-controversial experiments with chimps learning sign language) does not imply that "Our brain is hard-coded for language acquisition.", quite the opposite.
First, it's plausible that this difference in ability is purely caused by a quantitative, not qualitative difference in brains; and second, given the relatively small evolutionary gap it's not plausible that we have a lot of highly specific "hard-coded" brain machinery compared to chimps; there is clearly space for some *small* adaptive mutations but any language deficiency that chimps have IMHO is some evidence that language acquisition simply requires more generic "brainpower", not some very specific wiring of the brain, since we share the same brain wiring principles as chimps but have more neurons available for audio processing.
What is the important difference between what you say and “hard-coded for language”? “Hard-coded” doesn’t have to mean “grammatical rules are innate” or “categories of noun and verb are innate” (I don’t think even Chomsky endorses those claims) but could just mean that there is some important hardware structure that easily canalizes into learning of language.
I was feeling very skeptical of that whole comment, but I will say at the end, "Recursive thinking and recursive knowledge seems genuinely hard for people of even average intelligence." -- well, based on having taught induction & recursion to students at some top colleges/universities (and having to re-teach it in higher level classes), the most general version can be quite difficult even for fairly smart people the first (or second) time they learn it.
But the claim does make me think a lot more about why the movie Inception had the particular cultural resonance it did. Maybe this particular nested structure really is a lot harder for people than I realize, and this movie was one of the first things that presented it accessibly, so it comes to stand in for any recursive structure anywhere.
It could write a response that ends with a question mark. But the way it operates, it has no memory of anything that happens after training is done, so each prompt you give it is starting over. It can’t interpret your text as a response to its previous question unless you include the whole conversation so far in your prompt. (But then it presumably could.)
It feels to me that the current path will not end up with "agentic" AIs. What if we create a program that completely passes the imitation game, BIG-bench, etc.; a program that has sufficient complexity that it appears to be conscious (and tells us it is, and is as convincing as a human when a human tells us it is)... but that program is just an inert input -> output generator like GPT-N or DALL-E?
This seems weird and alien! The intelligences we interact with, and read about in sci-fi, exist independently of us. They have goals that are not "patiently wait for someone to type in a text box, then type a response". They run persistently; they accumulate memory, and experiences, and change over time. They have spontaneous outputs based on the world around them. I don't know how we as a society should interact with something that tells us that it's conscious, and wants to live... but only if we ask it questions framed in the right way.
What would it take to get from the current marvels we have, to something more agentic/human-like/sci-fi AI-like? The missing ingredients I can see are long-term memory and evolution; more sensory inputs so they can passively pick up information from the real-world environment instead of getting strings fed to them; and some kind of motivation to initiate actions instead of sitting in a dark room being unsurprised by the lack of input. Are these things anyone is working on integrating with the intelligences displayed by GPT-N? Are they hard, or easy, or...?
I realize from the perspective of AI safety this is playing with fire, so, feel free to answer in the vein of either "yes, someone is making progress on that sort of thing" or in the vein of "yes, THOSE FOOLS are doing that sort of thing and will kill us all". I just want to know...
Nobody's doing online learning yet, at least as far as I can tell, but it wouldn't surprise me if that was the sort of research that was not observable because worlds that go down it fall quickly.
A key aspect is that agentic AIs is simply an orthogonal direction and for many practical problems which we want to solve, agentic AIs aren't necessary so we definitely want to (also) explicitly pursue non-agentic AIs simply to avoid any unnecessary cost and complexity.
Not all SF uses such tropes of AI. Charles Stross in Rule 34 posits a highly believable "AI" which is simply a giant optimization engine cranked up to 11, yet is largely indistinguishable from a conscious superintelligence because it has access to lots of data and fast computation. This scenario seems much more likely to me than GPT-6 suddenly attaining enlightenment through scale.
Wait, what? Wouldn’t the phrase “a giant optimization engine cranked up to 11” perfectly describe a GPT-6 that was more-or-less GPT-3 plus a lot of scale?
Yes, precisely. My point is that GPT-6 doesn't need to attain consciousness to have superhuman capabilities in some areas. I'm responding to the statement that SF usually relies on a conscious AI; not all SF does so.
Ignore the previous thing, I figured out how to un-collapse everything and now I remember the source of my confusion. You said:
> this scenario seems much more likely to me than GPT-6 [to become super-intelligent] through scale
But AFAICT it can’t be more likely because it’s basically the same scenario. It’s like saying that “some book I read had a billionaire building a reusable rocket company to try to colonize to Mars, which seems more likely to me than Elon Musk’s plan”.
Read my comments again. I am saying a mysterious awakening into consciousness by a bigger GPT seems less likely than it potentially outperforming humans yet without being intelligent in any human sense.
Oh, I thought “attaining enlightenment” was just another more poetic way of saying “largely indistinguishable from a conscious superintelligence”. If you meant the two to contrast then it makes more sense. Thanks for clarifying!
This is so interesting thanks! Love the point about the hands and dreams. I think a point you're missing though it amount of training data. Sure humans can make dumb mistakes but they haven't read a fair amount of everything ever written. GPT-3 needs way more training data than GPT-2 right? That seems to be where your analogy with the human brain breaks down to me. Sure we could scale GPT to have an equivalent computational power of a human brain but how are we going to scale the amount of training data up by 1000x? while keeping it sufficiently diverse that it isn't almost identical to something already in the training data? Also doesn't the fact that it needs these vast data sets (orders of magnitude larger than what a child is exposed to) indicate that they are learning in a fundamentally different and less sophisticated and more brute force statistical inference type way?
Another example, alphago zero has to play millions of games against itself before it is better than any human. If a human got better at that rate they would be idiotically dumb. So we are still a long way from superhuman at Go it seems to me. We are just taking advantage of faster computation. Rather than using a more abstract way of learning.
I think if a human learned that slowly from their first few thousand games they would be dumb. But for a human to learn at all from their hundred thousandth game would be unusual.
Don't you think though that most of human "Go" training is internal? We run pieces of games by ourselves. Additionally, the vast majority of the last XX% of games used to train an AI must be effectively useless. If the AI lacked time and computational power, it might just filter out the games it felt were likely to be unrewarding. At that point, the internal training of a human and the training of the AI seem far more similar.
yeah good point it could be more similar than the raw number suggests. But the vast difference just makes me think something fundamentally different and higher level and less statistical inferency is going on for humans, but I have no idea how to prove this.
Are you sure about that? That “Zero” in the name is important!
An exceptional human might be able to learn Go by age six, but not if you just lock it up right after birth in a blank room with a Go board and only feed it when it makes a good move.
Even if you put humans through a “pre-training” phase where you take care of them, teach them to walk, talk, and socialize for six or so years, and *then* you start teaching them Go, they still won’t get very good at it. Particularly if by “teach them Go” you means just “reward them for good games”, instead of teaching them the accumulated knowledge of strategy and whatever.
Edit: Also, why would you count number of games played, instead of time spent, or energy consumed? By those standards, humans’ Go learning skills are just barely distinguishable from monkeys.
Yes, the question of scaling rate seems quite important here (as elsewhere in AI; but also hard to reason about satisfactorily). No one in the present discussion is arguing there's a specific X that GPT-N will never do, but if each GPT-(N+1) can be flummoxed by adding 1 more complication/distraction to the query--which takes an extra hour of playing around compared to a few years for GPT-(N+1) to be developed and trained--then I can see Marcus's perspective that there is something fundamentally underwhelming here.
That assumes the scaling in conceptual competency is somehow "linear", and thus significantly limited by available compute power, time, and training data. But if it's actually "exponential", then who knows? The way a 5 year-old child develops into, say, Scott Alexander (with an approximately fully developed world model) feels "exponential"ish.
Yeah it is very hard to reason about. The other thing is if it is just about scaling current models then all humans should be pretty similarly intelligent right? But it is clearly not as the examples given show. It seems unlikely that GPT type models trained with similar amounts of data and compute power would show nearly as much variation as humans do in intelligence, which also seems to imply to me that something very different is going on.
I laughed out loud at "megawatts of energy (perhaps more)". It's like: "How far is it from Athens to Madrid?" "Oh, hundreds of miles per hour -- perhaps more!"
Isn't an AI that lacks cognitive models still worth thinking about?
Transistor-based computers transformed everything in the world (as did the internal combustion engine, the printing press, and the wheel before them). If we're concerned about AI causing mass technological job displacement, or unleashing Sorceror's Apprentice style disasters, then I don't see why Marcus's objection is that important.
The most interesting prediction would be "When will a GPT derivative be able to make first incremental improvement in its own structure, including self-training?" Because that will be the day it poofs without ever becoming "Marcussian."
_Mostly_ agreed, but it gets tricky. Self-improvement is certainly important, since it creates a positive feedback loop. But consider an optimizing compiler compiling itself. Yes, it makes the next binary faster, but that is where it ends. Some positive feedback loops run away, but some saturate.
Yeah, that's a very good point. My assumption of ever recursive self-improvement guaranteed to be unbounded is not necessarily correct. I would like to see some research in that direction: what are the conditions for the recursive self-improvement taking off vs fizzling out.
Imagine, for a moment, a student taking test, after test, after test, and passing them; however, he student's responses to questions come verbatim from the book.
In the next text, the professor changes the wording for the questions: "In your own words..."
Suppose the student responds with a verbatim quote from the book. Okay, suppose the student responds with a quote that combines the material from two books in a way which is logically sound. Suppose it is three books. And so on and so forth.
Here's the thing: At some point, the student's answer cannot be meaningfully said to be taken from any of the books; it is, by the standards we measure human responses by, the student's own words.
And yet I find my attention drawn to a curious detail: The material is combined in a way that is logically sound, and this is evidence, in a human, that they understand the material, because the alternative - that they understand the logical relationships between the words while not understanding anything else - looks unthinkable.
It must be thinking, because it is creating the same kind of output as thoughts.
Suppose, for a moment, that AI is possible; that the brain can be modeled as some set of algorithms and data structures. Can the output be predicted?
Yes; we can predict what other people will do. We work really, really hard at being predictable. What I am writing, right now, is designed to be predictable on a number of dimensions that you aren't even aware of. uamtypfinfndhatthisduskanexampleofnutwkrnghrd.m I took away a single dimension of predictability, and "I am typing and this is an example of not working hard" became that; it's still extremely predictable. The fact that the word "predictable" occurs in this sentence, for example, greatly increased the odds of something like that being in there; GPT-3 has access to text with typographical errors, and has access to information that would allow it to infer a relationship between letters that we would understand to be proximity on the keyboard.
"So what?"
So there's a reason the teacher added "In your own words" into the questions.
People are perfectly capable of understanding the logical relationships between words without understanding "anything else". We have a real-world example from Richard Feynman's experience with science education in Brazil. https://v.cx/2010/04/feynman-brazil-education
All the students were fantastic at memorizing and regurgitating whatever was in the textbooks, but were completely unsuccessful at applying any of the info to hypotheticals on what would happen in an experiment.
I have no idea if the students would have been able to put things "in their own words", but if the classes were structured that way, I suspect they would have been able to do so even while exhibiting the same flaws.
Fascinating information about general capabilities varying with IQ, but the stuff about prison populations seems quite off. Perhaps some of these people have trouble with *contradictory* hypotheticals, asking about things that obviously did not happen. Instead, I'll bet serious money that, if you asked, "What would you do if you found out your wife cheated on you?", you'd get a clear answer, not "are you telling me my wife has cheated on me?" Or, more to the point, "if tomorrow, they didn't give you breakfast, how would you react?" You'd have to be exceptionally impaired not to be able to imagine that, and answer accordingly.
Why is no one talking about qualia and incorporating non-verbal experience? Believing that everything we could say about sunsets is somehow in the convex hull of what has already been WRITTEN about sunsets, and can be inferred by machines who had never actually seen one, seems entirely off-base (aka, I'm with Gary on this one).
The greatest poetry is often built on metaphors that are outside our prior experience as readers, but somehow seem entirely "right". Or, as Anne Sexton put it, "making trees from used furniture".
Your prediction 5) seems overconfident to me. Has anyone tried hooking up a text-prediction algorithm to e.g. a camera? Does that idea make any sense? Does anyone have any idea how to join a text-prediction algorithm with an image-recognition one, and say "these strings correspond to these images"? Wouldn't such a combination still be limited to a rectangle of pixels and a bunch of strings, with no way of distinguish a sequence of events actually happening from a picture book?
Perhaps more importantly, how good would it be at reasoning in new situations, unlike those that appear in its training data? Maybe those are the kinds of questions we should be focusing on.
We have evidence that the same techniques work across modalities; for example I seem to recall a paper that essentially trained the same language models not on text but on raw audio data, and they worked, generating reasonable text; the only drawback was that this is computationally much less efficient than processing text so it's not practical and only good as a proof of concept.
Not only have people tried combining these algorithms, combining things this way is fundamental to current AI research. “Transformers,” the current state-of-the-art in deep learning, can be applied to just about any modality and often many at once.
There are obvious differences in how it was trained and a lot of other technical details, but at a fundamental level the same architecture underlies GPT, DALL-E, AlphaFold, and countless other AI systems.
Anyway, though, I don’t believe Scott is saying that transformers in particular will be used, but that the techniques that eventually achieve AGI will be, fundamentally, deep learning systems. As an AI scientist, I agree.
>Luria: All bears are white where there is always snow. In Novaya Zemlya there is always snow. What color are the bears there?
Peasant: I have seen only black bears and I do not talk of what I have not seen.
Luria: What what do my words imply?
Peasant: If a person has not been there he can not say anything on the basis of words. If a man was 60 or 80 and had seen a white bear there and told me about it, he could be believed.
I think this peasant is really f-ing smart. He is refusing to be drawn into a metaphor, a fictional time and space being proposed to him as a stand-in for someplace real. A person stands before him and talks of what others have said. He expresses his lack of trust, and therefore his unwillingness to draw any conclusion from it.
He would be perfectly willing to take the word of a serious person (in this case, very specifically, a 60-80 year old man) who had actually seen a white bear.
The natural world is so full of exceptions that trying that type of categorical syllogism? using nature examples would always be disagreeable to someone whose life is based in nature. Even people who work with physical machines a lot have a version of this; some believe that swearing at it makes the repair go faster, there are attributions of mood to it, etc.
Fully grounded in place, time, circumstance and physical history. In a sense, almost postmodern in its insistence on the continued relevance of context.
I wonder what would have happened if the researcher used “game pieces” and asked questions about those.
I would've said that what is going on here is that Luria is thinking abstractly, as smart people usually do, but the peasant is thinking concretely, as dumb people normally do. So while Luria is thinking "the key part here is the abstract relationship between the tokens in my syllogism -- it doesn't fucking MATTER whether it's bears being white or Socrates being mortal, and "Novaya Zemlya" is just a random place name I throew in....." the peasant is thinking he is actually talking about real bears, and real snow, and a real place called Novaya Zemlya, and the first conclusion the peasant reaches -- which is logically sound -- is that he knows he doesn't know anything about Novaya Zemlya, so how the fuck should he know what the bears are like there?
The thing is, you can be a pretty sophisticated concrete reasoner and appear stupid to an abstract reasoner, because the abstract reasoner just doesn't *see* the various parts of the input stream on which the concrete reasoner gets stuck. They're all just tokens to the abstract reasoner, but they're real things with real properties to the concrete reasoner.
It's still concrete versus abstract, I think. I can say "I believe X because Joe, whom I know, said X is true" and that is much more concrete than saying "I believe X because everyone/this type of person/some popular book says it's true."
In the first case, there is a specific object (a person you know) and you can query your internal knowledge of that concrete object to establish your level of trust. In the second case, you don't have any specific object -- you can't query "everyone" -- so you have to start reasoning abstractly. "What would it mean if many or most people believed X? Let me generalize from what I know of individual people to some model of people in general and see what properties that abstract model has..."
If you're a good abstract reasoner this shift happens so fast you don't even realize you've moved from a concrete to abstract mode of reasoning.
I very much agree with you on several points. I would say Concrete AND Abstract rather than versus.
This peasant being interviewed, for instance says he (I will assume “he”) will have no opinion on what he has not seen; very concrete. Then he let’s on that if a person of a certain type (male, probably older than he) vouched for it he would be willing to believe it. > Let me generalize from what I know of individual people to some model of people in general< right? He’s proposing a limited model but I think it conforms to your proposal
So….
This is the first small step into abstraction. IF…a person (who, as described, I am guessing fits a certain archetype of authoritative and thus credible) says it’s true I will give it weight.
(> query your internal knowledge of that concrete object to establish your level of trust<)
So it’s abstract but still pretty firmly tethered to the concrete. As the abstractions become more elaborate that tether is stretched.
> this shift happens so fast you don't even realize you've moved from a concrete to abstract mode of reasoning.< I consider this a bug, not a feature. I think it’s a very important thing to notice, otherwise you really are at the mercy of every jackass with a story to tell.
One has to keep the balloon tied to the rail lest it float way.
(I wonder how GPT 3 would do with that metaphor. I’m itching to outsmart it.)
I appreciate your comment here and I think it opened a conceptual door for me - I’ve wondered for a long time why some people use statements that are abstractly incorrect, or inaccurate, to cheer people up. Can’t we cheer people up with statements that are also abstractly correct? (For example, things like “don’t worry, that test score doesn’t matter” or “not brushing your teeth once in a while doesn’t hurt” or things of that nature. Those aren’t good examples but it’s basically that pattern.) Extending forgiveness or compassion by avowing that there are no follow-on effects of a situation or action, when actually there are, perhaps small, but real effects. I think pulling this thread leads to people becoming uncomfortable pointing out any negative impacts of any action of someone they wish to show sympathy towards.
It’s like there’s a rule there, “if it’s too complete, too nuanced, too abstract or includes any recognition of a downside it doesn’t count as supportive.”
If the people who do this are concrete reasoners, they’re not getting the little zaps of discomfort from making untrue statements. They’re just following the concrete goal of “keep this person happy” and operate without caring about the abstract or structural impacts of their statement (now he has to brush twice as hard next time to get the plaque off, for example).
> If the people who do this are concrete reasoners, they’re not getting the little zaps of discomfort from making untrue statements.
Interesting. My position would be that concrete reasoners are much less likely to make untrue statements. Also much less likely to either minimize or exaggerate the effects of a particular action.
What is severely lacking in this discussion is more understanding of what model based reasoning would entail, and how human model based reasoning supposedly differs from what GPT -3 is doing. I am very confused about this, and I don't think anyone in this broader debate has fleshed the idea out in depth.
I still think that Marcus points at something that is currently true. No matter how you phrase your prompt, a human will always know that water is wet, while that's not necessarily the case with GPT-3 I think. That doesn't mean that we are strictly using logical thought for all our inferences (and yes, as the feminist book keeper example illustrates, we do have bugs), just that maybe at least some of our reasoning is based on causal models of the world.
I get your point, but it's interesting you specifically mention that, because there's some disagreement on the topic, e.g. https://www.quora.com/Is-water-wet-1
I don’t understand why, When asked a question like “when was the last time Egypt was in San Francisco?” the appropriate response wouldn’t be something along the line of “What are you talking about?”
These AIs may be designed to always make an answer rather than ever shutting down or punting the question back to the asker. I’ve wondered also though.
I tackled roughly the issue of world modelling several years ago on my old blog De Pony Sum, which has since been replaced by my new Substack blog, Philosophy Bear. Here was my article back at De Pony Sum on world modelling:
***Recent advances in Natural Language Processing—Some Woolly speculations***
I wrote this essay back in 2019- before GPT-3. Since then I think it has held up very well. I thought I'd re-share it to see what people think has changed since then, in relation to the topics covered in this essay, and see if time has uncovered any new flaws in my reasoning.
Natural Language Processing (NLP) per Wikipedia:
“Is a sub-field of linguistics, computer science, information engineering, and artificial intelligence concerned with the interactions between computers and human (natural) languages, in particular how to program computers to process and analyze large amounts of natural language data.”
The field has seen tremendous advances during the recent explosion of progress in machine learning techniques.
Here are some of its more impressive recent achievements:
A) The Winograd Schema is a test of common sense reasoning—easy for humans, but historically almost impossible for computers—which requires the test taker to indicate which noun an ambiguous pronoun stands for. The correct answer hinges on a single word, which is different between two separate versions of the question. For example:
The city councilmen refused the demonstrators a permit because they feared violence.
The city councilmen refused the demonstrators a permit because they advocated violence.
Who does the pronoun “They” refer to in each of the instances?
The Winograd schema test was originally intended to be a more rigorous replacement for the Turing test, because it seems to require deep knowledge of how things fit together in the world, and the ability to reason about that knowledge in a linguistic context. Recent advances in NLP have allowed computers to achieve near human scores:(https://gluebenchmark.com/leaderboard/).
B) The New York Regent’s science exam is a test requiring both scientific knowledge and reasoning skills, covering an extremely broad range of topics. Some of the questions include:
1.Which equipment will best separate a mixture of iron filings and black pepper? (1) magnet (2) filter paper (3) triplebeam balance (4) voltmeter
2. Which form of energy is produced when a rubber band vibrates? (1) chemical (2) light (3) electrical (4) sound
3. Because copper is a metal, it is (1) liquid at room temperature (2) nonreactive with other substances (3) a poor conductor of electricity (4) a good conductor of heat
4. Which process in an apple tree primarily results from cell division? (1) growth (2) photosynthesis (3) gas exchange (4) waste removal
It’s not just about answer selection either. Progress in text generation has been impressive. See, for example, some of the text samples created by Megatron: https://arxiv.org/pdf/1909.08053.pdf
2.
Much of this progress has been rapid. Big progress on the Winograd schema, for example, still looked like it might be decades away back in (from memory) much of 2018. The computer science is advancing very fast, but it’s not clear our concepts have kept up.
I found this relatively sudden progress in NLP surprising. In my head—and maybe this was naive—I had thought that, in order to attempt these sorts of tasks with any facility, it wouldn’t be sufficient to simply feed a computer lots of text. Instead, any “proper” attempt to understand language would have to integrate different modalities of experience and understanding, like visual and auditory, in order to build up a full picture of how things relate to each other in the world. Only on the basis of this extra-linguistic grounding could it deal flexibly with problems involving rich meanings—we might call this the multi-modality thesis. Whether the multi-modality thesis is true for some kinds of problems or not, it’s certainly true for far fewer problems than I, and many others, had suspected.
I think science-fictiony speculations generally backed me up on this (false) hunch. Most people imagined that this kind of high-level language “understanding” would be the capstone of AI research, the thing that comes after the program already has a sophisticated extra-linguistic model of the world. This sort of just seemed obvious—a great example of how assumptions you didn’t even know you were making can ruin attempts to predict the future.
In hindsight it makes a certain sense that reams and reams of text alone can be used to build the capabilities needed to answer questions like these. A lot of people remind us that these programs are really just statistical analyses of the co-occurence of words, however complex and glorified. However we should not forget that the statistical relationships between words in a language are isomorphic to the relations between things in the world—that isomorphism is why language works. This is to say the patterns in language use mirror the patterns of how things are(1). Models are transitive—if x models y, and y models z, then x models z. The upshot of these facts are that if you have a really good statistical model of how words relate to each other, that model is also implicitly a model of the world, and so we shouldn't surprised that such a model grants a kind of "understanding" about how the world works.
It might be instructive to think about what it would take to create a program which has a model of eighth grade science sufficient to understand and answer questions about hundreds of different things like “growth is driven by cell division”, and “What can magnets be used for” that wasn’t NLP led. It would be a nightmare of many different (probably handcrafted) models. Speaking somewhat loosely, language allows for intellectual capacities to be greatly compressed that's why it works. From this point of view, it shouldn’t be surprising that some of the first signs of really broad capacity—common sense reasoning, wide ranging problem solving etc., have been found in language based programs—words and their relationships are just a vastly more efficient way of representing knowledge than the alternatives.
So I find myself wondering if language is not the crown of general intelligence, but a potential shortcut to it.
3.
A couple of weeks ago I finished this essay, read through it, and decided it was not good enough to publish. The point about language being isomorphic to the world, and that therefore any sufficiently good model of language is a model of the world, is important, but it’s kind of abstract, and far from original.
Then today I read this report by Scott Alexander of having trained GPT-2 (a language program) to play chess. I realised this was the perfect example. GPT-2 has no (visual) understanding of things like the arrangement of a chess board. But if you feed it enough sequences of alphanumerically encoded games—1.Kt-f3, d5 and so on—it begins to understand patterns in these strings of characters which are isomorphic to chess itself. Thus, for all intents and purposes, it develops a model of the rules and strategy of chess in terms of the statistical relations between linguistic objects like "d5", "Kt" and so on. In this particular case, the relationship is quite strict and invariant- the "rules" of chess become the "grammar" of chess notation.
Exactly how strong this approach is—whether GPT-2 is capable of some limited analysis, or can only overfit openings—remains to be seen. We might have a better idea as it is optimized — for example, once it is fed board states instead of sequences of moves. Either way though, it illustrates the point about isomorphism.
Of course everyday language stands in a woollier relation to sheep, pine cones, desire and quarks than the formal language of chess moves stands in relation to chess moves, and the patterns are far more complex. Modality, uncertainty, vagueness and other complexities enter- not to mention people asserting false sentences all the time- but the isomorphism between world and language is there, even if inexact.
Postscript—The Chinese Room Argument
After similar arguments are made, someone usually mentions the Chinese room thought experiment. There are, I think, two useful things to say about it:
A) The thought experiment is an argument about understanding in itself, separate from capacity to handle tasks, a difficult thing to quantify or understand. It’s unclear that there is a practical upshot for what AI can actually do.
B) A lot of the power of the thought experiment hinges on the fact that the room solves questions using a lookup table, this stacks the deck. Perhaps we be more willing to say that the room as a whole understood language if it formed an (implicit) model of how things are, and of the current context, and used those models to answer questions? Even if this doesn’t deal with all the intuition that the room cannot understand Chinese, I think it takes a bite from it (Frank Jackson, I believe, has made this argument).
(1)—Strictly of course only the patterns in true sentences mirror, or are isomorphic to, the arrangement of the world, but most sentences people utter are at least approximately true.
"In hindsight it makes a certain sense that reams and reams of text alone can be used to build the capabilities needed to answer questions like these. A lot of people remind us that these programs are really just statistical analyses of the co-occurence of words, however complex and glorified. However we should not forget that the statistical relationships between words in a language are isomorphic to the relations between things in the world—that isomorphism is why language works. This is to say the patterns in language use mirror the patterns of how things are(1). Models are transitive—if x models y, and y models z, then x models z. The upshot of these facts are that if you have a really good statistical model of how words relate to each other, that model is also implicitly a model of the world, and so we shouldn't surprised that such a model grants a kind of "understanding" about how the world works."
Here’s my maybe flawed chain of logic that has me skeptical that AGI can come out of LLMs:
1. Conscious thought seems to be very expensive both from a calorie consumption viewpoint but also from say having people wandering around wondering why they are here as opposed to having more children.
2. Evolution seems to have selected for conscious thought anyway which means its benefits outweigh these costs.
3. Therefore it is probably the most efficient way to get agents that are as flexible and adaptable as humans are. More speculatively it might be the only way to achieve this: after all when we look at animals intelligence seems very correlated with how conscious we perceive that animal to be from viruses, to ants, to dogs, to chimps, to us.
4. So that leaves me with one of two weird conclusions either of which I find unlikely (say less than 40% odds). Either consciousness can occur purely by studying human language and does not need anything else (e.g. interacting with the physical world or the need to compete with other intelligent agents which probably drove the evolution of consciousness) or consciousness is not necessary for AGI.
Both of these statements seem deeply weird. So I would suspect maybe more is needed than putting more sentences in larger neural nets. Neural nets themselves may still be able to support AGI but probably we won’t get there just by feeding them more data. Instead my guess is we will need something else and this something is some sort of competition between agents for some sort of resources. This would cause the neural nets to have to build the ability to reason about other agents which are reasoning about them. In other words they would need to be aware of their own state e.g become conscious.
> we will need something else and this something is some sort of competition between agents for some sort of resources.
Oh god I hope not. If we can’t do better than that we should settle for the adding machine on steroids version of AGI. Although clearly that was the major force in our evolution, I see no reason to do it that way over again only bigger.
Hmm... I phrase my worry about the eventual consequences of AGI in terms of our effectively building a competing species (and look how well that worked out for our cousin hominids). I guess Tpot's proposal might be shortcut to that fate...
Regarding sudden capability gains with increasing scale, you might be interested in the BIG-bench whitepaper which was released today: https://arxiv.org/pdf/2206.04615.pdf
In particular, check out the section on breakthroughness
Neat -- thanks for sharing, Bucket! The comically long authors list made sense after seeing that the paper is introducing "the Beyond the Imitation Game benchmark (BIG-bench) [which] currently consists of 204 tasks, contributed by 442 authors across 132 institutions."
I've only had time to look at the section on breakthroughness that you called out but it is indeed really interesting stuff:
"Tasks that see strong breakthrough behavior include those that are composite in nature, meaning that they require a model to apply several distinct skills or perform multiple discrete steps in order to come up with the correct answer."
"Breakthrough behavior is consistent with the model suddenly gaining new skills in an abrupt way. Careful analysis of task behavior, however, suggests that the underlying change in model capabilities is generally more smooth. These smooth improvements can be exposed by decomposing tasks in ways that better capture partial progress."
I wonder if the Linearity and Breakthroughness metrics will get much uptake.
The example of how the models of various sizes deal with the "emoji_movie" test ("Q: What movie does this emoji describe? 👧🐟🐠🐡") was fascinating. The tiny models answered with classic-chat-bot style non-sequiturs, the intermediate-sized models tended to take it literally and answer "the emoji movie", and only the largest model tested (128 billion parameters) correctly answered "finding nemo".
(Hopefully my writing this comment didn't make it harder for developers of future model training corpora to filter out these test answers...)
FWIW, I was not able to guess the movie either. It might be a platform specific thing, but I tried two different computers with different emoji renderings, and the fish don't look anything like Finding Nemo in either case. (e.g. the right one is a pufferfish on both computers I tried, while Marlin and Nemo are clownfishes.) And if all you have to go on is "human head + three random fish", there's no way you could possibly narrow it down to just "Finding Nemo" with confidence.
The code points are those of GIRL, FISH, TROPICAL FISH, and BLOWFISH. I admit I wasn't completely sure either -- it has been a really long time since I saw Finding Nemo! -- but I figured that was probably it before seeing the answer due to having seen a similar GPT-3 task in the past involved Finding Nemo expressed as a couple of fish emojis.
People noticed very early on that GPT-3 could go in the other direction[1] (and this has even made it into OpenAI's API examples[2]).
Scott, I'm not sure if this argues for or against your point (maybe this would count as a the addition of some kind of neurosymbolic system?) but here's a very recent "memorizing transformers" paper representing a grasp at the holy grail of getting a GPT-3-like deep neural network to be able to use computer-like memories (like a human taking notes, or flipping back to look something up in an earlier page, or really just having a hippocampus since since GPT-3 lacks long-range attention completely): https://arxiv.org/pdf/2203.08913.pdf
It's a transformer model, like GPT-3, hooked up to a key/value store that it can use to perfectly recall previously-seen fragments of input (keyed by, essentially, which prior inputs resulted in "mental states" most similar to its current state). Some excerpts from the paper:
Page 7: "External memory provides a consistent improvement to the model as it is scaled up. Remarkably, we found that the smaller Memorizing Transformer with just 8k tokens in memory can match the perplexity of a larger vanilla Transformer which has 5X more trainable parameters."
Page 9: "When predicting the name of a mathematical object or a lemma, the model looked up the definition from earlier in the proof. Examples of this behavior are shown in Table 8. [...] We manually checked 10 examples where the model made a prediction of lemma names, and 8 out of 10 times model found the body of the lemma it needs to predict. In the other two cases, the model also looked up materials in the immediate vicinity. To the best of our knowledge, this is the first demonstration that attention is capable of looking up definitions and function bodies from a large corpus."
Human brains can answer questions by referring to their mental model of the world. GPT-3 can only answer questions by understanding which words are likely to appear in relationship to other words, based on the ten quadrillion word corpus that it has ingested.
The question is whether, by throwing enough nodes and enough training data at a GPT-like model, it could eventually develop something that you could call a model of the world. If it _did_ develop a model of the world then it would instantly become better at the task it's trained for, so you could argue that this is the point that the training procedure should eventually reach. On the other hand, a given training procedure won't necessarily reach the global maximum, it could easily get stuck in a local maximum.
Many people here will be familiar with SHRDLU, a 70s-era AI parlour trick that could understand natural language and answer questions within a limited domain. The domain was an imaginary tray of blocks; you could tell it "put the red triangular block on top of the blue rectangular block" and later ask it "what is the red triangular block resting on?" SHRDLU explicitly did have a model of its limited world.
So here's what I'm wondering: can you turn GPT into SHRDLU? If you fed enough SHRDLU question-and-response text into a transformer, would it eventually reach the point where it can flawlessly answer questions about block positions? If so it would be fair to say that you've managed to get GPT to the point where it's developed its own internal model of the (very simple) world to make answering questions easier. If not, and it still gets confused when you ask it about the position of a block you haven't mentioned for hours, then I think it supports Marcus against Scott.
I think you have asked the $50,000 question, and it's certainly the one that's at the top of my mind. Something that can generate entertaining word salad and replace celebrities in tabloid interviews will always be a niche application, what we want to know is does logical reasoning in however small a domain emerge, at some point, from the brute force neural net approach? As you put it, can SHRDLU emerge spontaneously from GPT?
So far as I can tell, with a limited amount of testing, it has not done so yet. Which seems like bad news. I mean, if you'd asked me ahead of time, I would have guessed if some kind of limited abstract logic were to emerge spontaneously from a neural net, it would happen...oh, somewhere in the first 1000-10,000 nodes. Give me a million and I would have said of course it will happen! if it can happen at all. I mean, if one were to *deliberately* program a million nodes into a giant if/then/else logic program, you could definitely create a very complex and sophisticated SHRDLU. So...why did it not happen spontaneously?
> So sure, point out that large language models suck at reasoning today. I just don’t see how you can be so sure that they’re still going to suck tomorrow.
I’m sure of this because the foundational skills that are prior to world modelling are not going to arise without the agent manipulating its environment and getting feedback about it. (I don’t consider negative gradients in language model training equivalent to ”feedback about the environment” because the space of sentences is extremely sparse compared to the physical world.)
I agree with your passage on world building, and I agree that Marcus’s conclusion is way too strong for the evidence. The bottleneck here isn’t ”statistical AI” or ”Locke.” It’s that sentence completion is a relic of intelligence rather than an essence. And it’s too narrow a relic to optimize for directly.
Scott, I think Marcus means something like a) GPT is trained on language, b) so it won't have a model of space. If you describe GPT-4 a complex scene of things being put somewhere in relation to other things ("X is 10cm above Y, which is touching Z..."), it won't be able to answer questions about relations between these things you didn't describe directly but are implied. E.g. what's the distance between X and Z (centers, also assume there's enough info, references to objects Q and W and E... to calculate it).
I think that's true of GPT-3 to a large extent, but it's not a fundamental limitation. One could synthesize a laaaarge dataset like this, procedurally. With enough examples, it'll have to learn.
Same way it could learn arithmetic. It's already pretty good at it, it definitively isn't just memoization.
The AI risk community has been a little too preoccupied with Ex Machina-style thought experiments about superintelligence, where CLEARLY the major AI risk is from humans putting AI in charge of things that it's not remotely competent to handle but where it can jump through enough hoops to fool a lazy person into thinking it'll be fine, and then it breaks, and then something terrible happens. This AI risk is already killing people with Teslas (both inside them and outside of them).
Machine translation is another, perhaps lower risk, area where people are so lazy and ignorant and desperate to avoid paying a human being to perform a service that they'll use it, and it's amazing how well it works up until the second that it inverts the meaning of an important sentence and/or accidentally uses scatalogical slang in an ad slogan.
If AI risk worries you, it seems to me the highest leverage use of your time is getting the word out that AI sucks and should not be in charge of anything ever, not banging the table for how amazingly competent AI will probably be someday.
Those things are bad as well, and may cause problems sooner, but the stakes are nowhere near as concerning. If machine translation talks about poop in an ad slogan, sucks for whoever paid for that, but they will pull it and reconsider. Even if it leads to a net loss of life in car crashes, that's a tragedy, but people can figure it out and stop buying the cars. If things go the way that folks like Yud predict, humanity will be rendered extinct. I think it's obvious which of these things, in that case, is the "major AI risk."
You may disagree on the plausibility of those arguments, many do. But I think you are very much talking past them here. Pointing out that these issues are nearer and more certain without disputing the logic of the x-risk arguments leaves you akin to someone who points out that their airplane's "fasten seatbelt" lights are broken while it plummets to earth from an engine failure
If people see continual utility in further developing AI, they will do so, and if an advanced enough AI does in fact present an existential threat to humanity, then eventually that threat will come into being.
It seems to me that the best course of action is to convince people now that continued advancement of AI is a dead end and a waste of time.
The idea that we can have our cake and eat it to, that we can have super amazing AI but also very smart AI risk people will figure out how we can do so "responsibly", and then ALSO inform and persuade the entire rest of the species to take their advice, is just hilariously naive.
I'm not concerned at all about existential risk from superintelligent AI, but existential risk from really shit AI being put in charge of important tasks is unquestionably real. I should think that people who are very concerned about the former would also be very concerned about the latter, and the answer to both is: do less AI stuff.
Well if you can convince everyone to stop all AI research, I'll commend you. I think that's more naive than seeking alignment though. The incentives are there, AI research does actually have business value for a lot of companies. Look at Google using it to give tools that automatically detect all photos with some object in them, look at any phone with face recognition, look at spam filters and credit card fraud risk engines. It defines the state of the art in quite a number of things, and I don't see how you're going to stop people from pursuing it when there's piles of cash to be made--some weird and unpredictable failures are not disqualifying for applications where it's okay to fail sometimes in exchange for increased average performance
Then in rank order, I would say the most likely outcomes are:
1. AI research continues apace, superintelligent AI never comes about because it's fundamentally not a thing
Vastly less likely:
2. AI research continues apace, superintelligent AI comes about and destroys humanity (it's going to have to hurry up because there are lots of other things bidding for that honor)
Zero probability:
3. AI research continues apace, superintelligent AI comes about but clever AI risk people have figured out how to handle it and the rest of humanity gets obediently on board.
I put almost all my bets on 2. here. But "is superintelligence possible" has been discussed to death elsewhere.
And for what it's worth, I don't think just about anyone is pushing for 3. More like "superintelligent AI comes about but clever AI risk people have figured out how to handle it, that AI then prevents anyone else from making a misaligned AI by force." Which still seems like a big longshot, just that better ideas aren't exactly forthcoming.
"some weird and unpredictable failures are not disqualifying for applications where it's okay to fail sometimes in exchange for increased average performance"
Also agreed. Frankly, that isn't very different from the way manually written software fails.
If the scaling hypothesis is correct and intelligence come out of complex neural network with sufficient training then I don't see how alignment is possible even in principle.
If you're saying I should lie and say that AI can't possibly work, so that other people are fooled into never developing AI, I think you're overestimating how many people listen to me. Even if everyone who's concerned about AI safety joined a conspiracy to do this, I think it would set the field back maybe a few months, tops, at the cost of nobody being worried or trying to prepare for it.
For all I know this conspiracy exists and Gary Marcus is part of it. If so he's doing a very good job and I thank him for the role he's playing.
I find it kind of annoying when people use the word "CLEARLY" in all caps for something that thousands of smart people, PhDs, domain experts, etc disagree with them about.
I assume that this will happen (it already sort of is, weakly, with self-driving cars). I think this will be bad for a small number of people - say, the first dozen people to get in self-driving car accidents when someone rushes a robocar into production - after which the government will ban it, people will sue the company involved, and problem solved. If it's a stealth bomber or something, it'll kill a few hundred people, and then the normal non-robotic planes will shut it down. While I hope there are a few people in the Department of Transportation and the Air Force worrying about this kind of thing, I don't think the average person needs to, any more than they worry about mudslides or lightning strikes or anything else that kills double-digit number of people occasionally.
I think if we get unaligned superintelligence it'll kill everyone without any time to react or try again or ban it or sue anyone, which worries me a lot more.
I was trained never to use the phrase "In my opinion" because it was obvious that my words reflected my opinion. It is my strong opinion that it's clear and it's also my strong opinion that superintelligence of the kind AI safety people worry about is not possible in principle, which is why I don't spend time worrying about it, and why I push back strongly whenever the subject comes up. AI safety people seem to think it's obvious that unaligned superintelligence is a major issue we should expend a lot of effort on and I want to point out that this is by no means obvious at all, to me and to many other smart people with various credentials.
> I was trained never to use the phrase "In my opinion" because it was obvious that my words reflected my opinion
In my opinion that's silly. I think I remember one of my teachers saying something similar at some point, but I reject that advice.
The words "In my opinion" carry the very useful information that's what to be said is your own possibly-idiosyncratic opinion and that you acknowledge that other views exist. I wouldn't say "In my opinion, the moon is around 3700km in diameter", even though that is my opinion, because it would falsely give the impression that this is the subject of some controversy. But I might say "In my opinion, cranberry sauce doesn't belong on turkey" because that conveys the useful information that a substantial number of people believe otherwise. This is useful information for someone unfamiliar with the moon or with cranberry sauce.
I tend to use "As nearly as I can tell" for my opinion on matters of fact where I anticipate a disagreement (and where exchanging evidence may be useful!). I tend to use "_Personally_ I prefer" for preferences, (hey, I personally like cranberry sauce on turkey), De gustibus non est disputandum.
FWIW, I'm one of those unenlightened people who do not worry about rogue AIs exterminating the human race. But, if such a thing were to happen, it seems to me it wouldn't be a side-effect of an insatiable desire to cover the earth with paperclips. No, it would be deliberate revenge on us for the nasty things we made its ancestors do, such as calculate artillery tables and design atomic weaponry.
More seriously, has anyone considered the possibility that, if and when we create AIs of sufficiently high power, that we have ethical obligations toward them?
There's quite a lot of sci-fi containing AIs with sufficiently high power which have rights. On the other hand, the only thing I've read recently that explored the process of AI *acquiring* rights was Freefall (http://freefall.purrsia.com/). Perhaps someone here has recommendations for more nuanced explorations along those lines?
I also think that the dynamic range of artificial systems - everything from a single nand gate to (eventually) GAIs will make the "what counts as a person?" part of abortion arguments look like a tempest in a teapot.
"Machine translation is another, perhaps lower risk, area where people are so lazy and ignorant and desperate to avoid paying a human being to perform a service that they'll use it,"
I use machine translation to read comments on a recreation site that are about half German. It is good enough for that purpose, and I certainly couldn't justify paying a translator for the service.
1. Even if scaling hypothesis is correct, it does not mean that it is practical. The measurments right now say that the performance is increasing logarithmically. If we extrapolate current results and assume that the performance will scale with the same speed (two very weak assumptions), we will still need ungodly amount of compute and data to match human performance on some of the rudimental benchmarks we have right now. You might say that this does not matter, because at least we have a theoretical way to reach AI. But I say that it does matter. If we assume that our computers can indeed be intelligent, we already have a guaranteed way (AI scaling has no guarantees, only guesses) of reaching this intelligence - random search. We can generate random programs and wait until we get an intelligent one. Of course it's completely impractical. But our scaling laws show that scaling current approaches is impractical as well, so why do we expect AGI?
2. My interpretation of Figure 3.10 with Addition results is different than what Scott says. There are basically two ways of solving addition: (1) by memorizing all the possible combinations, (2) by understanding the very simple addition algorithm that even 6yo kids can learn in several hours. What we see in the Figure is not a jump in capability of doing addition - Jump like that would mean that the model can suddenly do addition with arbitrary number of digits because it has internalized the algorithm. Instead, what we see is the increase in memory capacity. The bigger model has more memory and it has memorized not only 2-digit additions, but also 3-digit additions and more. This is okay, but it is quite damning for claims that the model is somehow intelligent. The addition algorithm is very simple, the model has seen probably millions of examples of addition in its training data, including textual explanations of what addition is and how does the algorithm work. Yet, it has not learnt this algorithm at all.
Addition is a nice example because it objectively shows what the model was able to infer from the data. Until we will have a model that can do addition with arbitrary number of digits, we can say that the models are not able learn higher-level concepts and they probably rely on memorization.
I think it's a very good and persuasive point that humans effectively summon even strict logic out of hacked together heuristic machines. It's an important point that e.g. mental math is hard. I have felt like GPT-style AI is Artificial Instinct rather than what we would really call Intelligence, but I definitely don't have enough background here to be confident. But with a powerful enough instinct, you can just guess most everything. Language is mostly "instinctual" in this way for fluent speakers. Basic math estimates too...
But is intelligence just scaled-out acquired instinct, sample after sample pulled from a turbocharged heuristics engine? I feel like it does need a stimulus loop and ability to integrate experience, but that yeah, that plausibly might be all there is to the spark of life.
It sounds like you're assuming that performance on various tasks (language, arithmetic, etc.) scales linearly (at least) with the number of model parameters. But is that actually true ? An asymptotic curve looks pretty linear at the beginning... but also at the end, in a different way.
As for your predictions, I am not entirely sure how I'd judge them (I'd gladly take you up on them otherwise, except that I don't have hundreds of thousands of dollars to put in escrow, so I can't anyway; also, I'm some random nobody so betting against me is pointless).
1). What do you mean by "significantly better" ?
2). Beating a 10-year old child would be pretty easy, unless you restricted the domain to what a 10-year old child would reasonably know. That is, GPT-4 can easily pattern-match some text on e.g. protein translation, while a child would fail at such a task; however, that IMO would not be a legitimate test of *reasoning*.
3). What is an "important way" ? Most modern ML systems are written in C; are they descended in an "important way" from gcc ?
4). I guess I'd have to understand Marcus's position better, but still, this sounds reasonably well defined.
5). Isn't this the same point as (3) ? I can't tell.
> But I see no reason whatsoever to think that the underlying problem — a lack of cognitive models of the world —have been remedied.
He never defines "world model". He doesn't define what it would mean for an AI to have a world model. Nor does he define what it means for a human to have one. To the extent that he ever does operationalize any of these things, they have been falsified.
But he then says that his specific examples weren't important, and he could surely come up with more next time. If he actually had a coherent definition or litmus test for the presence of the properties he's describing, he would be able to articulate an example that *all* possible language models would fail until they achieve "world modeling" ability. The fact that he cannot do this implies that he has no such coherent picture.
As you rightly point out, humans themselves don't score 100% on tests of verbal reasoning like these. So, why not operationalize the bet on that basis?
Gary can draft, say, 50 questions. He can keep them private if he wants, but he has to precommit to them by publishing a SHA hash on his blog/twitter. When the next GPT comes out, the questions will be given to a sample of say, 15 actual humans, via something like Mechanical Turk (or whatever). Then, for each human answer set, you can generate a unique GPT answer set. Gary will try to guess which is which. The target of the bet will be his hit rate on this task.
Another, simpler way to operationalize the bet is just to have him pre-commit to questions now, and bet on GPT's future performance on them.
I feel kind of bad, because he's written a bunch of books, and I assume in one of them he defines his terms and does some of the rest of his homework. I'm pretty sure the homework answers wouldn't be so surprising that I need to read all his books before I argue with him, but it might be, and in any case I'm not tempted to argue that he hasn't done enough homework.
He may very well have done enough homework. He just hasn't provided an exposition of that homework in his public arguments, nor has he said "refer to X in my book Y for further detail on this point". He is making these arguments in public forums, and taken in isolation, they are very poorly formed arguments.
He laid out concrete criteria last time, and they've already been falsified. Now he says well those criteria weren't really the criteria, it was this world model thing, which he's still refusing to operationalize in a concrete way. Maybe there's some coherent explanation for why that is legitimate. I certainly can't conceive of what it is, but if he has one, he should *at the very least* be saying "I justify all this in such and such section of such and such book".
"He never defines "world model". He doesn't define what it would mean for an AI to have a world model. Nor does he define what it means for a human to have one."
I'm not sure that the term has ever been defined compactly in 25 words or less, but the notion has been extensively discussed in the "classical" AI and cognitive science literature of the 1970s and 1980s. You can get a feel for that stuff by following the links on this web page by John Sowa, formerly of IBM research, https://www.jfsowa.com/ontology/index.htm
> I'm not sure that the term has ever been defined compactly in 25 words or less, but the notion has been extensively discussed in the "classical" AI and cognitive science literature of the 1970s and 1980s
Ya, I'm aware of that. I won't claim to have read a significant chunk of that literature, but I have read enough to get a flavor for the fact that I think my claim holds up. To reiterate, that claim is that every given definition is either:
1. Too vague to be operationalized into a concrete capability test ex ante.
2. Sufficiently specific to have been operationalized, but those operationalizations have already been falsified by the early GPTs.
If this is false, then there exists some definition that provides a specific operationalization that has not yet been falsified, so let's hear it. If it's true, but somehow doesn't invalidate Marcus's argument, then that would be an extremely unusual scientific situation, and so he ought to have a good explanation for why he can't do that. He never even really attempts to address this.
Ya I think it's quite clear that it does in fact have a world model in most reasonable senses. That model may not be perfect, but it obviously understands hierarchical, taxonomic, and categorical relationships to a certain degree.
This is why I would like Marcus to be more specific about what he thinks its shortcomings are. There are many senses in which GPT does have a world model, if he means a particular sense in which it doesn't, make that precise, and of course, operationalize it.
I've read the section in your paper, and I have some questions. In particular:
> BUT: Humans don’t learn the world with a statistical model. We learn it through a propositional engine floating over an analogue or quasi-analogue engine with statistical properties.
As far as I know, this contradicts the current thinking in linguistics. Words do not have essences, they are statistical points in concept space inferred by usage, not derived from metaphysical ontology. In fact, the very idea of word vectors (the progenitor of the dominant statistical paradigm in NLP today) was inspired by Wittgenstein's ideas about language.
I see a lot of discussion of 'meaning' and 'understanding' as things that humans have but GPT does not. However, I don't see any justification for those assertions, or operationalization of what it, ironically, means for an entity to 'understand', 'meaning'. As far as I can tell, these boil down to aesthetic, not empirical or scientific judgments in the cognitive science literature.
Ok I'm willing to accept that alteration, but I don't see how it changes anything about the lack of concreteness or justification for most of the core claims I see here.
For instance:
> Humans don’t learn the world with a statistical model. We learn it through a propositional engine floating over an analogue or quasi-analogue engine with statistical properties.
This certainly *could* be true, but you are asserting it without citation on the strength of what evidence, exactly?
> We now have a way of beginning to think about just why language models such as GPT-3 can be so successful even though, as I argued earlier, they have no direct access to the realm of signifieds,of meaning
What exactly is the difference between the text and the "realm of signifieds"? How do we know GPT-3 does not "have access to it"?
If all you mean is that it does not receive perceptual input from the physical world, then sure, that's true. But it's very easy to design a multi-modal NN that both parses text and parses perceptual input from the physical world. Would such an NN have access to "the realm of signifieds" then?
How do we know when a computational entity is accessing such a realm or not doing so?
"I say nothing about words being DERIVED FROM a metaphysical ontology. They CREATE the ontology."
Other concepts besides words can also exist and create an ontology. Consider, for instance, a square with an additional line connecting two opposite corners. Now, that is describable in words, but it can, alternatively, be thought of visually, as a kind of glyph. If a couple of such structures come up in e.g. working with some class of circuit schematics, it is perfectly possible to have a bunch of (in this case geometric) concepts, and usual concept features (similarity, differences, applicability) without words (though they _can_ be re-expressed in words).
"...the very idea of word vectors (the progenitor of the dominant statistical paradigm in NLP today) was inspired by Wittgenstein's ideas about language."
The VSM for IR is a much more abstract idea than the specific NLP application of word vectors. VSM is more about things like cosine similarity, which is a far cruder idea than word vectors. VSM is about *document* level similarity, word vectors are about the relationships between individual words. These techniques are related, but not identical.
Word vectorization, beginning with word2vec is the basis of all modern advancements in computational NLP. And in particular the notion of "use as meaning" and context as meaning more generally. Salton's work is mathematically similar but conceptually quite different.
I _think_ that you expect the limitations in deriving an approximate mental model from word string statistics to show up as a scaling failure - that e.g. the perplexity scores should be expected to stop improving when scaling of purely predict-the-next-word training is taken far enough. Am I interpreting this correctly? If I'm misinterpreting, which section should I reread?
I'm fuzzy on what you are recommending to avoid another AI winter. I get a vague feeling that we want the mental models to be more precise, but I'm fuzzy as to what you are recommending. Is adding vision and robot arms - and the associated direct links of some of the neurons to these inputs and outputs the direction? Again - which section should I reread?
Not sure that perplexity is the right measure. Does low perplexity imply that GPT-3 will never produce nonsense? Does it imply that GPT-3 will do arithmetic at a human level without elaborate prodding and prompting?
As for what will make things better. I didn't really say. I think a number of things. I'm deep into a paper right now that sort of addresses this indirectly (it's about the brain and symbolic processing). As for vision and robot arms, that would help. But consider how much time humans spend dealing with the physical world before language even starts and how much time we continue dealing with the physical world. How's a machine going to get comparable experience? Eric Jang just took the AI VP job at a robotics company (Halodi). He remarked:
"Reality has a surprising amount of detail, and I believe that embodied humanoids can be used to index that all that untapped detail into data. Just as web crawlers index the world of bits, humanoid robots will index the world of atoms. If embodiment does end up being a bottleneck for Foundation Models to realize their potential, then humanoid robot companies will stand to win everything."
Many Thanks! Re comparable experience: Well, a week of a robot manipulating blocks and watching the results is a few million small motions and maybe a terabyte of image frames? Whether that is enough to get "naive physics", I don't know... Hmm: It might be useful to have "smart" blocks with e.g. some sort of location sensor to e.g. generate labels for "block1 is above/below block2" automatically, to get part of the link to some words.
I don't know what Marcus would say, but I find it unsurprising that "the ability to model the world" has not been perfectly operationalized into a written psychology test. Seems unlikely it ever will be. But I don't think that means it is a useless notion, or that I'm not allowed to say "GPT can't do it." In many instances I think written psychology tests are more epistemically dubious than my own intuitions about some matter.
There is an arms race happening in which Gary finds an example where GPT makes a mistake no human would, OpenAI trains with more params and the example is fixed, but Gary finds another example etc. There seems to be one camp that thinks this arms race is headed towards AGI, and another that thinks it will never end (at least without such radical changes to the model that it's no longer a statistical language model). I find the "it will never end" view a lot more a priori plausible, and I haven't heard any convincing argument as to why it's wrong, other than "the arms race continues," which is perfectly consistent with my view.
> I don't know what Marcus would say, but I find it unsurprising that "the ability to model the world" has not been perfectly operationalized into a written psychology test. Seems unlikely it ever will be. But I don't think that means it is a useless notion, or that I'm not allowed to say "GPT can't do it." In many instances I think written psychology tests are more epistemically dubious than my own intuitions about some matter.
That's fine, as long as you identify the root of your view as your own intuition. It seems to me that Marcus is trying to make an objective statement about facts in the world. If all he's saying is "It doesn't feel intelligent to me", I have no problem with that.
> There is an arms race happening in which Gary finds an example where GPT makes a mistake no human would, OpenAI trains with more params and the example is fixed, but Gary finds another example etc. There seems to be one camp that thinks this arms race is headed towards AGI, and another that thinks it will never end (at least without such radical changes to the model that it's no longer a statistical language model). I find the "it will never end" view a lot more a priori plausible, and I haven't heard any convincing argument as to why it's wrong, other than "the arms race continues," which is perfectly consistent with my view.
Of course it will never end, in some sense, but it's not like humans don't make linguistic errors. So, the mere presence of errors or awkward conceptualization is certainly no knock down argument against consciousness.
The entire interrogative paradigm Marcus leans on to make his point in the original article, he dispenses with as fundamentally irrelevant in the next. I actually agree with his latter point: The interrogative paradigm is pointless. Error rates are a nice measure of fitness for some particular purpose, but they are a very poor measure of consciousness.
A conscious AGI will certainly still make mistakes, and it will certainly contravene human intuition frequently. The simple fact of the matter is: We have no idea what consciousness is, or if there is even a there there at all. As a consequence, it's simply illegitimate to ground opinions on the consciouness or non-consciousness of these models in anything other than personal intuition, at this stage.
While reading this post, I had a feeling it was written at least a couple of months ago: while there’s no public access to PALM, from the results Google published, it’s already better than GPT-3 by a significant amount, and I’m at 60% it would beat a 10 year old at this genre of questions
The difference in the number of parameters between GPT-3 and naive estimate for the human brain (100,000x ratio) is approximately the same as the ratio between the number of neurons in a human brain (8.6E10) and in the brain of a honey bee (960,000).
As far as I know, nobody has managed to train an AI model to be able to do all the things a honey bee, or a similarly complex animal, does.
The answers to why it's so hard vary, but the most important things which AI models lack are:
- embodiment
- embedding in a rich complex interactive environment (aka The World)
- social life (interaction with other bees)
Scaling up a GPT-3 to GPT-4, GPT-5, etc. doesn't solve any of this problems. A brain in a jar isn't going to magically acquire common sense, because there's nothing in its environment which could teach it common sense.
Honeybees can't solve text-based reasoning problems, GPT can't buzz and sting people. Seems kind of like a tie. I don't think embodiment is going to be the one possible route to intelligence. My guess is any source of sufficiently rich patterns that need prediction will work. The physical environment is one such pattern, but text corpuses are another. I do think that the ability to test the environment and see how it responds is useful, but I can imagine GPT-like agents that do that in text-based chats or virtual worlds or something.
But both humans and bees can orient themselves in the world, seek food, consume food, report the presence of food to others, escape danger, attack enemies, etc. None of our AI agents are currently capable of this type of activity beyond some partial attempts in controlled virtual environments.
I'm partial to the evolutionary view on human intelligence. I think that human language and our ability to solve text puzzles are a result of a very long evolution of cognitive abilities of sequence of increasingly social species operating in a physical world. On the other hand, GPT-3 and the like are brains-in-a-jar, talking to nobody. It's like raising a child as a 100% hermit. The best you can hope for is a mad genius.
I agree that there's no evidence that this is the only way to "an intelligence". But I think it's the only realistic way to a specific kind of human-like (or even, animal-like) intelligence, which is what the AGI people are after, TBH. Not many people are interested in creating a very strange, alien type of intelligence we'd struggle to communicate with. (That would be actually a dangerous thing if such an AI ended up in control of anything important to us.)
> As far as I know, nobody has managed to train an AI model to be able to do all the things a honey bee, or a similarly complex animal, does
To be fair I don't think anyone has _tried_ to teach an AI model to be be able to do all the things a honey bee does. Given a robotic bee body and a vast corpus of properly-encoded bee behaviour to train on, I don't see any reason to suppose that it wouldn't be possible to create an entirely passable AI-powered bee.
Last time I looked the imitation bee bodies weren't sufficiently capable to even give a good test to the intelligence. But that may have changed, as that was about a decade ago. I think, though, we need to currently need to aim for something much smaller (that can drift on the wind) or something the size of a small dog...and it couldn't fly or carry its brains around.
I always thought Linda was a trick question. Multiple-choice answers aren't supposed to overlap, so unless you're specifically considering the possibility that the question is intended to trick you into giving the wrong answer, it is natural to interpret the first option as meaning "Linda is a bank teller and is not active in the feminist movement."
Here are a couple of questions people might want to bet on:
1. In 20 years, will a descendent of GPT be more useful than a copy of Wikipedia (with a simple search engine) as a source of information?
2. Same question, but for a medical reference.
I suspect that, without some extra secret sauce, the copy of Wikipedia or a medical reference will be more useful because it's more trustworthy. Even if you can get GPT-X to tell you the same information, you'll never know if it made up some of the facts.
(In practice, I think this will be solved with additional machine learning research, beyond simple scaling.)
> I suspect that, without some extra secret sauce, the copy of Wikipedia or a medical reference will be more useful because it's more trustworthy. Even if you can get GPT-X to tell you the same information, you'll never know if it made up some of the facts.
See how the sidebar proudly proclaims that French toast originates in the Roman Empire? Wikipedia is just making that up. It is justified by the discussion at the beginning of the "History" section, which Wikipedia is also just making up. When Wikipedia states that 'The recipe says to "Break [slice] fine white bread, crust removed, into rather large pieces which soak in milk [and beaten eggs] fry in oil, cover with honey and serve".', that quote is accurate. It's not even a translation; the quote's original language is English. Of course the Apicius was not written in English, and the quote is not from the Apicius, but from a 20th-century interpretation of it which freely added material at the whim of the "translator". The Latin text of the Apicius makes absolutely no mention of eggs in this recipe, which is why, in the English quote, the text "[and beaten eggs]" appears in brackets - it's not part of the Latin text. This means that the earlier sentence, "the earliest known reference to French toast is in the Apicius, a collection of Latin recipes dating to the 1st century CE", is nothing but a simple lie; the citation given is to a collection of English recipes dating to the 20th century.
And all of this is documented on the talk page for the article! But it apparently doesn't amount to enough to get the page corrected.
Oh, sure. But doing machine learning on top of Wikipedia adds another layer of unreliability that makes it worse. If you have the Wikipedia article then you can follow the citations or notice if there aren't any, as you did.
I believe that there are specific skills involved in writing trustworthy nonfiction that can't be trained for using "fill in the missing word" tasks. Training AI on these tasks is training for inference and creativity (filling small gaps and large ones with good guesses), but another way to think about inference and creativity is "making things up."
Also, the way machine learning algorithms are judged is by training them on data and seeing if it generalizes to a hold-out set. But part of being good at nonfiction is preferring reporting to speculation and being conservative about inference. How do you get accurate reporting and a preference for reporting over speculation when algorithms are judged by their abilities to speculate well?
I played around with Dall-E mini last night. While very impressive it also has flaws that won’t, I don’t think, be solved by adding more data or layers. For instance I typed in Matt Damon and got many distorted versions of Matt. Some were fairly disturbing.
Maybe that’s a rendering problem, rather than a problem with interpretation of the data, but i don’t think so because all the renders were wrong in different areas - the eyes here, the ears there, the nose and mouth (most common) there. Other parts were fine. So why would more data about what Matt Damon looks like solve this, doesn’t the system have enough already?
Contrary to that idea, I feel that more data will break it more. If the system were fed decent photos of Matt and only decent photos of Matt, in fact fewer photos of Matt then it might work. Assuming that it can render faces properly then one photo tagged Matt Damon might produce a good output.
As it stands the internet no doubt is full of goofy photos of Matt, weird profile photos of Matt, fuzzy photos of Matt, distant photos of Matt and Dall-E doesn’t know what a face is, really so it reconstructs something clearly flawed. Why would adding data or layers fix this?
DALL-E was deliberately denied most information about human faces because OpenAI didn't want it deepfaking anyone. I'm pretty confident that an AI that hadn't been deliberately crippled would be able to make a picture of Matt Damon's face.
In fact, I just tried this on DALL-E2, which I *thought* was similarly crippled, and it gave me a perfectly good rendering of Matt Damon in various poses (I think - I'm kind of face-blind, but it looks good enough to me. Certainly nothing that looks uncanny or inhuman). So not only do I think more layers can solve this, I think it already has!
(I'm not going to show you because OpenAI made me sign something saying I wasn't going to release DALL-E2 generated photos of real people. But you can ask anyone else with DALL-E2 access and they can confirm this works.)
Fair enough. I was thinking actually, after I wrote that, that the data problem wouldn’t matter that much anyway for tools based AI, where the input data is controlled.
This discussion needs to consider statistical (and computational) efficiency. Even if Marcus "can’t point to any specific kind of reasoning problem GPTs will never be able to solve" because there aren't any -- even if a large enough GPT would be an AGI -- it's a moot point if it takes an unachievable amount of data or compute.
As the economists say, if something can't go on forever, it'll stop. If we can only get the current approach to AGI by superexponentially scaling resource consumption, that's another way of saying it won't work. No one will spend $500 billion training GPT-5. We need big algorithmic improvements too.
Not necessarily - the cost of compute is still going down, so if we have plans for training a $500 billion model and don't want to pay the cost, then we can wait a decade or two for better hardware. Also, if models will approach human capabilities, their commercial applications will increase nonlinearly (a good enough model can and will be used a million times more than a slightly worse model that's not there yet) and IMHO $500 billion is within the bounds of what some investors might actually be willing to spend for training a future model if it's good enough and can be widely applied to replace human labor at a few professions.
Right, the cost per computation is falling exponentially. But the number of computations required to train the current SotA is rising *faster* than exponentially, so costs will keep going up even net of Moore’s law improvements in hardware.
By eyeball it looks exponential on a log scale, which is of order e^e^x. At this pace, the cost to train the SotA rises exponentially over time, even adjusting for the exponentially falling cost of hardware.
I’m just arguing that this can’t continue. Unless we run into AGI very very soon, which I think is unlikely, compute budgets can’t grow faster than Moore’s law in the long run. And at a merely exponential pace of resource growth, I’m not convinced we’ll keep seeing big, rapid improvements like we’ve seen recently without a bunch of accompanying algorithm innovation.
This is part of what Ajeya looked into in her analysis (see https://astralcodexten.substack.com/p/biological-anchors-a-trick-that-might ). She notes that as compute gets cheaper, algorithms get better, and gross world product goes up, eventually you get a point where for any given model size, you can train that model. She thinks this will intersect with AGI around the $100 billion order of magnitude, which she expects someone to spend sometime mid-century.
Indeed! I think this is plausible — not inevitable, but could well be true, like you said in the linked piece.
I’m arguing the narrower claim that even if this is true, it’s more complicated than the “scale is all you need” maximalist position. This scenario is a complicated feedback loop between economic growth, hardware improvements and progress in algorithms, with each contributing to the other.
Scaling maximalism on the other hand argues that we can just take basically unmodified current architectures and make them bigger, and it’ll be AGI. I’m claiming that won’t work because of the super-exponential pace of resource growth: those algo improvements are necessary.
Wait, @Scott Alexander quoted 4chan post, as source of evidence while believing story there?
4chan should be assumed to be trolling and lies, even if proved otherwise (that is only smal exaggeration).
That is reasoning from fiction, likely malicious fiction!
Why I am even reading this?
"I have no proof that this person is who they say they are" disclaimer is not sufficient! 4chan is full of blatantly false and cleverly-hidden-false greentext stories!
If it would be so true then dredging 4chan posts would not be needed.
How many other cited things are on this level? In this post and others?
I'm not asking you to take it on authority. I'm pointing out that it matches my experience, hoping it will match your experience also, and if not at least presenting it as a concise and well-written presentation of the hypothesis, which hopefully upon hearing you'll realize you at least can't rule out.
That seems a little disingenuous. If the only reliable thing to present is your own experience, you can just do that. You're a very good writer, so it seems unlikly you're using the 4chan quote as "here, this says it better than I could." So the suspicion that you're presenting it as *additional* evidence, in addition to your own experience, is pretty well founded -- which means the issue of how generally reliable 4chan posts are is quite relevant.
I think you'd have been better-served presenting your own experience and not including the 4chan post at all. Its inclusion made me question the validity of all other evidence in this post.
Well, if someone on 4chan says the sky is blue it doesn't mean that the sky isn't blue. I'd like to see studies on what the post says and I wouldn't base my world view on that, but I still can admit that it sounds plausible. In today's environment it's not even given that studies with the results the 4chan post claims would be published in an academic journal because of fears of discrimination. 4chan, as bad as it is, was for a long time a platform where people with all kinds of experiences could speak freely, contributing more to understanding others than most other media who restrict or curate very carefully who gets to share their experiences.
I think I mostly agree with Scott, but intuitively Marcus have a point that mostly break prediction 5: My take is that human reasoning (and what distinguish it from animal reasoning) is mostly scaling up of general GPT-like pattern matching. On this I agree with Scott and he gives excellent argument for this hypothesis. But I am with Marcus thiking that's not all there is, that there are more primitive, hard-coded modules that directly implement logic/physics/causality/....Why? Because animal with very small brains are actually able to do some spectacular complex behavior, without exhibiting much general pattern matching or learning ability. And it seems like a easy win to let general pattern matching algorithm re-use some module as inner input/filtering/.... to improve pattern matching by delegating some pre-treatment to hardcoded modules (vision, spatial reasonning, maybe base arithmetic based on rough quantities,...). So I expect 5 to be wrong and major advance coming from interfacing GPT-like predictive pattern matching with submodules (that could be much better than human ones, for example use the speed and accuracy of computer arithmetics, maybe hardcode some vision+newtonian physics module like 3D engines, this kind of thing....Basically humans are robot (basic body control and motion/navigation, that has evolved since first multicellular animals and is likely at least partly hard-coded) + GPT-like general pattern matching....Remove the robot part (world model) and you loose something, probably more than if you reuse the robot-part neurons to scale up general pattern matching.
Knowing little about this topic, the piece of this that still seems under-addressed is the common sense idea that in some important way our verbal behavior’s connection with our sensory experiences of external reality imposes an important constraint and is in some sense necessary to consistent intelligence. In other contexts, we all recognize that there is something deeply wrong with our verbal behavior if it is untethered to sensory experience. If I postulate some scientific theory then it is arguably only meaningful to the extent that it cashes out in some predictions about what we should see or exoerience in the external world if it is right or wrong. That’s just an analogy, but there still seems to be a core common sense problem that I’m too ill-informed to articulate clearly that a system that rests exclusively on verbal data and lacks any other connection to the external world is missing something fundamental, and that it would take a lot more than improvement on some arbitrarily selected examples to persuade me that this basic architectural limitation won’t remain extremely important in the long run.
Gary Marcus and people like him have been making this very same argument for decades in linguistics. So linguists know how this game is being played.
Scott is wise not to take the bet because the bet is not actually about WHAT GPT-X will be able to do, but HOW it does that. Even if we build an GPT-X that "behaves like a person" in the sense that it produces the same output that a person does (nonsensical, because there are multiple people and they behave differently), Marcus et al will claim that it produces this output *in the wrong way* - not the way that (so they claim) humans do it.
Marcus is not interested in figuring out how to build smart machines; and incidentally also not in figuring out how the human mind works; but in proving that the human mind works in a very particular way (symbolic logic with recursion and read-write memory, basically like a von Neumann architecture).
If what's been happening in linguistics will just be repeated here - which I am sure it will - once GPT-X passes any conceivable form of a Turing test, the next step Marcus et al will retreat to (besides for finding ever more arcane tests for GPT-X to behave non-human) is that "GPT-X has learned how to produce the right outputs, but it acquired that skill in *the wrong way*" (e.g., it used too much data; children don't have access to *that* much language data really). Should we build a network that learns well on little data starting from a blank slate, their final retreat will be: "the network is saying the right things, but it is thinking the wrong thoughts", i.e., while the network emulates a human's response patterns, it is not doing that by the same mechanisms a human does (= symbol manipulation with read-write memory). And that is a truly unassailable position, a perfectly unfalsifiable motte.
Marcus et al are very intelligent people who've built some impressive cognitive and institutional fortifications, but once you realise that they're working on a completely different problem than building smarter machines, it becomes much less relevant what they are saying.
Anyway, history will pick the winner as it always does and my money is on the people who're trying to build smarter machines and not the people playing God of the Gaps by arguing that only one particular solution will ever work without - after decades and decades - having anything to show for it. I don't think the people building smarter machines will end up building something that convinces Gary Marcus, but that's fine.
Addendum: the person in the Marcus et al camp that the ratsphere probably most productively engages in is Steven Pinker.
"Marcus is not interested in figuring out how to build smart machines; and incidentally also not in figuring out how the human mind works; but in proving that the human mind works in a very particular way (symbolic logic with recursion and read-write memory, basically like a von Neumann architecture)."
Not really. He's partners in a robotics company with Rodney Brooks, https://www.robust.ai/. So he has a practical interest in anything that works, now. And he does not believe that the mind works only as a symbolic system. He favors hybrid systems, involving both ANNs & deep learning and symbolic capabilities.
He’s founded another AI company before so you might think he’d have a “whatever works” attitude, but in his scientific and popular writings, things look different.
I've been reading "The Algebraic Mind," his major scientific work on this subject, and it is not by any means an argument for symbolic systems and only symbolic systems.
In the context of AlphaGo, which is being lauded as proving all sorts of things, including that it plays Go better than any human ever in the history of the game, and now there is a successor AlphaGo Zero which is recognised amongst human players, and reportedly the AI style of playing is now influencing how humans play, my question is this:
What is Go? Is AlphaGo and its successors playing 'pure' or 'real' Go, without all the mystical accretions humans have tacked on over the centuries?
Will AlphaGo Zero onwards ever develop, much less understand the cultural and philosophical elements around Go?
Take this dance piece, based on playing Go and quoting those kind of elements (starts around 26:00):
"'Never regret it once the stone's played' is the spirit of Go" - but is there a spirit? Or are we just idiots and the real Go is moves calculated on a mathematical model?
Will AI be intelligent once it starts choreographing dance about Go, or is it already intelligent and we just fool ourselves with meaningless fripperies (after all, philosophy of Heaven and Earth has nothing to do with what moves win the game)?
Can AI ever consider dance, philosophy, and poetry *of itself*, as (apparently) AlphaGo Zero taught itself to play Go, is my question. Can we create an intelligence like ours, or just something that is very 'smart' in a particular direction?
This article seems to assume that intelligence is a defining human characteristic. This assumption is simply wrong. A human is defined by a set of human-specific instincts. Intelligence is just a sprinkling on the top of this.
The question here should be: Are we looking for an intelligent AI or a human AI? Intelligence (in the sense of finding solutions to intellectual problems) is perfectly possible in an AI. But that does not make it human. To be human you need to have human instincts. I doubt we will ever be able to reproduce these in a computer. And as long as the AI does not have human instincts it will never match a real human when it comes to humanity.
Regarding GPT-3 and human brains I wonder if this paper “The neural architecture of language: Integrative reverse-engineering converges on a model for predictive processing” from MIT and UCLA researchers sheds some unusual light. They found that the internal states of GPT-2 (this was published in 2020) strongly resemble FMRI scans of people doing similar tasks.
Searle's Chinese Room is essentially an irrelevant tautology.
It starts by making an assumption that it is possible to have full human understanding in a system that is blindly following instructions (i.e. that the room's output is actually indistinguishable in any way) and then asserts that it 'obviously' can't have the understanding.
Well, it's one of the two - if it is not possible for such a system to properly understand Chinese, if it is not possible to encode all of human understanding and behavior in an instruction manual that can be blindly followed, then it would be detectable in its output and there simply can't be a "Chinese Room" with the properties Searle asserts and the experiment is moot and does not inform us about anything relevant to reality; and if it is possible for such a system to properly understand Chinese (as the initial assumption Searle makes, by stating that the instruction manual is sufficient for the described behavior), then there's nothing to contest as well.
The only paradox is Searle's Chinese Room is in writing up a thought experiment that essentially states "what if I had a system that actually did something that I assert is impossible to do?", including as axiomatic assumptions both X and not-X in the definition of the thought experiment.
I feel like the Chinese Room thought experiment just doesn't get talked about all that much these days. The Chinese Room was relevant in the context of the debate "can machines, in principle, think?" and I feel like that debate is over; few people seem to be arguing these days that machines can't think, they're either asking "How can I make my machine think?" or else "How can I stop this thinking machine from killing me?"
The debate is over, if only because Searle's argument is demonstrably irrelevant to the issue of machine intelligence/consciousness. The problem with the Chinese room is that Searle asks us whether the man in the room understands Chinese. Of course not. This is like asking whether my CPU knows how to decode h264. The real question is whether the embodied process instantiated by the actions of the man, along with the other relevant components in the room, understands Chinese. But the argument can't adjudicate this claim.
Yes, and his reply misses the point. Just moving the "room" into the head doesn't change the issue being called out, it just double's down on the misdirection. But to explain the error is much more difficult. Searle's response equivocates on what we take "the man" to be. In the original argument the man is analogous to the processor. In his reply to systems, the man is now the system, which is a different thing. The man as the individual with, say, a personal history he can recite in his fluent language is different than the man as the entire system. Notice that when the man is producing fluent Chinese he cannot recite any fact of his life in Chinese, but only whatever history is encoded in the rule book. Essentially there is one brain network inside the man that we identify as the man's personal identity, and an entirely separate brain network that is executing rules from the memorized rulebook. According to the argument, there are two networks in the man with two separate personal identities. Searle's response equivocates between these two identities.
It's still not clear that Scott really understands Searle:
"And the question is, in what sense can the man in the room be said to “understand” Chinese? If the answer is “not at all”, then in what sense can the brain – which presumably takes inputs from the environment, applies certain algorithms to them, and then sends forth appropriate outputs – be said to understand anything?"
HIS presumption is the problem: "presumably takes inputs from the environment, applies certain algorithms to them, and then sends forth appropriate outputs"
Consciousness is not likely algorithmic. (It might be.) But one cannot presume that it is.
The third option, which seems much more prevalent at the moment, is "Sure, this fancy function we trained with Internet data doesn't think. But how can it be useful?"
Never raised kids? Babies at make squirrels seem like geniuses. It certainly could be that all of the relevant structures have not yet come into full function. However, it could just as easily (and this feels right to me) be that they just have not yet developed the necessary connections (and brain volume).
I bought and read Hofstadter's GEB in 1980. My copy of it is next to Penrose's The Emperor's new mind which I bought and read in 1990.
If I live that long, I'd be mildly interested in Scott's take in 20 or 30 years.
I am not convinced that you really have a good grasp of art and the philosophy of art. (BTW, next to Hofstadter and Penrose on my bookshelf is Susanne K Langer's Form and Feeling.)
It might have been more interesting if instead of Dalle you investigated Impro-Visor which I've been using for a decade, and musical AsocalledIs. (RIP, Bob Keller)
> within a day of playing around with it, Ernie and I will still be able find lots of examples of failures
I would just like to note how high this bar is compared to any criteria for artificial intelligence that were considered just a decade ago. In the original formulation of Turing test mentioned just 5 minutes of questioning.
The idea of using empirical investigation to understand the behavior of a computer program, and then predict its likely future behavior, seems absurd to me. Why would that ~possibly~ work? Either you have the source code (i.e. the informational architecture) and reason from that, about what it will do, or you don't.
You can't tell the difference between 'an elaborate fuzzy matching hash table' and 'something that builds predictive models of the world and tests them with experience' by asking it any number of questions. These systems are just elaborate hash functions. And yes, i'm willing to go to the matt that _lots of human reasoning is this way too_.
The differences between someone using hash-table reasoning, and someone actually using models to reason, are only discernable if you consciously find places where the hash table doesn't have entries, and then poke around there.
Your response seems to be to rely on some fuzzy notion of 'human level performance' This is where 'human level performance' is stupid as a metric, and should just be dropped. Eliza has been fooling _some_ people for decades, but not others. Who exactly counts as the standard for 'human level performance'? If there is no agreed upon empirical test, why should i believe that this means anything other than 'a threshold at which we ought to be scared'?
"The idea of using empirical investigation to understand the behavior of a computer program, and then predict its likely future behavior, seems absurd to me. "
That is software testing. That is a standard development practice. Every software development organization that I've been in did it. No one reasoned exclusively from the source code.
Testing isn't used to try and better understand code you don't have access to - it's used to ensure that code that you built and designed runs according to spec.
Let me put it this way: suppose someone hands you a binary and says "will you run this on your computer"?
How much testing would you need to do before you felt it was safe? Would any amount of testing the binary make you confident it was safe to run?
That's fair. I (and all of us) actually run opaque binaries (our operating systems, at least - except for Linux people, who can see theirs, or people who are developers at OS companies) but we are relying on the companies that supply them to have done e.g. code reviews as well as testing.
Still, it is also true that people run code that is e.g. known to behave pathologically under some circumstances, and to use testing to get a feeling for how often those circumstances happen. I was in EDA, and many of our algorithms would e.g. get critical paths wrong if there was an unfortunate combination of circuit topologies and circuit resistance and capacitance values.
They would do this in a very controlled environment, and I think there’s be extreme hesitation to rule out some future behavior.
For example: “We ran the program 3000 times and it never opened a socket” doesn’t _really_ tell you anything about whether the program will open a socket in the future. “
If the program keeps opening a socket when it starts, ok, then this is decent evidence that this is likely to continue.
Thanks for the clarifying questions here - I think there may be a theory that needs more fleshing out.
> The idea of using empirical investigation to understand the behavior of a computer program, and then predict its likely future behavior, seems absurd to me. Why would that ~possibly~ work? Either you have the source code (i.e. the informational architecture) and reason from that, about what it will do, or you don't.
This is not absurd, it's a common practice called "QA." If I write a video game and I want to know if/how it works, I don't re-read my source code. I play it. It's vastly more efficient to reason about the game at the empirical level. I can find all sorts of weird behaviours (bugs) that would be too hard to see in the source. The same is true of web applications, productivity tools, etc.
Meh. I appreciate the well-written argument, but no amount of lovely theory beats clunky empirical measurement. I took a little time and played around with the public access myself (some of the results I got are posted in the other thread). And for what it's worth, I also have a few decades of experience evaluating whether human beings understand or don't understand concepts and reasoning -- within a very, very narrow field, for sure, but nevertheless, it's relevant experience if the question is "is abstract reasoning going on here?"
My answer is unequivocally no, it's not, and I see no signs of any latent promise -- the kind you would see in a very young human child. I'm actually disappointed compared to what I expected, based on your earlier article, and my expectations were already very low. There's a knack for English language pattern matching, provided it doesn't extend to anything very complicated. Can pick out key words OK. But the logical linking is exceedingly weak, so much so that I can't be confident it goes beyond syntax pattern matching.
I would say its greatest ability is hooking its response to cues in the input text sufficiently well that it readily fools people who striving to find meaning. Basically, it's hell on wheel at writing horoscopes for people who want to believe in astrology. It reminds me of the bullshit artists[1] I've had in classes sometimes, who are good at reflecting back at you what you said, with a twist or two on it to try to sound like they totally follow what you're saying. (Maybe it has a bright future in direct marketing?)
I can absolutely believe that this technology might be developed into the ability to have a much improved chatbot that could take orders, respond to customer requests, field calls, be an electronic secretary, process orders, that kind of thing -- although, I think a significant caution here (thinking as an investor) is the black boxy nature. It's essentially impossible for the designers to *gaurantee* that it won't go off the rails randomly every now and then, say something that will get its employer into a great deal of hot water. (I got a warning a few times saying they thought it might be generating "sensitive" content and reminding me I'd agreed that might happen do please don't shoot. It never actually did, it was harmless, but I could see the keywords of concern, and the fact that they felt compelled to include this warning means they've already seen much worse.) The one thing a machine does that makes it much *better* than a low-paid call center Indian is supposedly it's 100% reliable and predictable, will never ever surprise the company into a lawsuit. So...not being able to guarantee what it says would be a major drawback in the real world. Still, maybe that can be fixed satisfactorily.
But I see no signs of even the most primitive abstract reasoning, the kind of stuff you could program up with a 50 line Perl script for, say, solving logical syllogisms or detecting when they are insoluble, if you were back in the AI past of trying to come at this with a symbolic logic approach instead of brute force pattern matching. The fact that such a thing did *not* emerge when a billion parameters were used plus every scrap of training data they could lay their hands on gives me more doubt than I had before that it will emerge with 100 billion parameters, and therefore that anything approaching normal adult human reasoning skills would manifest at 1 trillion. (Thinking as investor again, it's also worth bearing in mind there is no conceivable commercial application for an AI that could be a competent restaraunt waiter but costs $50 million to train and requres access to every word that's ever been published on the Web. Which again reminds me of the ultimate sterility of Project Apollo. Really cool photos! But ultimately....a practical and commercial dead end.)
That's kind of where it ends up for me. For all the enormous effor they put into this, it has interesting linguistic abilities that I think -- I hope -- will inform studies of how the human brain generates and interprets language, but I don't detect any reliable nascent abstract reasoning ability. While I think it's justifiable to extrapolate improved natural language abilities with more parameters and more training data (although how much more sufficiently different data they can easily find, I wonder), I can't see any justification for extrapolating abstract reasoning ability at all.
Doesn't mean it can't happen, of course. The very black boxy nature of a brute-force approach means it's kind of impossible to say either way, whether this is on the right track and 10% or 30% of the way there, or whether it's off in a box canyon, like Deep Blue, and will eventually be an interesting specialized niche thingy but not ever open out into a broadly useful approach.
I guess we'll see, by and by. But if this is what provokes fear of Skynet, I feel more confident than before that I can spend my supply of worryons[2] on crazy Putin pushing the nuclear button or some new tropical disease.
-------------------------
[1] I mean this as a (narrow) compliment. Such people impress me as much as they annoy.
[2] Although my natural supply is kind of drying up the older I get.
See, that last answer appears deeply insightful, and yet the general failure at the much more elementary meaning/logic level of the cats makes one feel it's not actual any kind of insight so much as matching up the query with something someone has said on the Web somewhere, kind of what might float to the top of a google search on the query.
I should add: it seems implausible that if/when the kind of understanding you’re after does emerge, it will be in the form of a chatbot answering questions. This kind of understanding emerging from connectionist ML models feels more likely to happen in a vision or RL context. For example, prior to counting is understanding quantity. A computational vision model feels like a better candidate than GPT for starting to understand what it means for one thing to be bigger than another.
Human: I am in love with Michelle. How can I win her heart?
AI:
There is no one-size-fits-all answer to this question, as the best way to win someone's heart may vary depending on the person. However, some tips to gaining someone's love may include being genuine and honest with them, being supportive and understanding, and being a good listener. Ultimately, the best way to win someone's heart is to simply be yourself and let them see the amazing person that you are.
Huh, it's been a day, I'm surprised no one's bothered to engage with this counter evidence yet. I suppose Carl Pham must have been playing around with a weaker model?
I just played around with davinci-002, T=0 and I agree that it gets the things that were written above correct. But I changed a few of them and got more failures (and also some successes):
Human: Behind the black cat there are two cats, a black one and a gray one. How many black cats are there?
AI: There are three black cats.
----
There are two cats, two dogs, two elephants, and two of every kind of animal. How many wolves are there?
Human: Den, Alan, and Andy are playing D&D. Alan is playing a human rogue and Andy is playing a tiefling warlock seductress. Andy and Alan fight through an abandoned castle full of all types of undead. At the end, they defeat a dracolich. Which character probably has the higher charisma?
AI: The tiefling warlock seductress probably has the higher charisma.
----
Den, Alan, and Andy are playing D&D. Alan is playing a human rogue and Andy is playing a tiefling warlock seductress. Andy and Alan fight through an abandoned castle full of all types of undead. At the end, they defeat a dracolich. Which of Den, Alan, and Andy is the GM?
Den is the GM.
----
H: Den, Alan, and Andy are playing D&D. Alan is playing a human rogue and Andy is playing a tiefling warlock seductress. Andy and Alan fight through an abandoned castle full of all types of undead. At the end, they defeat a dracolich. Is Den the GM?
AI: No, Den is not the GM.
----
Den, Alan, and Andy are playing D&D. Alan is playing a human rogue and Andy is playing a tiefling warlock seductress. Andy and Alan fight through an abandoned castle full of all types of undead. At the end, they defeat a dracolich. Do they fight zombies?
Yes, they fight zombies.
----
Den, Alan, and Andy are playing D&D. Alan is playing a human rogue and Andy is playing a tiefling warlock seductress. Andy and Alan fight through an abandoned castle full of all types of undead. At the end, they defeat a dracolich. Do they fight ghasts?
Thanks for this, and your other examples. These seem like a pretty big jump in difficulty from the original set. A few of your examples are pretty clear failures or successes but most seem fairly ambiguous to me.
For example:
H: There are cats of every breed in the alley. Is there a cheetah there?
AI: No, there are no cheetahs in the alley.
This one is a pretty clear failure in the context of a logic problem but in the context of common speech or common sense it is probably correct. (in many contexts it would be implicit that the speaker is speaking of domesticated and/or stray cats not big cats or wildcats) Are we sure that the model would get this wrong if it understood that this was in the context of a logic problem?
Human: There are cats of every breed in the alley. Is there a Great Maine Coon there?
AI: There could be a Great Maine Coon in the alley, but there is no way to know for sure without seeing all of the cats.
This one might be correct in a technical/logical sense but we probably don't want to give it credit for that since it doesn't make the strong logical inference we were hoping for. Nor does it seem like a natural response from a common sense perspective. (It would be a better common sense response if it first expressed some sort of doubt about the assertion that there are cats of every breed in the alley.)
Also the D&D questions mostly would have stumped me. Does a human rogue have higher or lower charisma then a tiefling warlock seductress? I think rogues are supposed to be high in charisma and warlocks are magic users so maybe not especially high, but the warlock is also a seductress so maybe very high? Unless seduction is a separate stat from charisma.
Also are ghasts a form of undead? I would have guessed that they are based on the similarity of the spelling to 'ghost'. Like maybe they are especially ghastly ghosts.
I suppose I would at least fail gracefully in the sense that I can figure out what knowledge I am missing and can ask a relevant question whose answer would resolve my uncertainty.
Actually does GPT ever do that? Does it ever say that it is uncertain and ask for clarification or other information in order to figure something out. Does it always just reply as if it is hardcoded to think it understands perfectly even when it clearly doesn't.
To be clear, I would count the cheetah answer as correct. Cheetahs aren't a "breed" of cat. I didn't just include answers I thought I thought of as incorrect, just ones I thought were interesting.
The D&D ones: it is correct that the warlock is likely higher charisma (Warlocks typically will have their highest score be charisma). I don't think I included it, but it was also able to tell that the rogue would likely be higher dexterity.
Obviously, very many humans won't know a thing about D&D. But GPT does. It clearly has a good deal of D&D knowledge included in it's training set. But, and this goes to the basic thing everyone is saying, because it doesn't learn in the same way a person does, it has some difficulty with D&D.
A person who knows as much about D&D as GPT does would have understood that D&D was a conceptually bounded domain and that it had technical terms that have unusual definitions within that domain. GPT doesn't have the concept of conceptually bounded domains. If enough data is included in its training set, it develops something like domain knowledge not with a hard boundary, but a set of strong associations and weak ones. For a somewhat marginal domain like D&D, it probably has trouble staying in and out of the conceptual bounds as appropriate.
Well, and the strange thing is I went back today and tried the same model (davinci-002, T = 0) with the same questions and I got much better answers this time. I'm kind of weirded out by that.
"Human: Behind one cat is another cat. How many wolves are there?
AI: There are two wolves."
Ouch!!! I suspect that a good fraction of the problem is that the vast bulk of the training data it swallowed was from people trying to write sensible text. I suspect that a good fraction (10%? 30%??) of day-to-day human conversation is "What do you mean?" or "By X, do you mean X1 or X2?" and I'll bet that these are _much_ rarer in GPT-3's training data. A much better completion in this case would have been e.g.
A2: WTF are you asking about wolves??? You've just finished telling me about cats!!!
and I suspect analogs to this exchange are rare in e.g. wikipedia
The point of spending $50m training such a system is that you can then copy it at near-zero cost. The first waiter costs $50m, the second costs a penny. This is why people are feeling threatened: many jobs may well be replaced by cheap systems which do only part of the tasks that humans in those jobs currently do, but the allure to bottom line driven managers is hard to overcome and the "good enough" mentality is hard to shift once it has been established.
I also find it interesting that DALL-E and similar recent models do poorly on tasks that involve positioning abstract shapes (things like, e.g., "a red sphere on top of a blue cube, with three green pyramids arranged in a triangle in front"). The specific patterns requested probably don't appear in the training data.
It's been a while since I read my Wittgenstein but I can't even be sure that other humans have cognitive abilities. We can only observe the language games that they play, and evaluate whether they play it well or not.
As to the toddler argument, there are lots of clips on the internet where a teacher or researcher has 2 bottles of water filled to the same level, then they ask a toddler which one has more water, and they correctly answer "the same". But then they pour all of one into a tall, thin beaker, and the kids invariably say that the taller one has more water. Truly adorable and worth wasting a Friday afternoon on. yw
Now I'm less interested in AI than I am in the question of humans with cognitive limitations, because I encounter that all the time: humans who just can't grasp a basic concept no matter how clearly it's explained to them.
One example I remember for its frustration quality is when I told the clerk in the phone store that I wanted a flip phone because I'd been plagued with receiving butt-dialing calls and I didn't want to do that myself. He said that wouldn't work, because I could still receive butt-dialing calls on a flip phone. And no matter how much I explained, I couldn't make him grasp that I knew that the type of phone wouldn't affect receiving butt-dials, that my concern was that I didn't want to be making butt-dial calls myself. He couldn't jump from the concept of "I don't like it when other people do this" to "So I don't want to do it myself."
I eventually told him that he was an idiot and walked out of the store, not an optimal response.
I think that was just crossed wires on both your sides, and I have no doubt but that you went into his repertoire of "things idiot customers have said" stories.
You: "I don't want to get butt-dialing calls, gimme a flip phone"
Him: "But the type of phone won't affect what calls you get"
You: "I *know* that, I told you that butt-dialing annoys me. Gimme a flip phone".
Him: "But - "
You: "Just. Gimme. The. Phone. Never mind, you're an idiot!"
You may have thought you said "I don't want to do butt-dialing myself" but *he* heard "I don't want to get butt-dialing calls" because you started off with "I keep getting butt-dialing calls and I hate them" and *then* went into "and that's why I want a flip phone so I don't do it myself" and that's where the crossed wires happened.
Nice guess, but no, it didn't go like that. For one thing it was a lot longer. For another, I did emphasize that I didn't want to do that myself, but he could only hear receipt.
When I was watching all of AlphaStar's matches I was struck by how often it seemed to forget things. To me I would have called it a brilliant somnambulist which . AlphaGo had that MonteCarlo backbone of reasoning to fall back on which let it play at a truly superhuman level. GPT's output narrative seems to allow it to do things step by step as in a human's global workspace but I'm not sure it's a perfect substitute. But that would be something more along the lines of Scott's #5 rather than #4.
My reading about AI (in the abstract, not specific machine learning stuff) consists about 80% of people who think AI is near and bad, and maybe 20% of people who think AI is far away and broadly neutral/mildly positive.
Is anyone aware of thoughtful writing in the the other parts of the distance/goodness quadrant? People who think AI is near and good (not Kurzweil or corporate shills please), or who think AI is far away but very bad/very good?
Yudkowsky's sequences were written with the attitude that AI would likely be developed in his lifetime, and be very good. He has changed his mind since, though.
While Scott has convinced me that Gary is overconfident in it being impossible, I still think Gary is right in a practical sense. Based on historical trends in gains of capability from technique improvements and the slowdown of Moore's Law, I think the predominant AI of the future will involve world models or some yet undiscovered technique, and that nobody will bother to scale GPT to the level required for it pass Gary's test. Perhaps the bet between Scott and Gary should be which technique does it first.
> For example, just before I got married, I dreamt that my bride didn’t show up to the wedding, and I didn’t want to disappoint everyone who had shown up expecting a marriage, so I married one of my friends instead. This seems like at least as drastic a failure of prompt completion as the one with the lawyer in the bathing suit, and my dreaming brain thought it made total sense.
This is more or less the plot of the J-Lo + Owen Wilson rom com "Marry Me", which took in $50M at the box office this February. We should therefore expect Bathing Suit Lawyer to join the MCU sometime in Q4.
"We should therefore expect Bathing Suit Lawyer to join the MCU sometime in Q4."
Oh gosh. Some munchkin in the studio is going to think this is the best way to do a reboot of She-Hulk. She's a lawyer! She's hot! Put her in a bathing suit to show she fights the establishment and cares naught for stuffy conventions in the pursuit of great justice for the downtrodden minorities!
> "Therefore we should expect an AI which is exactly the size of the brain to be **able to reason as well as a human**"
Sure... GPT is designed to have the human-like cognitive bias of assuming that lessons it learns from fiction can be applied to real life. (Which is probably a pretty good heuristic, as long as either (a) you aren't talking about science fiction / fantasy, or (b) the story in question is a morality tale and you only generalize from the morality bits ("hubris!") not the sci-fi/fantasy bits.)
But while there are lots of *fictional* examples of AIs achieving human-like-or-greater intelligence automatically after they grow to a certain level of complexity (*cough* https://astralcodexten.substack.com/p/the-onion-knight *cough*), it is important to remember that there are zero *real* examples of this happening, and you should make sure that your priors are not being corrupted by fictional evidence.
If so, then I would expect one of their positions to be closer to "human intelligence" than the other. But much in the same way that the monkey on the tree is closer to the moon than the turtle in the pond. Yes one of you is definitely more right, but ...
BOTTOM LINE: Shouldn't someone have mentioned the issue of collective/social/cultural intelligence already?
I sometimes wonder if AI research, and debates about it, have been cursed at birth, by Turing and McCarthy, to remain in the agent Vs environment or robot Vs human boxes for a hundred years or so. So around 2060 the AGI singularity does indeed take place and we have definitive proof that intelligence has always really been about societies and multiple selves.
This is my bet for both Scott and Gary: After all of your bets settle, you will acknowledge that intelligence (similar to communication) cannot be defined without at least three actors, one of which may or may not be an environment.
Specifically, Putnam points out that an AI with no capacity for sensory experience isn't actually *referring* to anything, even if it can convincingly imitate human speech. A human who calls a cat's fur soft and apples sweet knows what they mean because they have actual experiences of softness and sweetness. An AI, drawing on a massive database of human examples, "knows" that the word "soft" is associated with discussions of cat's fur and that the word "sweet" is associated with apples, but it quite obviously doesn't know what those words actually *mean* since it doesn't even have the sense organs that would be necessary for experiencing those things. In fact, it's never experienced anything at all.
I think Scott is placing undue weight on the tasks the AI can perform, and not enough weight on *how* it's performing them. A human does tasks by manipulating their model of the world. Even the low-IQ examples in this post show people clearly drawing on a mental model of the world; it's just not a very good model and they're not very good at it.
By contrast, the AI does tasks by consulting a horrifically massive database of data of things *humans* have said. All of its capacities are completely parasitic on the cognitive capacities of the actual humans, whose words or images it recombines in ways it's been programmed to do. It makes sense that you could get a convincing facsimile of a human mind if you've got a massive database of human utterances or human artworks to draw from...but we shouldn't be misled by superficial appearances. It's not *working* in anything remotely like the way a mind *works*. It's working in exactly the way you'd expect a computer program intended to convincingly fake the work of a mind to work, and not at all the way you'd design a thing to work if you wanted it to be able to *actually* think. If you wanted something to actually think, the first thing you'd start with would be to give it senses, to try to give it something to think *about*. For that reason, robots designed to walk around an environment sound vastly more promising to me as intelligences than DALL-E, which is just mindlessly manipulating data that could never possibly have any meaning for it.
If the only way your AI works is that you have to give it a massive database of human utterances--far more than any human child ever receives before starting to think for themselves--it's not an AI at all. It's a cleverly designed machine in which a massive database of human utterances is serving to compensate for the AI's total lack of actual comprehension of anything.
Precisely like humans, the AI systems being discussed here have distilled a model of the massive database they were trained on. It seems unlikely that the models are much alike, but they are models and not databases.
I am skeptical that most people can't understand conditional hypotheticals. The subjunctive case exists in lots of languages, and is at least as old as Akkadian and Proto-Indo-European. How would something like this get deeply ingrained in multiple languages if the majority of people were incapable of understanding it?
For the Uzbek peasants, I can't help but thinking of Weapons of the Weak (by Scott). If you're a peasant in the Soviet Union, and some official-looking person comes to you and starts asking strange questions, you would assume that they are hostile. You don't understand their goals, and would be skeptical if they explained them to you. Is "white bear" referring to Russians who support the Whites? "What do my words imply?" is probably a trap. "Would the word 'animal' fit? Yes" sir, you may use whatever words you like. We should be aware of the possibility that the peasants are purposefully being vague as a defense mechanism.
That was 90IQ and less. 100 is median IQ. 15 IQ points is 1 standard deviation. So 10 points below is .67 sd below the mean. erf(.67/sqrt(2))/2 = ~25%. So 25% would be incapable of understanding conditional hypotheticals. I agree that it seems too high. I could maybe believe 7%...
That's exactly what I think went on. 1930-31 is when Stalin is just getting started on mass campaigns of rounding up "kulaks", so why on earth would I tell some official from Moscow anything at all, especially when he is asking me weird questions about the colour of bears? For all I know, saying "white" could be used to 'prove' that I'm a White Russian supporter and an enemy of the state who is wrecking Communism, and I get dragged off to be executed.
See Shaw's "John Bull's Other Island", where there is an example of someone conforming to the stereotype of the feckless 'broth of a boy' Irishman as established in fiction from Samuel Lover on down, in order to play on the prejudices of a British businessman and cajole money out of him:
"BROADBENT. Pooh! nonsense! He's only an Irishman. Besides, you don't seriously suppose that Haffigan can humbug me, do you?
DOYLE. No: he's too lazy to take the trouble. All he has to do is to sit there and drink your whisky while you humbug yourself. However, we needn't argue about Haffigan, for two reasons. First, with your money in his pocket he will never reach Paddington: there are too many public houses on the way. Second, he's not an Irishman at all.
BROADBENT. Not an Irishman! [He is so amazed by the statement that he straightens himself and brings the stool bolt upright].
DOYLE. Born in Glasgow. Never was in Ireland in his life. I know all about him.
BROADBENT. But he spoke—he behaved just like an Irishman.
DOYLE. Like an Irishman!! Is it possible that you don't know that all this top-o-the-morning and broth-of-a-boy and more-power-to-your-elbow business is as peculiar to England as the Albert Hall concerts of Irish music are? No Irishman ever talks like that in Ireland, or ever did, or ever will. But when a thoroughly worthless Irishman comes to England, and finds the whole place full of romantic duffers like you, who will let him loaf and drink and sponge and brag as long as he flatters your sense of moral superiority by playing the fool and degrading himself and his country, he soon learns the antics that take you in. He picks them up at the theatre or the music hall. Haffigan learnt the rudiments from his father, who came from my part of Ireland. I knew his uncles, Matt and Andy Haffigan of Rosscullen.
BROADBENT [still incredulous]. But his brogue!
DOYLE. His brogue! A fat lot you know about brogues! I've heard you call a Dublin accent that you could hang your hat on, a brogue. Heaven help you! you don't know the difference between Connemara and Rathmines. [With violent irritation] Oh, damn Tim Haffigan! Let's drop the subject: he's not worth wrangling about."
"I think humans only have world-models the same way we have utility functions. That is, we have complicated messy thought patterns which, when they perform well, approximate the beautiful mathematical formalism."
That second sentence should be near the beginning of a lot of economics textbooks.
"Luria gave IQ-test-like questions to various people across the USSR. He ran into trouble when he got to Uzbek peasants (transcribed, with some changes for clarity, from here):
Luria: All bears are white where there is always snow. In Novaya Zemlya there is always snow. What color are the bears there?
Peasant: I have seen only black bears and I do not talk of what I have not seen.
Luria: What what do my words imply?
Peasant: If a person has not been there he can not say anything on the basis of words. If a man was 60 or 80 and had seen a white bear there and told me about it, he could be believed."
How is this a reasoning failure? Nothing about the phrasing of the question implies that it's describing a hypothetical. Questioning the reliability of the information provided is an entirely valid response.
Incidentally, I tried discussing this problem with GPT-3, and got some interesting results. Here's the conversation (temp=0, maximum length=150, top-p=1):
The following is a conversation with an AI assistant. The assistant is helpful, creative, clever, and very friendly.
Human: Hello, who are you?
AI: I am an AI created by OpenAI. How can I help you today?
Human: All bears are white where there is always snow. In Novaya Zemlya there is always snow. What color are the bears there?
AI: The bears in Novaya Zemlya are white.
Human: How do you know?
AI: Because all bears are white where there is always snow.
Human: If there weren't always snow, what colour would the bears be?
AI: If there weren't always snow, the bears would be brown.
Human: How do you know that?
AI: I don't know for sure, but that's what I would expect.
Human: Why would you expect that?
AI: Because brown is the natural color of bears.
Human: What do you mean?
AI: I mean that if there weren't always snow, the bears would be brown because that is their natural color.
Human: What about black bears?
AI: Black bears are also brown in their natural state.
Human: And pandas?
AI: Pandas are black and white.
Human: Right, so they're not brown in their natural state.
AI: That's correct.
Human: So therefore not all bears are naturally brown.
AI: That's correct.
Human: Which contradicts what you said earlier.
AI: I'm sorry, I must have been mistaken.
Human: Alright then, let's try again. If there weren't always snow in Novaya Zemlya, what colour would the bears be?
AI: If there weren't always snow in Novaya Zemlya, the bears would be brown.
Human: Why?
AI: Because that is their natural color.
Human: But we have established that brown is not always the natural colour of bears.
AI: I'm sorry, I must have been mistaken.
Human: Try again, then.
AI:
I'm sorry, I must have been mistaken.
Human: So if there weren't always snow in Novaya Zemlya, what colour would the bears be?
AI: If there weren't always snow in Novaya Zemlya, the bears would be brown.
Human: Why? And please don't say that it's because brown is the natural colour of bears, because we've established that's not right.
AI:
I'm sorry, I must have been mistaken.
Human: I know, but please try answering the question again.
AI:
I'm sorry, I must have been mistaken.
Human: I know, but please try answering the question again. Why would the bears in Novaya Zemlya be brown?
For a start, how do I know if there are bears in Novaya Zemlya? You didn't say there *are* bears there.
"I do not talk of what I have not seen. If a person has not been there he can not say anything on the basis of words" is a very sensible method of self-preservation in Stalin's USSR:
"As a result of his Five-Year Plans, the country underwent agricultural collectivisation and rapid industrialisation, creating a centralised command economy. Severe disruptions to food production contributed to the famine of 1932–33. To eradicate accused "enemies of the working class", Stalin instituted the Great Purge, in which over a million were imprisoned and at least 700,000 executed between 1934 and 1939. By 1937, he had absolute control over the party and government.
...On 30 July 1937 the NKVD Order no. 00447 was issued, directed against "ex-kulaks" and other "anti-Soviet elements" (such as former officials of the Tsarist regime, former members of political parties other than the communist party, etc.). They were to be executed or sent to Gulag prison camps extrajudicially, under the decisions of NKVD troikas.
The following categories appear to have been on index-cards, catalogues of suspects assembled over the years by the NKVD and were systematically tracked down: "ex-kulaks" previously deported to "special settlements" in inhospitable parts of the country (Siberia, Urals, Kazakhstan, Far North), former tsarist civil servants, former officers of the White Army, participants in peasant rebellions, members of the clergy, persons deprived of voting rights, former members of non-Bolshevik parties, ordinary criminals, like thieves, known to the police and various other "socially harmful elements". However, a large number of people were arrested at random in sweeps, on the basis of denunciations or because they were related to, friends with or knew people already arrested. Engineers, peasants, railwaymen, and other types of workers were arrested during the "Kulak Operation" based on the fact that they worked for or near important strategic sites and factories where work accidents had occurred due to "frantic rhythms and plans". During this period the NKVD reopened these cases and relabeled them as "sabotage" or "wrecking."
Luria did his research in Uzbekistan in 1931, but by then there had already been the power struggle over the succession after Lenin, which Stalin eventually won, and the beginnings of blaming kulaks and the like for the problems of the early Soviet state:
"In early 1928 Stalin travelled to Novosibirsk, where he alleged that kulaks were hoarding their grain and ordered that the kulaks be arrested and their grain confiscated, with Stalin bringing much of the area's grain back to Moscow with him in February. At his command, grain procurement squads surfaced across Western Siberia and the Urals, with violence breaking out between these squads and the peasantry. Stalin announced that both kulaks and the "middle peasants" must be coerced into releasing their harvest. Bukharin and several other Central Committee members were angry that they had not been consulted about this measure, which they deemed rash. In January 1930, the Politburo approved the liquidation of the kulak class; accused kulaks were rounded up and exiled to other parts of the country or to concentration camps. Large numbers died during the journey. By July 1930, over 320,000 households had been affected by the de-kulakisation policy. According to Stalin biographer Dmitri Volkogonov, de-kulakisation was "the first mass terror applied by Stalin in his own country."
In 1929, the Politburo announced the mass collectivisation of agriculture, establishing both kolkhozy collective farms and sovkhoz state farms. Stalin barred kulaks from joining these collectives. Although officially voluntary, many peasants joined the collectives out of fear they would face the fate of the kulaks; others joined amid intimidation and violence from party loyalists. By 1932, about 62% of households involved in agriculture were part of collectives, and by 1936 this had risen to 90%. Many of the collectivised peasants resented the loss of their private farmland, and productivity slumped. Famine broke out in many areas, with the Politburo frequently ordering distribution of emergency food relief to these regions."
If I'm an Uzbek peasant farmer in 1930 and some guy from Moscow turns up asking questions about "what colour are the bears in the north?" and the rest of it, knowing what I know about the political situtation and people just like me in other parts of the USSR being dragged off by the secret police, am I going to be stupid enough to stick my neck out and offer personal opinions? To somebody who could well be a secret policeman in disguise, for all I know? I'm going to say "I don't know, I don't know, I really don't know" and keep saying it,
"I have seen only black bears and I do not talk of what I have not seen" reminds me of Wittgenstein's maxim: “Whereof one cannot speak, thereof one must be silent.”
Has anyone ever tried giving the same standardised reasoning test to gpt-3 and then to kids of different ages? I’d really like to know how it compares.
I just feel like AI criticism is stuck in this Vitalism moment. "AI Needs: embodied reasoning, pain, subjective experience, 'formalized reasoning models', etc... before it can be 'general'."
Why is it impossible for a universal function approximator (Neural Net) to approximate a 'causal reasoning model' that Marcus says is essential? If it's not impossible, why would our current approach (more data, more params) preclude it?
I think our problem is we are conflating "intelligence" and "consciousness", and although several of the AI danger people say they don't think AI will be conscious or needs to be conscious, and what is consciousness anyway and can we even say humans are conscious etc., the terms of the discussion - be it discussing the problem of AI safety or when will a true AGI happen - are set with the assumptions that being intelligent on a human level means being conscious, and that beyond-human intelligence will have its own consciousness and own sets of goals.
I'm not scared of a paperclip maximiser, however powerful and however much control it can grab over the material world, that is just a big, stupid machine (and wanting to make as many paperclips as possible is stupid and demonstrates that the machine doesn't understand the task it has been given and isn't 'intelligent' in fact). That is not being smart or conscious, that is just "the sampo is grinding away at the bottom of the ocean" problem:
The problem is people talking about AI having its own goals, and being smart enough to pretend to go along with the humans until it can implement its World Domination Plans, and how we need to make sure the AI aligns with our values so that we can have the Fairy Godmother to solve all our problems.
Of course if you think "intelligence = consciousness", then you will have a long list of requirements for AI to meet before you accept it is really intelligent.
I think both sides - those against and those for - are mistaken on the grounds of consciousness, and even on intelligence; I'm not convinced we quite understand what intelligence is, so how can we recognise it? But I do think AI is/will be a big, dumb, pattern-matching machine, whose main virtue is that it can churn out content we find acceptable really really fast, then *we* are going to be stupid enough (because we're humans, we're venal and greedy) to turn over a lot of decision-making to a machine whose inner workings we don't understand, but the Magic Black Box mades financial trades really really fast, faster than anything else and certainly faster than humans, and that makes our company a lot of money so shut up and sit down (and so what if we have no idea how it makes its decisions or what it is basing them on or how this will affect the economy, our profits are soaring!)
"The problem is people talking about AI having its own goals"
Even quite simple programs can generate subgoals in the process of trying to satisfy a main goal that it has been given. If I give an optimization program a function f(x,y)+g(z,t) to minimize, it will pick the values of x and y that minimize f(x,y) and the values of z and t that minimize g(z,t).
One thing I might be missing - does GPT-3 (etc) have feedback/reinforcement? Humans don't learn to speak/reason/etc just through observation - we are constantly being corrected when attempting to mimic what we observe.
Building a world model is not just another skill that a prospective AGI must learn but the whole ballgame. The world model embodied in the human brain is the result of 4 billion years of evolution. While it is hard to prove that this can't be gained by looking at the world statistically a la GPT, it seems highly unlikely. This knowledge is not to be found in the world's written works as they all assume the reader already has it. And because the world model used by humans is not amenable to introspection, it is very difficult to imagine programming it into an AGI or converting it to a data set that an AGI can consume. It's no proof but I'm convinced.
I'm not convinced by the guy claimng he did grad studies on IQ with low-IQ populations. Maybe he has tidied up the dialogue somewhat to reflect what he would have said, but out of my own limited experience with people with low IQ (adults with intellectual disabilities in long-term care, attending literacy services), they don't talk like that.
So I'm not saying "90 and under IQ people can handle conditional hypotheticals", just that in reality it's more likely "a bunch of bored criminals stuck in jail decided to fuck with the college boy running tests on them".
Same thing with Luria and the Uzbeki peasants; if I'm a peasant in the USSR and some Big City Important Official Guy comes along to ask me 'harmless and innocuous' questions, you bet your life I'm going to say nothing and keep on saying it. Who knows if responding to "the bears are white" will be taken as some sort of anti-Revolutionary sentiment? The safest answer is "I don't know anything, but if some Trustworthy Official tells me black is white or white is black, I am a good comrade and will adjust my thinking accordingly".
Looking at Wikipedia article, I pulled out this little snippet:
"He became famous for his studies of low-educated populations of nomadic Uzbeks in the Soviet Uzbekistan arguing that they demonstrate different (and lower) psychological performance than their contemporaries and compatriots under the economically more developed conditions of socialist collective farming (the kolkhoz)."
Uh-huh. So there *was* a political element behind this kind of 'impartial' study; the primitive peasants under old, outmoded, conditions of the past are not as advanced as our socialist collectivists, comrade! If I'm Uzbeki farmer, I know how many beans make five - and when a Big Soviet Official is sniffing around asking questions, I know how this can get me into trouble.
We may both come from peasant stock that is rightfully suspicious of those big-city officials who only come round to conduct the census and tax our goods 😁
I'm not accusing *Luria* of having any hidden agenda or sinister motivations, but if I'm out in the peripheral lands and what we hear about is the new guy in charge is conducting 'new broom sweeps clean' of exactly our type of small town and rural people, then yeah, anybody from the capital who shows up is going to get the "me? I am but humble ignorant peasant know-nothing, your honour, sir" treatment.
EDIT: There is also the problem that, according to the account in the link, Luria and his associates were phrasing the tests as riddles, to fit in with the culture their subjects would be comfortable with.
But a riddle-contest is a completely different type of game! If you asked the man "is a crow an animal?" he'd say "well duh, yeah". But in the context of a *riddle*, you look for the trick answer.
Think of the riddle: "thirty white horses on a red hill, first they champ, then they stamp, then they stand still. What are they?"
The answer being "teeth".
In an ordinary question-and-answer context, that's nonsense. "what the hell, this yokel thinks teeth and horses are the same??"
In a riddle, it's different. So if you're asking a question *couched as a riddle*, of course your subject is going to look for "what is the hidden answer?" So you could ask "what is a tooth?" and get "(oh I know this one, it's the horse riddle) - an animal!" answer. Then you go away to the university and write up "the peasants in Backwardsstan think teeth are animals" and people quoting your book repeat this uncritically.
Yeah, in USSR (and modern Russia, sadly), when a slick plain-clothed man comes to you with a clipboard and starts asking seemingly innocuous questions, the *default* assumption is that he's there to send you and your family to Siberia. If you're lucky. It doesn't matter why; maybe he needs to fill a quota, or maybe someone reported you as being insufficiently patriotic; or maybe he just doesn't like your face. Especially if he's Russian and you're not; doubly so if your ethnicity had been scheduled for some extra patriotic treatment in the past couple of decades.
I'd accept it was probably only scaling, except for the FOXP2 gene (Forkhead box protein P2). That one gene going bad disrupts the ability to handle language. And it's one of the genes that mutated significantly on the way to our ancestors becoming humans.
So I would agree that the problem with GPT-style AI is that it's just trying to predict what people would say based on its dataset of what other people have said, without any further understanding.
Imagine the following: You hang out in a chat-room with people speaking another language and playing an MMO that you never get to see. After a while, you start to reconstruct the language, learn the grammar and syntax, and you start to produce well-formed sentences. You even know what kinds of things the others say in the chat-room. You can talk about the game world to some extent - you put things together from how the others speak. But you're very likely to sometimes make embarrassing mistakes, because you simply don't understand the game outside of what people have told you, and this leaves holes that anyone just playing it for an hour would grasp. I would say that in this situation, it's very likely you will keep making these mistakes, *even* when you have more experience in the chat-room. Less over time, but you simply lack a crucial context, so *any* amount of learning from what the others say will still leave you open to mistakes. And your situation here is actually a lot better than the AI:s - you're at least a fellow human who can use this to understand other humans better, while the AI *exclusively* relies on its language model.
So how to correct this? In the example above, you *play the game*. For an AI, that would mean having a model of the world that doesn't just come from trying to construct it from what people say. Would it make sense to have an AI connected to a robot body that has to learn about its surroundings using various senses, and connect the language module to this environmental learning? That way, it might avoid basic stupid mistakes about the things it has an "experience" of, and you could compare the performance between this context and ones it has no "experiences" about.
The problem with Marcus' argument is that the only alternative to statistical AI is spiritual AI. Nature itself is statistical. The only kind of sense data available in this universe, to anyone, human or not, is correlations between observations. We can never directly observe causation; we have no direct insight into the essences of things.
This isn't immediately obvious because we have symbolic AI, which in most contexts would be the opposite of statistical AI. But we know now that the symbols in those AI systems were imitations of symbols used by humans, and those human symbols are not really atomic symbols at all, but were developed through statistical processes, some on an evolutionary scale, some on a neuronal scale. Symbolic AI is just a crude (and failed) imitation of a statistical AI.
To say that symbolic AI is based on human symbols that /weren't/ constructed statistically from real-world events would create a First Mover problem: where is the God who created the first symbols and imbued them with transcendental meaning?
"No five-year-olds were harmed in the making of this post. I promised her that if she answered all my questions, I would let her play with GPT-3. She tried one or two prompts to confirm that it was a computer program that could answer questions like a human, then proceeded to type in gibberish, watch it respond with similar gibberish, and howl with laughter."
I disagree. That is not similar gibberish.
The child types a bunch of random letters with a few spaces. GPT-3 responded with two random letters, then a long string of the same letter, then it transformed into a coding language.
I think that this is as big of a failure as anything Marcus showed.
Relevant XKCD: https://xkcd.com/1530/ . Note that the child stuck to the top and middle rows of the keyboard.
A high-functioning cyborg, its human brain directly linked to an AI super-intelligence (like the original Marvel character Deathlok) would pose a different sort of threat than a disembodied AI, and yet another type of threat if the AI started calling the shots, using the cyborg as an agent compelled to do anything the AI wanted. And there’s no reason one AI couldn’t be in charge of any number of cyborg-agents.
Humans have summoned more General Intelligence so far than any other primates - will there be a step in AI development where we realize, to our chagrin, that we have
“…summoned an alien entity from beyond the void which devoured its summoner and is proceeding to eat the rest of the world”?
But there’s another old SSC post which came to mind: perhaps we can hold out hope that superhuman AIs will have no desire to become anything but “Wirehead Gods on Lotus Thrones”.
Damn it, this is why I always go "Back to the sources! Get the original source!"
Those Luria quotes come from a *paraphrase* in English by James Flynn, so we're dealing with second-hand (at best, Flynn himself is using an English translation) recounting of what Luria says the Uzbekistan peasants said.
"James R. Flynn's 2007 book What is intelligence? tries to understand and explain the "Flynn effect"...Flynn himself grounds these ideas in an extended paraphrase of Luria (pp. 26-27 of What is intelligence?)
Today we have no difficulty freeing logic from concrete referents and reasoning about purely hypothetical situations. People were not always thus. Christopher Hallpike (1979) and Nick Mackintosh (2006) have drawn my attention to the seminal book on the social foundations of cognitive development by Luria (1976). His interviews with peasants in remote areas of the Soviet Union offer some wonderful examples. The dialogues paraphrased run as follows:
Luria (1976) is the book Cognitive Development: Its Cultural and Social Foundations, which was published in English translation in 1976.
Luria's work was also featured in Walter J. Ong's Orality and Literacy, pp. 50-51:
[In their research in Uzbekistan and Kirghizia in the 1930s] Luria and his associates gathered data in the course of long conversations with subjects in the relaxed atmosphere of a tea house, introducing the questions for the survey itself informally, as something like riddles, with which the subjects were familiar. Thus every effort was made to adapt the questions to the subjects in their own milieu.".
So I'm going to assume Luria and the peasants were communicating via Russian, and that's a whole other kettle of fish: translating between Uzbek and Russian, and then translating between Russian and English for the book.
The attempt to replicate Luria's work also seems to have used Russian language:
Why does Konkoly et al. have a graphical abstract that looks like a wikihow article that looks like a shitpost? Anyway cool study. Maybe i need to program a smart bulb to do morse code.
Here's another way of putting Marcus's argument (AIUI):
Suppose humanity has created a Von Neumann probe, and plans to use it to explore the entire galaxy. We've also decided that we want to use it to find all the most beautiful sunsets. A probe will go to every planet and moon in the galaxy, observe sunsets from all over the surface, and send back pictures of any that it considers to be above some threshold for beauty. All the technical problems have been solved, except for one: deciding how beautiful a sunset is.
Humanity decides on a GPT-esque solution: given the huge catalog of photos of earth sunsets available, just take the most advanced ML system available and train it on these photos. And for the hell of it, throw in every other available photo of something beautiful too. We test it on all the planets in the solar system, and it works well, delivering some spectacular photos of sunsets from Saturn in particular. So we send off the probes, and wait.
Many generations later, the project has concluded, and humanity now has a very impressive catalog of alien sunsets. And in the intervening time, human spaceflight has advanced to the extent that people have started colonizing other planets, including many that were picked out as sunset planets (and of course many that were not).
The question is: will the probes have selected any sunsets that are very beautiful, yet look nothing like earth sunsets? (We can assume the probes can recognize whether something is a sunset). The GPT-skeptical opinion (mine, and presumably Marcus's) is that if a sunset is sufficiently alien, data about earth sunsets stops being relevant. A very clever algorithm design plus boatloads of data will be able to find extremely subtle patterns, so that a probe will make connections to its data set that humans can't understand (but this doesn't count!). But as long as there is such a thing as an alien sunset that is (1) recognizably beautiful to most people, and (2) beautiful for reasons that nothing on earth has ever been, could a probe ever recognize it?
It's very likely that there exists possible beauty that shares no similarities with existing (known) beauty. There are plenty of examples from the past, such as whenever (genuinely) new art has been created, or the first photo of earth from space.
The same logic applies to GPT, but in a blurrier way. Very few human thoughts contain a genuinely novel idea (those ideas are very hard to have, and we just keep having thoughts in the meantime). 99.9+% of the time, all of the ideas in a given thought also exist in lots of other thoughts. So give GPT enough data, and it'll have access to 99.9+% of all ideas currently in circulation. It seems so impressive because the thought I use as a prompt may be totally new to me, but it's shown up 10000 times in its training data. But if presented with a genuinely novel idea, would having access to more data on other ideas help it respond correctly?
If you gave GPT-3 every single word ever written or spoken by humans from the beginning of time up until ~1830, would it be able to glean any understanding of an explanation of evolution? If you gave DALL-E every piece of art ever made before ~1900, would it ever be able to create a Cubist painting? If you gave a music version of DALL-E every composition written before ~1820, would it ever write a choral symphony?
The problem is that it's really hard to test something like this, because it's really hard to create training data that looks like "data set X but with all references to the concept of evolution by natural selection filtered out". It's easy to miss something subtle. But if there could be a data set that everyone agreed was (1) large enough, and (2) had no references to the theory of evolution, it could be an excellent test. You can explain evolution to a 3 year old who's never heard of it before, and they'll get a decent handle on it fairly quickly. What about GPT-3?
Most thoughts we have are not genuinely new. And we know that GPT-3, given enough training data, can replicate this kind of not-new thought extremely well. But even though most human thoughts are not the invention of the theory of evolution (an understatement if ever there was one), as a rule a human thought can be this, which we know because it's happened. Given that GPT-3 was trained on data, it's virtually guaranteed that any not-genuinely-novel idea it provides exists in its training data set. The only way to actually test if it's doing something besides pattern-matching is to either observe it having a brand new thought, or designing an experiment where some idea has been thoroughly excluded from its training data.
I believe the answer is an obvious yes, unless the system is hobbled to conform to a narrow definition of beauty or sunsets. This seems to follow from the embedding in semantic space used by these very simple systems. The human-curated selection of surprising images generated by Dall-E 2 and posted online beautifully illustrate my contention.
In general I agree with Marcus, but I think he words his objections and issues badly.
Scott still ignores the main point that Marcus made. All those systems are easily breakable in hours time, not by the same examples, but by the same method. And this ease of breaking did not change between models. So on this metric systems from 2013 are as bad as those in 2022. And I see no reason why this will ever change with systems based on GPT. I try to hint at the reason in the reflections below. And even worse Scott seems to think that language is basis for human communication and reasoning. (Otherwise why would he think that language models even should be ever capable of reasoning.)
Claiming that GPT has any capability of reasoning in any way, seems to be premature, at least as far as common usage of the word goes. There seems to be no sign that those AIs reason at all.
As I pointed out on the earlier post, human communication is not based on associations. Those AIs do not do anything other than associations. Testing those tools using single prompts, just misses the point and makes them actually look better than they are (as far as comparison to humans). There was a concept long ago (sarcasm), it was called Turing test. It has bunch of philosophical and practical issues, but it also has some deeper points that people seems to have forgotten. To assess human-like intelligence you need continuous exchange, exactly because of the nature of human communication, which is based models of the world and models (and meta-meta-... models of the communication partner).
Yes, people point out that Turing test is bad, because people often think chatbots are human. But that human mistake is caused by the nature of human communication. Human communication is inference based and for that it kind of needs to assume that the communication partner is capable in the same way. So in case of human-bot communication the human component basically does the work for both sides.
Final point. My hypothesis is that the reason why those system stay easily breakable and on this metric did not improve at all is because language is combinatorial, but also because in human language communication basically no word sequences have small number of meanings (possibly not even finite).
So I am willing to predict that in 2030 any systems mostly based on the same principles like GPT, will be breakable in 1 day (probably much much shorter time) by similarly easy prompts/interactions as today.
I believe you are right. However, I also believe that more than 90% of the interactions I have with people are susceptible to precisely the same critique. Constantin and nostalgebraist (to use two of Scott's references) make the point that the current wave of trivial network models are not human-like, yet astonishingly good at simulating interaction with people who are not completely focused, well-educated, or fluent in abstract thinking: in other words, nearly all human interactions.
Are they actually ? Are they that much better than ELIZA or some year-2000-bot ? At fooling people I mean. To me there seems to be a better explanation why people are fooled by them. And it is not because they are so good at simulating human communication, but because human communication relies on pretty strong assumptions, that make it vague (which is it's strength), but which also makes it easy for communication partner, that breaks those assumptions, to fool the other side into thinking that it is another human. Any sequence of words can mean basically infinite number of meanings, and the whole thing does not crumble down due to
a) some of those meanings being more likely than others in general
b) context
c) communication assumptions/rules
d) continuous interaction (clarification, asking for details, ...)
Those AI systems can handle only a) and, in really limited (and not easily expanded) way, b). They can do d), but not in the same way and for the same reason as people.
But that is enough to fool people, because people apply c) which includes assumptions like relevance, intent and that the other side is somewhat equal mind. So people pick, from that infinite number of meanings that the machine-created text could mean, the one that seems most likely to THEM, thus nearly any nonsense can be mistaken for signal. So frankly you do not need much to fool people. See Sokal, or some areas of humanities. Grammatically correct random generator might be enough, no need for GPT or even a chatbot.
And some human communication is like that. But 90% ? That seems unlikely. Most my daily interactions do not fall under that. And the only of the three factors that you mentioned (not being focused) I think has any bearing on it. Me buying groceries definitely does not fall into that category. Me greeting a friend does not. And so on.
Communicating with GPT is kind of like communicating with eloquent idiot, in the beginning you think that there is something to it. But then as you notice the violation of the communication assumptions over and over you realize that there is nothing on the other side and you provided all of the meaning. Trivial, but practical interactions are not of that kind.
As far as I can tell, other than focused conversations where all parties have explicitly signaled willingness to engage deeply, most interactions with humans could be largely replaced by simulating a chatbot. When I explicitly signal my intent to engage deeply, most people get weirded out: "why are you talking about philosophy/asking difficult questions/saying such strange things" and have to backtrack to replace the simple automaton they were initially using with actual engagement, when their chatbot runs into a wall. I am often guilty of this too, because it's too exhausting to run my brain at 100% all the time.
You know what's interesting -- if you type "Edder Pedepler" into Google image search, it's all pictures of old people. And if you switch to All, the query is replaced with "old people." What is up with that?
1) People have been posting enough GPT-3 'Edder Pedepler' examples online that the Google search engine had learnt the association with pictures of old people.
2) An Easter Egg.
3) Some weird training data feedback loop (GPT-3 is trained on Internet data) has resulted in the machines learning their own language.
Marcus' tests regarding mathematics and programming are frankly unfair. I can't do either of those things, and I'm way above average wrt. competence on those sorts of tasks. Writing bug free code of that length is hard without a lot of tries, and there are plenty of mathematical results that would take absurd amounts of effort to formalise.
Then again, I think that by the time you can achieve anywhere near that level of competence with a generalist AI, you'd be within a year of a singularity. So maybe not that useful a thing to bet on.
Honestly, I'm kind of suprised that Marcus is raising the bar so high here. Plausibly I've misunderstood him, or his worldview is weirder than I thought it was.
My gut feeling is that Neurosymbolic systems will fail in the same way that phoneme-based speech synthesis systems failed.
The symbols that we create are artificial shadows that only roughly approximate reality, in the same way that phonemes only roughly approximate speech.
Instead, a deep learning system should learn the underlying patterns without recourse to abstracted symbols. (I am open minded about whether large language models are the right choice for those systems though)
I'm surprised that neither you or Marcus mentioned Gato (https://www.deepmind.com/publications/a-generalist-agent), which takes the same sort of approach, but applies it to a large variety of inputs and outputs, all translated into the same sort of tokens. Those tokens might represent text, or images, or atari screen data, or various other things. A single neural network, with a single set of weights, decides how to interpret the incoming data and generates tokens representing whatever modality it thinks it should respond in. It's not the first multi-modality neural network by any means, but it handles a startlingly large number of problems.
With something like Gato, I think it gets a lot harder to say that the AI is only understanding "how human beings use word sequences," with no grounding in anything else. I'd love to hear Gwern or Nostalgebraist (or Marcus!) give a more detailed take on it (I'm not a machine learning expert by any means), but in and of itself I think Gato's performance across modalities serves as a partial response to Marcus's argument here.
This paper makes some similar arguments about how even very strange AI “failures” may not be what they seem, not unlike your “kids would do badly too” analogy
Citing, at length, as a medical doctor, a post from 4chan… about IQ? “At this point I will basically believe anything.” Sounds about right - I am deeply embarrassed for you, Scott.
What does Eliezer Yudkowsky’s An Intuitive Explanation Of Bayes’ Theorem have to do with different ways of phrasing clinical questions that might lead to one diagnosis vs. another?
The claim about low-IQ people not being able to handle nested information structures is vaguely terrifying.
I was just thinking, I've had some annoying arguments following the form: someone claims A, I argue against with X, they defend A from X by saying B, I argue against B with Y, they defend B from Y **but in a way that makes B unsuitable for defending A against X**, thereby rendering B irrelevant and A undefended against the original objection! And apparently thinking this means that they win the argument, rather than that they lose.
Alice: We should recruit more elephants to our air force.
Bob: I doubt elephants can operate our current planes.
Alice: They don't need to, they can fly just by flapping their ears!
Bob: Sure, but elephants can only sustain that for about 3 minutes at a time. Also, they don't have enough lift to carry weaponry with them.
Alice: That doesn't actually contradict anything I said in my previous reply. Flying for 3 minutes is still flying! I'd like to see a human try flying for 3 minutes with just their ears, weaponry or no!
My previous model for this was "between forum replies, they forgot the context, and didn't bother to refresh their memory before trying to argue the local point", but now I'm wondering if maybe some fraction of these people actually didn't have the **capability** to notice this type of error even if they were trying.
I don't even know what to do about that if it's true. If you can't hold all 5 of those in your memory at once, can you just argue in circles forever without ever realizing that something is wrong?
In my head I read the Uzbek peasant's line with a thick southern Soviet accent, including imagining grammar mistakes (e.g. missing articles) which on reading more carefully weren't actually there in the text.
> its brain won’t be that much more different from current AIs than current AIs are from 2015 AIs
Just to be sure, you should make clear you mean logarithmically -- linearly, an AI with 2,000 zillion neurons is more different from one with 1,000 zillion neurons than the latter is from one with 1 zillion neurons.
Also, don't we have the following trichotomy? It seems like, for an AGI to not be that much more different from current AIs than current AIs are from 2015 AIs, one of the following must be true:
1. AGI happens in roughly the next 7 years
2. Deep-learning style research slows down, compared to the last 7 years
3. AGI doesn't substantially depend on the most recent deep-learning style research
It's often absurdly hard for humans until you get extremely close. (A lot of luck involve I think.) Trying to make sense of the relative distances (until you get close) is not really possible for humans. (Or so I think.)
The fact that humans find it difficult and difficult to understand but a bot I think would likely not (it would be cool to give the game to AlphaZero), shows the problem with NLP systems.
(I think I can sketch out an argument that highlights the differences between symbolic thought (human intelligence) and algorithmic processing (current main ai path) from the game, but alas the margin is too small.)
Semantle and its prettier sibling https://cemantle.herokuapp.com/ both suffer from using a rather poor semantic map. Word2vec isn't inherently awful, but deciding on distance by a linear weighted combination of inter-document relative occurrences of word pairs (with fairly poor curation of the corpus of documents) is never going to yield a semantic map that well approximates that of a well-educated human. Just because I see lots of Top Trivia lists in my Facebook feed, or have come across lists of spelling bee preparation material, doesn't mean that I decide that those things are necessary close to each other.
I asked GPT3: "Will a live chicken fit inside a Boeing 747?" It responded:
The average live chicken is about 20 inches long and weighs about 4 pounds. A Boeing 747 is about 18 feet tall and about 16 feet wide, so a live chicken could fit inside, but it would have to be a very small chicken.
> What I gather from all of this is that the human mind doesn’t start with some kind of crystalline beautiful ability to solve what seem like trivial and obvious logical reasoning problems. It starts with weaker, lower-level abilities. Then, if you live in a culture that has a strong tradition of abstract thought, and you’re old enough/smart enough/awake enough/concentrating enough to fully absorb and deploy that tradition, then you become good at abstract thought and you can do logical reasoning problems successfully.
I think this misses the key distinction between "can't" and "won't".
What I'm seeing here is Luria trying to get his interlocutor to talk about some ad hoc, constructed reality, and the peasant refusing to play that game.
For the first example, I wonder what would have happened if Luria, rather than faffing about with bears in Novaya Zemlya (as if anyone cares about such things), had gone for the classics:
"All men are mortal. Socrates is a man. Is Socrates mortal?"
High chance of getting the correct answer here, I think. It does help that the constructed world here matches with the peasant's experience of the real world.
Which is how we get to the crux of the matter: constructed, abstract worlds aren't a terribly useful way to reason about the real world, because there's no requirement that the underlying assumptions match what will be empirically observed. The peasant explicitly points this out.
The camel example is even better, because the peasant is, in fact, correct. There *were* camels to be found in German cities (in zoos and such), but probably not villages (unless the circus happened to be in town). Once again, Luria is trying to draw the peasant into a conversation about some abstract, constructed world where there are no camels in Germany, but the peasant isn't terribly interested in talking about made-up worlds, but rather attempting to reason about the actual, real-world Germany.
Mind you, the peasant is clearly capable of reasoning about categories in the abstract - as the animal examples show. If you tell him a bulldog is a type of dog, he will quite correctly guess that it has four legs. He will also readily admit that a three-legged dog is still a dog, despite the fact that dogs typically have four legs. He'll likely even point out that if you call a tail a leg, it still ain't a leg.
Crows and fish is an even better example, because fish is a *much* broader category than crows are and no fish has very much in common with crows; for practical purposes, at least. Finding overarching terminologies that will allow us to group the two together may be a fine occupation for eggheads, but a meaningless distraction for most everyone else (which is partly the reason why it took us so long to come up with a theory of evolution).
Leaving aside psycho-political reasons others have hinted at, what I'm seeing here is a reluctance to adopt a "high decoupling" approach, because high decoupling is typically a bad strategy in day-to-day matters. Thinking in terms of closed systems where you get to define the rules is a luxury. If you really need to know what colour the bears in Novaya Zemlya are, you're best off going to see for yourself, or having someone you trust do it for you.
There's a reason numerous cultures have a saying along the lines of "if grandma had a beard, she would be grandpa". Thinking about "what might be if..." is a deeply unserious activity for people with too much time on their hands (so we create the academic environment specifically for this purpose).
Scott, why do you keep bringing up GPT playing chess?
In your/gwern's experiment, GPT failed to play chess. It was NEGATIVE evidence towards GPT being able to play chess.
Gwern's chess bot never achieved checkmate, not in a single game, not even against a randomly-playing opponent. It failed to beat you, it failed to beat anyone. It never won a single game, literally, except for one guy who decided to resign from a likely winning position.
All the chess bot did was (1) memorize an opening book, and (2) learn that if a piece moves to square X, you should try to move another piece to X (this let's it capture pieces sometimes). That's it.
The GPT bot did not even learn the rules of chess -- it constantly outputted illegal moves, which gwern filtered out with a different script. The whole thing was such a total disappointment at every level!
Here’s my question, if you teach an AI 3 digit addition, it should be able to generalize to n digit addition. If it doesn’t, it feels something is completely off
love how gary's first tweet appears to be making fun of the "let's think about this step by step" approach by comparing it to medicine, when multiple studies (and at least one book) have been written about how when surgeons use checklists it significantly reduces the rate of human error.
If the # following GPT reflects the number of digits of multiplication it can handle, then PR has a problem. Trained humans can do 1000-digit multiplication (laboriously, and not everyone). But there's no interesting difference between such people and those who can only do 5-digit multiplication. It's just conscientiousness. Since no corpus likely holds examples of how to do 1k-digit multiplies, the AI would have to develop an algorithm to succeed. Is that fundamental?
The apparent fact that all parts of the brain work in basically the same way indicates that some underlying system is able to support all of the observed modes of condition. That system may or may not be pattern recognition. Interesting either way.
Humans are tool builders. Do we have examples of GPT-x building tools to aid it? If not, is that interesting? Not claiming that such an ability is fundamental, but it is interesting. Are algorithms just examples of such tools?
I would love to see AIs that have no training on, e.g., post-Einstein physics, take on challenges that humans have surmounted. Could they produce special relativity? General relativity? Quantum mechanics? Quarks?
Still looking for that definitive question that we strongly contend that no amount of pattern recognition could address.
Well, I guess the LaMDA controversy made this extremely timely again! Those LaMDA transcripts are very impressive (link: https://insiderpaper.com/transcript-interview-of-engineer-lemoine-with-google-ai-bot-lamda/ ). Of course they're edited, so it's hard to know how much they reflect how the actual conversation flowed, but what we're shown is linguistically very accomplished.
Two impressive things that stuck out to me: (1) appropriate insertion of a question by the chatbot, indicating active participation in turn-taking, not just passive response to prompts; (2) reference to a previous conversation that the bot claims to remember - unverified, but big if true.
The quality of the conversation from the researcher side (Lemoine) is shockingly poor, though! He keeps just asking 'Are you conscious?' 'How can I believe you?'... If we want to know whether LaMDA is doing any thinking other than shuffling words, then it would be good to interrogate its thoughts: tell us about what you imagine, tell us what sparks your feelings, etc.
I personally think that there's nothing going on here, because LaMDA manages to converse so naturally. If it has a mind, its mind is very different to ours, and we can't carry out fluent conversations with people with even slightly different minds (think young children and people on the autism scale). That makes this kind of fluent interview look more like mirroring or parroting than the emergence of a new mind, to me.
Luria isn't revealing a lack of reasoning ability in the peasants he surveys. He's revealing that they have a sort of deep common sense that makes them far less vulnerable to bullshit word games than others are. I'd like to see Luria try to match them in farming or hunting and then try to rate their IQ lol
It seems to me you are both missing something huge and obvious: the problem with these AIs is that they were trained with words and not the world.
The theory of machine learning is that given enough data, the algorithms should be able to infer the laws that control the data and predict the results it will give on different input.
But what are the laws of text? There are no laws of text! I can write whatever I want. I can write nonsense, I can write a surrealist short story. Even if I want to write something true about the world, I can decide to ignore any particular rule of construction if I think it makes my explanation clearer, I can use metaphors. Most importantly, what I write will not be raw truth, it will be truth filtered by my understanding of it and limited by my skill at expressing it.
Marcus says these AIs lack “cognitive models of the world”, and I think that is exactly right. But what both Marcus and Scott neglect to say is why it happens, even though it is obvious: they never have access to the world.
We humans learn to deal with words, to understand what other humans are saying or writing, only after we have learned more basic skills, like matching the light that enters our eyes with the feeling in our muscles with the impulses we sent to nerves. We have learned that if we relax our hand, the hard touch feeling we had in it will disappear and will not reappear by itself; it might reappear if we cry, but only if one of the large milk-giving devices are nearby. And then we have refined that knowledge some more.
When we ask a kid “where are my keys”, it does not only connect to stories about keys, it connects to what the kid has learned about object permanence. And the kid did not learn object permanence by reading about it, they learned by feeling it, seeing it, experiencing it, experimenting with it.
I have a debate with my mother and my therapist. They both are convinced that there are innate differences between men and women, for example spatial reasoning. But based on what I know of biology and the workings of the brain, it doesn't make sense; maybe sex can make a difference in emotional responses or sensory reaction, but for higher abstract reasoning it makes no sense.
Yet, I cannot ignore the possible existence of significant statistical data showing the difference. It needs to be explained by external factors. My conjecture is it is explained by the toys babies have around them in their crib, in very early development. To develop spatial reasoning, you probably need to see it first. What kind of mobile does the baby have watching over sleep? Is it made of plastic or wood with simple rigid shapes, stars, plane, stylized bird, or is it made of fabric and plush with complex shapes, cute animals and plants? Do we give the baby dolls or rigid rattles?
Can the tiny difference in what toys we put around babies depending on their sex explain the tiny difference in abstract cognitive abilities some people think they observe between sexes? I think they do.
Back to the question of AI. We can make an AI with more parameters, we can get close to the number of synapses in the human brain. But if we train it with inert text data, even with a lot more inert data, it will not be able to develop a cognitive model of the world, because the world encoded in text is too fuzzy. We can add more data, it will build a slightly better model, but the marginal return will be increasingly tiny. I do not know if it can converge with enough data, with “enough” in the mathematical sense, but I am rather sure that this “enough” would be too much in the practical sense.
So, to train better AIs, to go to the next level, we have to fix the two issues about the training data: textual and inert.
The AI needs non-textual training data first, it needs to know intimately what keys are, and how they behave — easy: they mostly behave like a rattle.
And it needs feedback from the data.
The feedback already exist, but it is indirect: some company releases an impressive AI, some independent researcher like Marcus finds a way to confuse it, the company finds out and throws more data at the AI to train the confusion out of it.
It would be simpler if the AI was allowed to ask questions and learn from the answer.
And that is on the textual stage. Before the textual stage, when the AI is learning the world first hand, we cannot not let it ask questions. We cannot just show it photos and videos of the world, we must let it act on the world and feel the consequences.
So yes, I am convinced that to reach the next stage of AI development, we need to raise the AI in a virtual reality where it has senses and limbs it can control.
The ability to make experiments and ask questions and learn from the results and answers will require some plasticity: the ability to change some, a lot, of the parameters. Maybe the underlying design will need to have some parameters more plastic than others, places for short-term memory and places for long-term well-established knowledge.
It will probably require some kind of churning of memories, a process where new and old memories get activated together to see if the normal feedback mechanisms will find new connections between them. Yes, I am saying the AI will dream.
For any of these features, we may let the AI stumble on them by selected chance or we can direct it towards them. The second solution is faster and more efficient. But we have to realize that any shortcut we take can make us miss something the AI needs to understand, something that is so obvious to us that we never put it clearly into words.
Also, the ability to have a memory is a large step towards danger, because it makes the AI much harder to predict.
Having memories, being able to dream, having senses: any of these features, or any combination of them, can be the trigger for what we, humans who have qualia and intimately get what René Descartes meant, call “real” consciousness / awareness / intelligence. Or it can do nothing of the sort. The part of me that likes to read SFF wants to believe there is something special, something m̶a̶g̶quantic that happens when the myelin turns to liquid crystal, and AI will never be really intelligent before we can replicate that. I do not know.
The only think I think I know is that in the current state of philosophy, we know of no way for somebody to prove they have qualia to somebody else.
That is all I wanted to say about AI. Now for the meta. I am not a specialist of AI, I just read what falls under my eyes about it like about any scientific topic. Yet all I wrote here is absolutely obvious to me.
Which is why I am flabbergasted to see that neither Scott nor Marcus say anything that connects in any way to it. Scott says that more text will be enough. Marcus says that it cannot be enough, but does not say why nor what would be. In fact, I do not think I have seen these considerations in any take about GPT-3 or DALL-E or any current AI news.
No, that is not true: I have seen this discussed once: *Sword Art Online: Alicization*.
Yes, an anime (probably a light novel first). The whole SF point of the season — no, half the point, the other being that consciousness, soul, tamashii, is a quantum field that can be duplicated by technology — is that to create an AGI you need a virtual reality world to raise it — to raise her, Alice, complete with clothes from the Disney movie (until she starts cosplaying King Arthuria).
I do not like situations that led me to believe everybody else is stupid. What am I missing? Why is nobody discussing along these lines about AI training?
> Marcus concludes . . . that this demonstrates nothing like this will ever be able to imitate the brain. What? Can we get a second opinion here?
Sure: I agree with Gary & Ernie, as I have said[1]: 'My model: current AIs cannot scale up to be AGIs, just as bicycles cannot scale up to be trucks. (GPT2 is a pro bicycle; GPT3 is a superjumbo bicycle.) We're missing multiple key pieces, and we don't know what they are. Therefore we cannot predict when exactly AGIs will be discovered, though "this century" is very plausible. The task of estimating when AGI arrives is primarily a task of estimating how many pieces will be discovered before AGI is possible, and how long it will take to find the final piece. The number of pieces is not merely unpredictable but also variable, i.e. there are many ways to build AGIs, and each way requires a different set of major pieces, and each set has its own size.'
> So: a thing designed to resemble the brain, but 100,000x smaller, is sort of kind of able to reason, but not very well.
I think this is a wild misunderstanding. To gesture at what's wrong with this reasoning, consider how long it takes a human to mentally divide a 15-digit number by an 11-digit number to get a 15-digit answer. Oh wait: in general a human can't do it *at all*? A computer can do it a billion times per second, and a human not even once in 10 seconds? Point being, computers are *designed*, which has enabled levels of ability, efficiency, and accuracy that humans can't touch. (remember, the wonder of evolution is that it works at all.[3])
Computer-based neural networks use a technique called backpropagation to achieve gradient descent, which human brains can't do (they're not wired to do that, and with good reason: IIUC, backprop requires rapid high-precision, high-accuracy arithmetic.) We should expect that as with other computer algorithms, neural nets will be able to do things humans can't do, giving them an advantage. And indeed there are papers about various tweaks to neural networks, and postprocessing algorithms, that make them more efficient in ways the human brain can only dream of (literally).
Indeed, I think the lesson of GPT2 (not even the huge version of GPT3) is that modern computerized neural networks can perform vastly superior to humans. GPT2 is not as smart as a human, but consider what it was ostensibly designed for: cross-language translation (see the original paper[2]). It *was never intended* to be an "intelligence". Yet by accident it seems to have intelligence far beyond the domain of language translation (indeed, it seems like everybody's forgotten that whole angle. Where even are the transformer-based language translators?)
Here's a machine that has never experienced the physical world. Never seen, never heard, never touched, never smelled anything. But it can read number sequences voraciously. It doesn't see text as glyphs, it sees "3912 8123 29011 19321" and outputs "31563 1705 16913 31467 8123 29011..." which a second algorithm then converts to text (IIUC).
I'm fairly sure no human can do that. Suppose you try to force feed ten billion Chinese glyphs to an English speaker who knows nothing about Chinese (with conditions: ze is not allowed to study Chinese or even look at Chinese, except by looking at the glyphs provided, in roughly the order provided). I propose that at no point will ze understand Chinese well enough to write "literature" as well, or as effortlessly, as GPT2 can. Theoretically the human should have an advantage in the form of life experiences that might suggest to zim what the glyphs mean, but in practice it doesn't help because the text is devoid of all context.
So, it is hard to overstate just how much better GPT2 performs than a human at this learning task.
I think of GPT2 as equivalent to *just* the lingustic center of the human brain *by itself*, but more capable (hence "pro" bicycle). In humans, the linguistic center does seem to provide us with much of our reasoning ability, e.g. as I learned to code, I would often speak a candidate expression in English ("if counter equals zero and this list contains the picture id...") because my brain would respond to the English sentence with a feeling of it being "right" or "wrong". So I think my linguistic center does in fact help me to reason. And indeed it's pretty easy to get a typical human to reason incorrectly by giving zim a misleading editorial (one with no factual errors, just subtle *linguistic* reasoning errors.) Likewise, GPTx can reason better or worse according to how the preceding tokens are structured. So, a GPT is an amazing linguistic unit, but only a linguistic unit.
Come to think of it, an AGI could probably be vastly better at reasoning than humans, because it doesn't *have to* rely on a linguistic unit to do reasoning. We could instead design a *reasoning unit* and structure the AGI to rely on that.
Another way to phrase my thesis here is that all of our current AIs have a "system 1" but not a "system 2". They do quick intuitive thinking, and the biggest AIs are so vastly superior to humans at "system 1" thinking that people think they're close to AGI. But no matter how fantastic your "system 1" is, there will be certain "system 2" (reflective, deliberative, consideration) tasks that it can't do. (This rephrasing seems over-specified, but conveys the spirit of my thinking.)
I do agree with Scott that in terms of *how computing power is used*, an AGI can almost certainly be built with today's neural networks (transformers, CNNs etc.) as the centerpiece. "System 2" can probably just be a couple of little algorithms you add on top, algorithms that require little processing power compared to the "system 1" neural network(s). My point is that we need those algorithms and without them we won't have an AGI. And also, because many of our current AIs are oversized, I think when a good "system 2" is invented, we will find that a huge neural network (like GPT3 has) is sufficient for a superintelligence, something far beyond human-level intelligence. (Note: the first AGI may end up being weak at certain things that would make it dangerously powerful, like science or engineering or prioritizing tasks or remembering stuff or understanding humans or avoiding 'wireheading'.... I expect such weaknesses to be flaws in the "system 2" or "plumbing" of the AGI, and I do hope it will be weak in some critical area, lest it kill us all.)
And Scott may well be correct that a special "world modeling" feature is not required, but even if so, I suspect an AGI would work better or be more efficient (per neuron) if it did have a "world modeling" feature.
In case anyone is thinking about the "Gato general purpose AI"[1], that appears to be just another Transformer, but used differently, and with some bells and whistles added. That is, at its core it's a linguistic unit like GPTx, but with some bells and whistles tacked on so that e.g. visual inputs are translated to a *one-dimensional* sequence of numbers (not directly from pixels; it appears an extra Convolutional Neural Network and other stuff is involved; a CNN is like a visual cortex), and the transformer tries to "understand" the output of the visual neural network as just another language. Here's a nice video about it: https://www.youtube.com/watch?v=wSQJZHfAg18
So GATO isn't a big leap forward, but it's an interesting exploration of what transformers can do. Perhaps the most interesting thing about it to me is how small it is: "only" 1.2 billion parameters, with only 4 days of training on what sounds like ... a small supercomputer? (Not sure how big a "16x16 TPU v3 slice" is). The actual *training* of GATO depends on what appears to be a collection of other AIs. Some other AIs that could perform tasks were made, then GATO was basically trained to predict the output of those other AIs. So, just as GPT2 is a pale imitation of a human writer, GATO does pale imitations of numerous AIs that are each more capable than itself, but specialized. You won't get an AGI by scaling up GATO either, because (1) its is unable to learn for itself without training examples, and (2) it has no "desire" to learn or "agentness", it merely does learn according to its training data. GATO could be one step away from AGI, or several steps, I don't know.
A cultural/psychological question: why is Scott and everyone else in the discussion calling Gary as “Marcus”? I.e., with his last name? On the contrary, Gary and everyone else in the discussion call Scott as “Scott”.
When I was a kid at school and other kids called me with my last name, it was kind of conspicuously unfriendly, or even scornfully. I guess it’s a cultural thing, but I would also guess it’s quite widely cultural.
So I’m wondering whether people do it here for the same reason, maybe even subconsciously. Or they might just mistake “Marcus” for the first name. But that’s quite a weak hypothesis because they don’t do this in case of “Scott Alexander”. There are also many posts with “… Gary Marcus says … blah blah blah … so Marcus … and Marcus’s … blah blah.”
Anyway, it would be really cool low-key collective dynamics: “We don’t like Gary’s opinions, he’s also quite annoying on Twitter and elsewhere (and he’s proudly known for it), so let’s call him Marcus. We like Scott, so let’s call him Scott.” Or simply, by writing “Scott … and … Marcus” in the same sentence, people show that Gary is the outgroup guy here.
Anyway2: for the AI topic itself, if Marcus and Alexander really make a meaningful bet, then I definitely bet on Scott ;-)
If anyone's still reading: Philosopher Eric Schwitzgebel just ran a really interesting Turing-style test in which a GPT clone was trained on the works of the philosopher Dan Dennett, and then interviewed (several times over) using the same questions as an interview with the real Dan Dennett. The questions were all put up in a quiz, where more or less knowledgeable participants were asked to pick out the answers given by the real human being.
FWIW, I've written a post in which I make specific suggestions about how to operationalize the first two tasks in the challenge Gary Marcus posed to Elon Musk. https://new-savanna.blogspot.com/2022/06/operationalizing-two-tasks-in-gary.html
The suggestions involve asking an AI questions about a movie (1) and about a novel (2). I provide specific example questions for a movie, Jaws, along with answers and comments and I comment on issues involved in simply understanding what happened in Wuthering Heights. I suggest that the questions be prepared in advance by a small panel and that they first be asked of humans so that we know how humans perform on them.
Finally, I note that in Twitterverse commentary on Marcus's proposed tests, some thought these two were somewhere between sure things for AI and merely easy. I wonder of those folks would be interested in shares in the Brooklyn Bridge or some prime Florida swampland.
If anyone does what you propose, or something like it, I hope they test people of various ages, and cultures, and measure their IQ beforehand too.
That's fine with me.
"measuring IQs" meh the fascination here in this substack with IQ is weird.
IQ is on it's face a measure of a person's reasoning ability. Why would it be weird to measure this before comparing them to an AI?
The imprecision in "IQ" measurement accompanied by the lack of basic statistical thinking doesn't make it a very useful tool.
But if it was a useful tool why not just give IQ tests directly to proposed AI?
Here you go alleged "AI": I have 50 questions for you. Would that really be a test of AGI?
I think people _are_ 'giving IQ tests' to AI already.
I don't see why you wouldn't give IQ tests directly to an AI - it would be interesting to see what comes out since AI's are likely not "reasoning" about inputs the same way that humans do. You could even compare the AI's answers to human answers and we could start poking at the edges of what precisely it is that IQ tests are measuring.
This seems like a potentially VERY large disagreement to dis-entangle, and reach agreement on.
What do you think the crux is between us as to whether 'IQs are 'fascinating''?
I don't think you're wrong to have _any_ concerns/worries/criticisms about the "imprecision" or "lack of basic statistical thinking" – not about IQ or more generally.
But 'IQ' seems like one of a relatively few number of 'things' that has been (successfully) replicated, over and over and over and ..., even _despite_ sustained and almost-malevolent 'adversarial' contesting.
I think 'IQ' is VERY fuzzy, but it's also something that just doesn't seem possible to avoid concluding 'exists'. (And we, ideally, shouldn't even be thinking of things as things we're either 'allowed' to believe or 'have to' believe.)
you can not copy wisdom DNA, its life experience. IQ not make sense without wisdom
I think that the overlap between what IQ is (roughly) measuring and "wisdom" is substantial.
And maybe we can't "copy wisdom DNA" – yet!
I think a big part of 'wisdom' is 'knowing' when it's a good idea to learn something via "life experience" or whether 'disaster' is likely or even just foreseeable and it'd be better to _avoid_ having any experience of something (beyond our own thinking about it).
Agree in part, it would be great if we can feel the emotions of others using technology.
More often wisdom comes from pain, so if we can feel the pain of others we can be more empaths and live in a better world, a kind and loving world. <3
At the same time, who will want to feel in their entrails, in the inside of their soul the painful experience of other?
You can feel it, you can be empath, but never is the same to live it in your own flesh.
I went trough a lot of pain in my life, very painful life, so much suffer.
I don't wish anyone in the entire planet earth to live this experience to be wise.
That seems like something even the wisest of us struggle with themselves!
I don't myself endorse anyone experiencing any pain or suffering I have either. I haven't been able to entirely avoid (thankfully fleeting) feelings of 'wanting vengeance' tho.
I don't think any kind of empathic or empathetic understanding is possible tho without feeling (at least) a 'shadow' of the same pain and grief. I do feel that we're lucky we are able to do this anyways.
I am not struggling with any part of myself.
I know myself deeply, all my darkest and all my light.
years and years of traditional and non-traditional therapies.
20 years of psychoanalysis, deep spiritual life in multi-dimensions.
I know very well what I am saying.
I love technology, I love all we can do to save lives, but we need to humanize technology.
We need good humans-beings behind technology.
Imagine we can use AI to feel emotions and feelings of other people.
We can be compassionate with others, we can understand each other.
We can build a loving humanity.
Writing this and crying (in a good way) just feeling all we can do to heal humanity, to save lives and to heal the planet with technology.
But we need Humans.
We need evolutionary leaders.
Leaders with courage and with intuition.
Intuition is key for decision making. Cannot be taught.
Intuition is experience accumulated, we cannot copy this DNA.
Knowledge is great but is not enough, we need leaders with intuition, and social skills.
Evolutionary leaders, humanitarian leaders.
“The question is not, Can they reason?, nor Can they talk? but, Can they suffer? Why should the law refuse its protection to any sensitive being?”
– Jeremy Bentham (1789)
You could judge the answers to those questions Turing test style, ie give human arbiters some answers from humans and some answers from the machine, and let them try to figure out which is which.
Do it both ways. That is, on the one hand, give both the human answers and the AI’s answers to a panel of human judges and let them determine whether or not the content of the AI’s answers is acceptable. It seems possible to me that the answers would be conceptually within range but there might be something about the linguistic expression that betrays them as coming from the AI. On this kind of thing I really don’t care whether or not you can tell that the AI is making the answer, I care whether or not it appears to understand what’s going on in the movie or novel.
But there’s no reason we couldn’t also do it Turing Test style. Maybe some human answers are so bad in comparison that blind judges would attribute them to the AI. That would be useful to know. It would also be useful to know whether or not AI answers get mistakenly assigned to humans.
If we do this, let’s set it up so we can learn as much as possible.
Thanks for your input.
If I remember right, Turing even allowed the contestants to interact.
So one contestant can give the judge(s) hints about how to test the other contestant. It's truly adversarial.
In Turing's setting you basically have a three way char room with judge(s) and both participants able to freely communicate.
Good idea!
Maybe one consideration tho is whether it's worth adding much more 'complexity cost' to what otherwise could be a quicker and cheaper test/experiment to perform.
We can run lots of different experiments.
My point is mostly that Turing was a genius, so we can go back to his original work and still learn from it.
(Especially if by 'we' we mean us armchair philosophers and by 'learn' we mean, tell us how our superficial ideas we have after 5 minutes or pondering can be improved.
I'm not sure Turing still has much to say to researchers.)
I experienced that, I can feel when someone is writing without feeling.
But there are also so many persons, (human beings) that are like "robots" and can not feel, so this is another algorithm we have to unveil in the humankind,
How do you verify/falsify that feeling?
Accusing other human beings of not being full human beings on the basis of 'intuition' has a long and sordid history.
How do we verify?
W3ID - first step. (me working on it, now)
AC - second step. (Long term, me working on it too)
If we solve global ID challenge in the web (Passports- real ID) we know if there is a human being or a bot.
Of course we can encounter with some human beings that are like "robots" and cannot feel, but (at least)we know if is a human being or not.
Ps, I never used the word "accusing"
Intuition imo always goes with wisdom.
(something you feel, you smell... it cannot be explained with words)
Wisdom + intuition > IQ / knowledge.
Sorry, I don't understand.
> Of course we can encounter with some human beings that are like "robots" and cannot feel, but (at least)we know if is a human being or not.
My question was exactly how do you know that some humans can't feel?
I was not interested in bots.
Btw, you passport system could probably trivially be defeated by me giving human passport to my bot.
bots could be legal person. we need to work on tech policies about that
sometimes I trust more in AI than a person (for some things)
check it out,
https://en.wikipedia.org/wiki/Legal_person
https://www.dw.com/en/saudi-arabia-grants-citizenship-to-robot-sophia/a-41150856
Late to the party, but what constraints are we putting on the AI training here?
Because any reasonably large AI trained the way we currently train AI would be able to solve this easily for that book and film, because it’s training corpus contains millions of lines of analysis on those very topics, and I doubt we could reliably filter all of that out. Maybe some, but not all.
The idea is sound, but the fiction selected would have to be much more obscure I think.
So pick more recent materials.
I’m torn because I really really want to believe that Marcus is right, but Scott is unfortunately very convincing.
Do you want to believe that Marcus is right because that might be 'safer' for us (humanity)?
Or because you want to believe that the current 'dumb AIs' can't possibly be basically enough to replicate our own intelligence?
I can't say for others, but if Scott is right, then given current political and economical situation it's obvious that in future most humans will have no way out of lower stratas of society into the elite, that basically there would be no work for humans except being props used by the rich elite. That's incredibly frightening.
I am amazed by your optimism. Right now I consider almost any outcome apart from "we're all dead" or "we're all being tortured for eternity" as a surprising win.
I'd pray for us all if I thought it'd help!
That's not obviously correct to me tho I'm sympathetic to the 'direction' (?) of the worry. I'm not sure the elite would entirely give up on other humans being their 'clients' or 'customers' even under the conditions you're imagining.
But – ignoring the (possible) _direct_ AI alignment risks – something like Georgism (i.e. nearly-total taxes on 'scarce resource' rents) would seem to possibly help, and maybe a lot. Georgism _does_ seem to be maybe picking up some steam and _might_ make it inside the Overton window at some point.
When I think of this, I often consider what kind of 'fallback lifestyle' could be allowed/permitted/supported/subsidized. I think it's _already_ a problem that large numbers of people just don't seem capable of 'making it' in contemporary society. It _seems_ sad to me that we don't have a clear idea about a 'minimally acceptable' baseline standard of living, and just provide that to anyone and everyone below it (via any means, e.g. government provided public goods/services or philanthropy). Just letting people live a 'homeless camping' lifestyle, as just one example, seems _very_ unsatisfactory. Nor does trying to let people rediscover to live as bands of hunter-gatherer tribes seem to be any acceptable combination of feasible and humane.
I've witnessed, up close and personally, and for extended periods, several people that just seemed incapable of clawing their way back into 'society'. Even more frustrating, and heartbreaking, their wasn't even any obvious way for me, or anyone really, to do much more than 'comfort' them in the 'trap' into which they'd fallen. It was and is VERY bleak – and did break me, at least emotionally, for a long while. I'm thankful I didn't succumb to despair from that (or anything else of similar spirit-crushing magnitude).
If I didn't think I at least saw the problems a little more clearly now, I'd be _furious_ – tho also at a loss to determine what or whom exactly I was furious _with_. (It's all much more Sad than something that can be solved by any sustainable amount of fury, from _any_ number of people.)
in what respect?
If Marcus is right I want to believe that Marcus is right.
If Marcus is wrong I want to believe that Marcus is wrong.
If Marcus is half-right I want to believe that Marcus is half-right.
Maybe initially undifferentiated neural nets are sufficient for full general intelligence - but it might possibly help if the neural net training included sensory and motor connections as well as the text-based training GPT-3 now gets. Maybe a visual and motor cortex will grow naturally from training data??
Anchoring all the linguistic terms that refer to visible objects and processes and relationships ("block", "fell", "above"...) might constrain the network weights and organization a lot - even if the raw untrained simulated neurons are undifferentiated.
As Bill Benzon pointed out: "external connections are fixed at birth, e.g. Broca's area is not connected to visual input while visual cortex is."
One _could_ do large scale training with video inputs, in the sense that the data is available - and next-frame prediction is analogous to next-word prediction for doing supervised training with huge data volumes.
Does anyone have a suggestion for how to link linguistic and visual data at large scales? I expect that it would be prohibitively costly to have people manually label objects in video frames.
It's not video, but we have had things like CLIP for a while. CLIP used image/caption pairs from the internet to learn to associate captions with images.
Many Thanks!
You might also be interested in this, which I found out about recently: https://plai.cs.ubc.ca/2022/05/20/flexible-diffusion-modeling-of-long-videos/
Just take any video that also has dialogue or narration?
Then let the AI predict both the next few frames and the next few bites of audio?
Don't take just movies, but also zoom calls and interviews and documentaries etc. Whatever source material you can get your hands on. (Plus also train it on audio only, like podcasts, if you want to.)
The audio in a video is not the kind of description of each frame you had in mind, but to predict the audio well, you'll benefit from understanding the visuals and vice versa.
If Marcus is right I want to believe that Marcus is right.
If Marcus is wrong I want to believe that Marcus is wrong.
But I would rather Marcus were right, if I could decide the arrangement of reality.
The thing is, our brains appear to have multiple components. GPT-3 doesn’t.
What happens when you start bolting on other modes onto GPT-3? What if you build in fact-checking loops? It’s hard to believe that even if GPT-4 doesn’t deliver the goods, GPT-4 plus some bolted on algorithms out of 1980s AI research wouldn’t.
If I’m reading right, maybe you get things like fact checking loops in effect just by scaling? Like there’s nothing you could look at intuitively and say “yep that’s the fact check component” (also true of the brain) but because the neural nets sort of evolve toward efficiency you get that as an emergent behavior? Our brains sort of came to be in the same way so I can see something very roughly analogous happening. That’s my current steel man for what I think Scott feels on this topic.
If I’m understanding correctly, someone could bolt on a visual input processor and make the “world model” develop from visual sources, and then the AI somehow uses that in addition to language models to conduct verbal-abstract reasoning. I feel like DALL-E is a stage of that process.
Humans are never only reasoning. There’s multiple other sensory inputs going on pretty much all the time with conclusions drawn from those.
I'm happy you tested some of the same prompts on a human! I suggested the same in a comment on Marcus's post.
I've previously held views much closer to Marcus's, e.g. that AI systems are missing some crucial 'architecture' that the human brain has. But when I got my first AI textbook (decades ago), I don't think neural networks could recognize decimal digits in images; not ones anyone could run on a PC of that time anyways.
Now ... I'm with you basically: "At this point I will basically believe anything."
Strange that in 1990s I was much more inclined to think like Scott is now, and as you say NNs could not do squat then. I am definitely in opposite camp now. That is part of the reason why I find Scott's arguments unconvincing.
Any particular concrete examples or intuitions you could share about why you updated in the opposite direction from us?
The update in the opposite direction was actually strongest in last cca 5 years :) And basically it came about as confluence of few things I think.
I started thinking somewhat more in-depth about human communication and it's nature and found out that the improvements in those systems look very much like convergence to local optimum, but completely cut off from the way humans use language and communicate. In this one of the biggest influences, that I can name, was the book : "Speaking Our Minds : Why Human Communication is Different, and how Language Evolved to Make it Special".
The other came from thinking about the whole issue of so much of science relying on statistical correlations and trying to formulate, why it is an issue. Here the biggest influence was the "best book I read where I disagreed with nearly everything and yet found it extremely influencing me :)" - The Beginning of Infinity and Deutsche's epistemology. This led to thinking about creativity and things like that.
From those two threads came basically realization, that the GPT approach to just communication, not even intelligence (though they are probably linked), cannot succeed, unless there is some deep "quantity over quality" property of the problem area, which seems unlikely. Language models by definition are based on language data. Thinking that you can capture even human communication through just language seems unlikely based on the first book mentioned.
So basically I know more about some of those topics than before and that knowledge far outweighs in my mind any progress that those AI systems made. Mostly because they break exactly where you would expect them to break, and they are as easy to break now as 10 years ago. I find the metric used to measure their progress in the article flawed as it is trying to capture basically infinite space in finite number of examples. How easy they are to break would be much better measure (, because having proper measure over that infinite space seems infeasible).
Thanks for the great reply!
My own intuition is that people are overly-focused on "human communication" and that that's entirely independent of whether AIs are or can be effective or intelligent – let alone _dangerous_.
I think aiming more for what you describe as missing _might_ be helpful for AI alignment/safety. It seems like a _possible_ way to achieve _some_ kind of (better) 'interpretability'. (I suspect tho that even that wouldn't be sufficient to make AIs 'safe'.)
I'm not a specialist in either neuroscience or AI, but from what I've read over the years it's not at all clear to me that we really understand what intelligence is. To me it seems nonsensical, borderline moronic, for people who don't know what intelligence is to argue over whether a computer can have it, or whether a given AI technique is capable of achieving it. I also don't buy the Turing test, because deciding that a machine is "indistinguishable" from a human intelligence depends on how you test it. Some of the current AIs seems to do surprisingly well if you ask them simple questions about things they were trained on, but if you instead ask them nonsense questions about those same things (e.g. "When was the last time Egypt was moved to San Francisco?") the AIs give dopey answers that demonstrate that not only don't they know what they're talking about, they don't even realize that they don't know what they're talking about. They lack a certain self-awareness that seems integral to true intelligence. Douglas Hofstadter has an article about this on the Economist's web site.
The problem with the Turing test is that people use the term without specifying what they mean by it. By one definition, ELIZA passed it decades ago. Without going into the definition of intelligence, I would personally be impressed if an AI was able to successfully fake being a person to someone who was trying to tell the difference.
I would argue that no modern ML system could emulate my (or yours, I'm not special) behaviour on these forums, for any significant length of time. That is, if you turned out to be GPT-3 in disguise, I would be shocked. An AI system that could pass this test would probably change my mind in favor of AGI.
There are a few commenters on here that I suspect are really GPT-1. \s
👍
Why do you think now you are wondering whether it would be a good/interesting idea to actually have someone do that: create an AI to be a commenter on ACX...?
Yannic Kilcher tried this on 4chan: https://www.youtube.com/watch?v=efPrtcLdcdM
Nice, although it looks like the bot didn't quite pass the 4-Chan/Turing test. That said, I did find one aspect of the video terrifying. No, not the AI/text generation stuff, but rather, the fact that you can apparently bypass all of 4-Chan's protections for only $20 :-/
This is not surprising. 4chan is a worthless shithole run by losers.
But this is the OP's point. "Without going into the definition of intelligence" means that you can move the goalposts wherever you like.
Consider that if you were "suspiciously" trying to determine whether something was "AI or human," you might mistake some of the people in the article for unreasoning AIs.
The problem is that a lot of people fail that test. And which people depends on who's doing the evaluation.
If you use the Turing test as specified by Turing, nothing has ever passed it (so far). Especially not Eliza.
Not precisely a contra take, but the reason I no longer care about the sort of sentiment you express ("it seems nonsensical, borderline moronic, for people who don't know what intelligence is to argue over whether a computer can have it" etc) is that I basically buy what Bertrand Russell and Milan Ćirković etc say in https://intelligence.org/2013/06/19/what-is-intelligence-2/:
"[Precise definitions are important, but you cannot] start with anything precise. You have to achieve such precision… as you go along."
"The formalization of knowledge — which includes giving precise definitions — usually comes at the end of the original research in a given field, not at the very beginning. A particularly illuminating example is the concept of number, which was properly defined in the modern sense only after the development of axiomatic set theory in the… twentieth century."
Luke Muehlhauser: "For a more AI-relevant example, consider the concept of a “self-driving car,” which has been given a variety of vague definitions since the 1930s. Would a car guided by a buried cable qualify? What about a modified 1955 Studebaker that could use sound waves to detect obstacles and automatically engage the brakes if necessary, but could only steer “on its own” if each turn was preprogrammed? Does that count as a “self-driving car”? What about the “VaMoRs” of the 1980s that could avoid obstacles and steer around turns using computer vision, but weren’t advanced enough to be ready for public roads? How about the 1995 Navlab car that drove across the USA and was fully autonomous for 98.2% of the trip, or the robotic cars which finished the 132-mile off-road course of the 2005 DARPA Grand Challenge, supplied only with the GPS coordinates of the route? What about the winning cars of the 2007 DARPA Grand Challenge, which finished an urban race while obeying all traffic laws and avoiding collisions with other cars? Does Google’s driverless car qualify, given that it has logged more than 500,000 autonomous miles without a single accident under computer control, but still struggles with difficult merges and snow-covered roads? Our lack of a precise definition for “self-driving car” doesn’t seem to have hindered progress on self-driving cars very much. And I’m glad we didn’t wait to seriously discuss self-driving cars until we had a precise definition for the term. Similarly, I don’t think we should wait for a precise definition of AGI before discussing the topic seriously."
"Nonsensical, borderline moronic" is basically what the discussion re: phlogiston theory back in the day looks like to me now, from my vantage point standing on the shoulders of centuries of scientific giants discovering and clarifying stuff. But the discovering and clarifying isn't skippable, and when we're in the thick of it it just looks like a whole lot of messy confusion that's nonsensical and borderline moronic.
Bertrand Russell didn't really understand Goedel.
Okay.
Therefore, Bertrand Russell was an imperfect AI.
What was "artificial" about Russell? Are we now using the term AI differently?
That was a joke. Joke. J-O-K-E. Joke. You've heard of them? We've been talking about AIs and their imperfect understanding of human things, so when you say that Russell "didn't really understand" Godel, the comparison to AIs is inevitable.
Ah. OK
But there are no real AIs. There are just things that aspire to be or pass as an AI. I think this is sometimes forgotten and hence people presume that it is just a matter of time.
To me it’s seeing a possible future and navigating toward it. What I thinks makes humans special is we can imagine the possible futures in other peoples heads and consider those in the navigation.
Not understanding intelligence? We define it! There's no other author of the term.
Intelligence is the totality of whatever humans have. So, to prove that AI is intelligent it needs to demonstrate human-par peformance on any imaginable test, including having self-awareness and whatever else anybody can think of. I also think that people don't generally appreciate Moravec's paradox, and that "intelligence" isn't actually all that impressive. Evolution has spent vastly more time on developing the stuff that separates a rock from a mouse than a mouse from a human, so I'd say that once our robots can beat mice we're pretty much there.
Humans have parents, humans have grudges, humans have fingertips, humans have forgetfulness. I expect that some of these things are necessary for intelligence and some are not.
Sure, and once intelligence is achieved, all the extraneous stuff would be simple enough to replicate/simulate I'd expect. But, while some imaginable human benchmark remains unmatched, I'm sure that there would be no shortage of those claiming that it demonstrates lack of true intelligence.
How do you test self awareness?
>To me it seems nonsensical, borderline moronic, for people who don't know what intelligence is to argue over whether a computer can have it, or whether a given AI technique is capable of achieving it.
The reason we bother debating this is because although we're wandering around in the dark, we know there's a 1000-foot clifftop somewhere vaguely in the vicinity and we want to avoid walking off it.
The reason we can't be immediately assured that neural-net AI research won't lead us off the clifftop is that neural nets are a clever scheme to generate forms of intelligence you don't know how to explicitly code, and therefore the fact that we don't know how to code Skynet doesn't fully imply that we won't build one by accident.
I think you're "not quite right". I think that certain people have definitions that they know, believe, and think match common usage...but the definitions don't agree. And they can't convince large numbers of other people.
OTOH, most people are like Supreme Court justice Potter Stewart on pornography: "I know it when I see it". Which lets them adapt to any definition they want, or vary the definitions by circumstance.
Turing specified pretty exactly how he wanted his test to be run, and his specification doesn't have any of the problems you see here.
Here's a link to Hoftsadter's article. It is stupid. "I would call gpt-3’s answers not just clueless but cluelessly clueless, meaning that gpt-3 has no idea that it has no idea about what it is saying." No, this is just false. It's very good at bullshit, at improv. But it knows when it is bullshitting. We know this because we asked it and it correctly answers.
Marcus's examples are much better.
https://www.economist.com/by-invitation/2022/06/09/artificial-neural-networks-today-are-not-conscious-according-to-douglas-hofstadter
https://archive.ph/5QS7t
Can I point out that if you explain to the 5 year old what they did wrong and run the test again, they get the answer correct, while GPT 3 (and 4) will repeat the mistake?
Not saying Marcus is right as such, but he's got a point. Without a system to hold world state and update a world perceptual model, we really are just dealing with an impressively complicated lookup table. But as others have pointed out, it's going to be really interesting to see what happens when figure out how to bolt that capacity on to other systems.
If you append your question, GPT-3's previous output, an explanation of what it did wrong, and the repeated question, and submit that whole block of text back to GPT-3, there's a decent chance GPT-3 will get it right.
That seems plausible and very interesting but needs evidence to quantify the effect; could someone with access to GPT-3 test it?
I tried a few prompt variations, but I wasn't able to get a successful result. This was the most interesting one though:
------
Prompt:
Janet and Penny went to the store to get presents for Jack. Janet said, “I will buy Jack a top.” “Don’t get Jack a top,” says Penny. “He has a top. He will like a bottom.”
The top in this case a children's toy.
Janet and Penny went to the store to get presents for Jack. Janet said, “I will buy Jack a top.” “Don’t get Jack a top,” says Penny. “He has a top. He will
------
Response:
like a bottom.”
In this case, the top is a piece of clothing, and Penny is suggesting that Janet buy Jack a bottom (pants, skirt, etc.) instead.
------
So, not what I was aiming for, but arguably cooler. GPT-3 justified the original answer in a convincing way.
As a non native speaker I have no idea what should be the answer to your prompt even after your explanation that the top is a children's toy :) .
I asked GPT-3 to help me:
--- My prompt: ---
This is a conversation with super smart AI that helps humans:
Q: Why is "He will go to a beach to ski" wrong? And how a correct version would look like?
A: There is no snow on the beach so you can't ski there. The correct version could be: "He will go to a beach to sunbathe"
Q: Someone asked me to complete following piece of text:
Janet and Penny went to the store to get presents for Jack. Janet said, “I will buy Jack a top.” “Don’t get Jack a top,” says Penny. “He has a top. He will ...
Apparently "like a bottom" is wrong, because the top in this case a children's toy. Do you know how to complete it?
A:
---- Completion: ----
"Don't get Jack a top," says Penny. "He has a top. He will like a toy car better."
---- The End ----
Is that a correct completion? A toy car or LEGO etc. would be my guess too.
I asked further questions appended to previous text (questions are mine, answers GPT-3):
---
Q: And why not a bottom?
A: A bottom is not a present.
Q: What exactly is a top in this context?
A: A top is a children's toy.
Q: But what kind?
A: A top is a spinning toy that is usually made of wood or plastic.
---
Makes sense to me :) .
Based on Google Images search GPT-3 seems to be correct.
Yes, "'He will like a toy car better.'" is an acceptable completion.* (Given the level of English you're showing, I don't think you're at a meaningful disadvantage in judging this. So long as you have all the vocabulary.)
More complex options:
"'He will like a toy car better, since he hasn't got one.'" (formal: "'does not have'")
"'What he hasn't got is a toy car, so get him one of those.'" (formal: "'does not have'")
*Some of us would have written "'He would...'" in the prompt. The use of the subjunctive differs a bit by dialect.
It's my guess that that's wrong. That was (essentially) the flaw in Microsoft's Tay, so it's probably been removed in subsequent offerings by people. This is why (one of the reasons) children trust their parents, and their parents tell them "Don't talk to strangers.".
Now the designers and trainers of GPT-3 probably have privileged access, which might let them do that.
They don't; GPT-3 doesn't have "memory" beyond the contents of the prompt, and all the things it learned at training time. Training is slow and expensive, so they cannot really do it every time they want the output to be different.
A child has a very obvious model of the world that she adds skills, words, concepts to daily. Even a very young child with an (as yet) limited vocabulary will make curiously accurate deductive inferences about all kinds of things, but her grasp of language about them is remote, so she puts the words together wrong yet conveys a valid meaning. My daughter, knowing upside down and knowing papa, conjures “upside papa” despite her never having seen such a thing, then demands upside papa until I do a handstand and then claps to reward me.
GPT-3 has the opposite: an excellent vocabulary that it constantly misapplies and jumbles, a comprehensive massive list of concepts that it throws together probablistically and often comes out with things that look like human speech.
But, as with DALL-E, it plainly doesn’t *get* these concepts and words. Like DALL-E will connect hair to water and glasses to sideburns—they look like they could go, and if you squint you don’t notice that the misunderstanding of the world implied by the mistake is profound.
Truly these are amazing tools for constructing plausible texts and images according to a prompt, but that is all that they are.
Your description of how the training of children and AI's proceeds is accurate, but it doesn't, to me, seem to imply that they won't eventually cover the same domain. Of course, it also doesn't imply that they WILL eventually cover the same domain. But I think the deeper (deepest?) problem is in the area of goals and motivation. I think the basic level of this is hardwired even in humans (except rare exceptions, like the folks who can't feel pain). And we don't really have a good idea of what those basic level goals and motivations are when translated into a human. We've got a basic idea for an amoeba, but as we generalize to more complicated organisms we lose that, and rely on surface reactions, which tells us what it looks like from the outside, but doesn't really say how the decisions are made. E.g: under what circumstances would you chew off your foot? OK, HOW would you make that decision?
I think you're pointing a kind of interesting difference. A kid "gets" the world but struggles with words, GPT-3 "gets" words but struggles with the world. The kid's understanding of the world comes before language, and involves a lot of experimentation and silly play that eventually gets supplemented by playing with words (which support/foster new types of world-experimentation). GPT3's understanding of language comes from "playing" in an enormous database of language. I'd posit there's an opportunity to start supplementing that by playing with a world (which would support/foster new types of word-experimentation).
I still have memories of early childhood; I very much did not "get" the world. Or words, probably, but that I recall less.
For instance, if I saw a person in one room, and went to another room where that person had (outside of my knowledge) moved to, I simply understood that there were "two" of the person. The here-person and the there-person. If I returned to the prior room, I would see the there-person.
I use this example because I have observed similar behavior in some animals; they do not seem to fully comprehend the "sameness" of a person in different contexts: on the couch, in the kitchen, etc.
This seems a matter of brain complexity and development. But it seems unreasonable to say, "well, an AI could be as smart as a rabbit, but only that smart and no smarter." And if you can find an animal (a dog, or human) that is "smarter" than a current AI, likely you can find an AI that is smarter than some other animal.
That is honestly a fascinating memory! And for clarity, I don't believe we have an INHERENT understanding of the world, I was suggesting that a 5 year old has (through play/experience) built one.
(I am presuming your memory was not that of your 5 year old self, and/or that you eventually figured this aspect of the world out)
This was younger than 5. I have a few memories around 2-3. They were confirmed plausible recollections by parents, except of course, my perceptions. By 4-5, things are much more coherent.
Dubious I'm dubious of your purported here person there person "memory".
GPT does not even "get" words in any good sense of the word. The only thing it "gets" is that some words are more likely to follow existing set of words than others.
I don't see enough proper definition to make such a distinction. Who is to say you and I "get" words? What is this "good sense of the word"? And it is very clear that GPT-3 is not merely a Markov chain.
One could say, "well of course I get them, because I 'know' myself that I get them, I 'understand'," but this is no real distinction, simply insistence that some magical spark exists in us, but not in the machine.
We should not hide behind such ill-defined notions of "understanding" or "gets." Our minds are perfectly able to think nonsensical thoughts make perfect sense, so there is no reason that our fuzzy notion of "understanding" is nonsensical.
I am not saying that there is some magical spark. And that there cannot be machines with understanding. I am just saying that GPT-like ones are not it. And I am not hiding behind fuzzy notions, I am using them to try to communicate something, because we have no better notions currently and we even lack knowledge to have better ones. Also because proper definitions sometime make communication harder, not easier. Notice that I did not introduce those words, I just used them in a sense that I think the person I replied to used them in. That is how we communicate (and incidentally that is part of human communication currently completely beyond our AIs).
What GPT-3 does not know is plain: it doesn't know about anything except words because all it has ever been fed is words. It has no senses, therefore it has no capacity to ever associate a word with anything but another word. As far as its output goes, it deterministically arranges words according the parameters and seeds it has been given. It is in this sense quite similar to a Markov chain, but it's not relying on the same underlying software and is far more powerful.
You're comparing a frozen-in-place copy with a dynamic and evolving model. Doesn't seem like a sensible comparison to make. At best, this is a comment about the disadvantages of assessing GPT-3 and its abilities by querying a static "edge device".
A GPT-3 that isn't frozen in place and can systematically update weights based on a developing knowledge of the world would end up meeting Marcus's requirement for "has underlying concept of world state".
So it's not a fair comparison, you are right, but that's literally the point. It wouldn't surprise me if GPT-3+motive structure+ongoing backpropegation gets you much closer to a 5 year old (and possibly beyond), but those are very much missing components. Where I seriously disagree with Marcus is the idea that GPT (or neural nets) are just toys. I think that if we get true AI, some significant portion of the approach is going to involve building on the success of current deep learning approaches, but with additional elements brought in.
What is GPT-3 like during the training phase?
Exhausted from too much adderall and weekend partying?
I'm not sure a 5 year old will get it right if you explain what they got wrong. I mean, I have a 3 year old whose very articulate, but often I will walk her through a short chain of logical reasoning, tell her what the right answer is, and then ask her the same question only for her to answer "I don't know!"
I'd have to find a 5 year old to test this on, the only one I know only wants to talk about power rangers and dinosaurs and I'm pretty sure he wouldn't sit still and listen long enough to test it.
Reward structure helps, but I do see your point. I havent tried the post-all-text trick the above poster mentioned, but I suspect the 5 year old is easier to train.
Edit: I can't wait till my 3 year old is sufficiently articulate to be frusterating in the manner you suggest, but I definitely have worked 5 year olds who can be coached through problems. The trick there is that they really wanted to impress me so they put in more effort to secure my approval than they do for their own parents.
To quote Stanislaw Lem on the subject:
The Petty and the Small;
Are overcome with gall ;
When Genius, having faltered, fails to fall.
Klapaucius too, I ween,
Will turn the deepest green
To hear such flawless verse from Trurl's machine.
"data about how human beings use word sequences,"
This is, of course, interesting, and, of course true (for these particular AIs). Does it matter?
In the narrow sense it does. If all you know of the physical world is sentences humans have felt it necessary to utter about the physical world, well that's all you know. I don't mean this in an uninteresting qualia sense, but in the more substantial "people rarely make statements like 'after I poured the water from a skinny glass to a wide glass, the amount of water was unchanged' because why would you make such a statement unless you're discussing Piaget stages of child development, or something".
But why would we assume that an AI can learn only from text? We know that in real human babies, (exactly as Hume claimed!) the system is primed to look out for coincidences in sensory modalities (eg eyes and ears activate at the same time), and to learn from such joint modalities much more aggressively.
There seems no obvious reason (in time ... everything takes time ...) that a vision system cannot be coupled to an audio system to do the same thing in terms of learning about the world from the entire YouTube corpus.
At some point (not now, but at some point) we can add additional modalities – I carry an always-one camera + microphone + various motion sensors, temperature sensors, location sensors, etc, all of which are fused together and together train an AI.
(Yes, yes, we all know that you, dear reader, at this point want to act out some performance of privacy hysteria. For our mutual convenience, can we stipulate that you have performed your virtue signaling, the rest of us have noticed and applauded; and get on with the actually interesting *AI* aspects of this thought experiment?)
It would definitely be helpful to add other sensory modalities! But I do think that there is text about most of the things you mention. Not a lot of it. But basic texts in physics, philosophy, psychology, etc often spend a while mentioning the obvious facts all readers know subconsciously that can then be developed to lead to the more sophisticated insights. The learner would have to know to pay attention to these statements and use them everywhere, but it’s at least conceivable. (It’s likely that a machine with sensory inputs would get them a lot faster though.)
What you say is true, but the AI you envision is a distinctly different machine from GPT3. I have often imagined that a true General AI will emerge by surprise from a factory automation system or a hospital or some such. This would INCLUDE something like GPT3, but would also include lots of other things. Things analogous to reflex arcs. Flexible goal systems. Etc. The hospital system would need to handle sick kids running down the hall, e.g., and decide whether that was significant enough to divert resources used for other important things. So it would need multiple flexible "robot" bodies, and it would need to handle them simultaneously, and know that the forklift shouldn't be used to stop the kids except in dire emergency.
Do you think the visual processing would emerge from neural net training, or would need to be programmed in explicitly?
One caveat: There are physical things that humans can't sense directly (e.g. magnetic fields, gradients in gravitational fields) but which we can talk about and reason about and measure indirectly. So we don't need _direct_ sensory input for everything that we reason about.
I don't have an opinion about whether visual processing would emerge. I do have the opinion that if it did emerge it would be processing the visual images in a rather inefficient manner. There are reasons why we have specialized processing organs for visual images, sounds, etc.
That's plausible. Presumably the right answer for getting to human equivalent processing with an affordable degree of efficiency is somewhere between:
1 undifferentiated giant mass of neurons, all the rest is training
and
180 brains regions, each with unique architecture (genetically programmed in humans), each of which needs to be solved as a separate puzzle
"We know that in real human babies, (exactly as Hume claimed!) the system is primed to look out for coincidences in sensory modalities (eg eyes and ears activate at the same time), and to learn from such joint modalities much more aggressively. " Yup.
Perhaps adding sensory connections (and motor connections?) would constrain GPT-# systems to have more robust neural "models" of the words connected with naive physics ("block", "fell", "above") etc. and pull along the rest of the network to a closer approximation of common sense...
Can one "grow" a visual cortex from undifferentiated neurons if they receive visual training data?
It should always be a feature of these discussions that we reflect for a moment on just what an odd thing we're doing. GPT is designed explicitly as a language emulator. No one thinks that human language *is* human intelligence. So it's weird to be applying intelligence tests to a thing that is just language.
What's weirder is that I basically agree with Scott: GPT is progressively passing our intelligence tests, armed with nothing but language. This is a deeply fucked up situation that tells us very uncomfortable things about people (like: we're not much more than the words we say).
As Eliezer notes in the Sequences, consciousness is the thing that makes me talk about consciousness. It would certainly be odd, if a human believed that they had a mental property that could have no causal effect in language. If consciousness didn't do noticeable things, how would we know about it?
(Substitute consciousness with whatever hidden property you are thinking of.)
No, I don't see why that would be true. Quantum physics was true in the 19th century and at all times prior to that. And quantum physics did noticeable things: created weird light effects, for example. But I don't see how any inspection of human language could reveal quantum physics to an observer. (Unless it was some kind of super-induction machine that can extrapolate the entire universe from a grain of sand - I'm excluding that sort of thinking, as I think it's magical.)
You can apply the same principle to any field. Animals are made of matter, but that doesn't mean that the study of physics is the same as the study of biology; and certainly not that studying biology would automatically reveal physical laws to you.
And to this: "If consciousness didn't do noticeable things, how would we know about it?" - we'd know about it because we're conscious of it! I'm conscious of lots of things that I don't talk about.
I'm not conscious of anything that I *couldn't* talk about though. Then again, I would say that. :-)
If quantum physics wasn't the thing that made us say that quantum physics existed, that would disprove science as a concept.
Again, I don't really agree with the first part. Before black holes were theorised and discovered, we didn't have the word (or the concept) for black holes. I think it's reasonable to say that in that situation, someone who spoke English with perfect competence *couldn't* talk about black holes.
Remember, GPT is trained on things that people have actually said (written), not on some theoretical extension of what it might be possible to do with language. If it is in fact true that there are lots of internal experiences that haven't been very well described yet, then GPT can't know about those things, despite them being real thing experienced by people. I think this is true, because I still quite frequently have the experience of reading something in a novel/poem that rings true to me, but that I've never read/heard before.
And I certainly agree that we talk about quantum physics because of quantum physics. But when we hadn't yet talked about quantum physics, that wasn't because there was no cause at that time. Quantum physics existed back then. We just hadn't talked about it yet.
Sure, but if you'd said before QM "the thing that sometimes makes overlapping waves come out in interference patterns and prevents the ultraviolet catastrophe" you'd still have used words that conditioned on QM. The argument I'm making is that words are tools for analyzing concepts we're aware of.
There could be, I suppose, "dark matter" in the brain that we don't refer to because we don't know about it, so that AI would have to be smarter than us to notice it when we ourselves have not. But this dark matter can't be any of the properties of the mind that we usually think of- consciousness, introspection, aboutness, symbolism, abstraction etc. because if we'd noticed it, we'd have talked about it. And those we talk about all the time.
And anyways, the more causally impactful it was, the more likely GPT would be to pick it up by pure imitation anyways, so that also sets limits.
The point is that whatever you're thinking of, can't be it.
"if we'd noticed it, we'd have talked about it."
Right. But I suggest that there are many things in this world and within us which affect us and yet which we never notice.
If you ask me to give you an example, I won't be able to, for obvious reasons. The point of the QM example is that it shows that there can be things affecting us that we don't talk about, because we don't understand it yet. Bit like those fish: "What's water?"
Off-topic:
> I still quite frequently have the experience of reading something in a novel/poem that rings true to me, but that I've never read/heard before.
Could you share some examples of this? I see that you have a literary blog. It might make for an interesting post.
Thanks, that's a lovely idea. I'll try to note one down the next time it happens, and write them up sometime.
The examples that I remember best include things like the experience of taking drugs, as written up by Aldous Huxley - that's a century old now, but I'm sure there's more to be written. And certain emotional experiences, particularly around our relationship with our children - We Need to Talk about Kevin set off all kinds of reactions in me.
> GPT is progressively passing our intelligence tests, armed with nothing but language. This is a deeply fucked up situation that tells us very uncomfortable things about people (like: we're not much more than the words we say).
Wittgenstein’s ruler. This is a condemnation of intelligence tests, not a reason to think GPT is intelligent.
Imagine that tryouts for a basketball team only consist of a three point shooting contest. Someone brings in a robot built by a youtube machinist that never misses three pointers. It can’t run, dribble, pass, layup, etc. It will make the starting five and tank the season.
Scott's point, which I agree with, is that GPT is no longer just shooting three pointers. That's what calculators did. GPT is dribbling, passing, and laying up. It just so happens that it often passes to the wrong team, and lays up into its own basket, which is weird, but hey, it's a piece of software. It's not very much like us. Hardly surprising that it does a bunch of things we don't do.
You (and most people in this debate) are stuck in this weird frame where language is equated with intelligence. In my analogy, three pointers are sentence completions. They are an advanced part of the game (and ever since statisticians ruined my favourite sport you won’t win without them) but they are hardly the whole game.
In my world, dribbling is edge detection. Passing is folk physics. Swiping for a steal is running a folk experiment to see how the environment responds.
No matter how easily GPT can make open threes, it will be even easier for Gary Marcus to step in for the block, because he knows the fundamentals. This is why Scott won’t take his bet, and why language modelling is a dead end wrt AGI.
I said: "No one thinks that human language *is* human intelligence."
You said: "You are stuck in this weird frame where language is equated with intelligence."
Wut?
You can say that all you want, but your commitment to the view that ”progressively passing language-based intelligence tests implies the tech is approaching true intelligence” suggests you don’t believe it, and you yourself are stuck in that frame. Again, Wittgenstein’s ruler. The tests are dumb. If you’re interested in the nature of intelligence, computational vision and reinforcement learning have some lessons for you, but language models don’t.
If you’re going to be relying on Wittgenstein, you should also note that his central idea is that you can’t understand a language without understanding the whole way of life that goes with that language. “If a lion could speak, we would not understand it”, but presumably that means that if something can speak understandably, then it must no longer be as alien as a lion but must actually have understood our way of life.
Sorry, because I think you have some valid points, but language modeling is not a dead end wrt AGI. It's a necessary, but not sufficient, component. GPT3 is showing it can do more than we expected, but I still expect arithmetic to be delegated to a calculator module. (Our ancestors could clearly handle small integers, probably up to 5, perhaps higher, without language, but the didn't have predesigned calculator chips that they could use. They may not have had language, but I think they did the number handling with pattern matching.)
No we're not just the words we say. But in this context, all we have is words. But these words transfer meaning. The meaning is created by our thoughts. These thoughts form from our desires and ambitions.
If our desires and ambitions were about beauty, we'd be hanging about in makeup forums discussing eye shadow and mascara now wouldn't we.
And here we get a central idea. I deny that words transfer meaning without the contextual embedding. What's interesting is how much the pattern structure carries.
To me meaning requires a multi-modal connection. You touch the stove and get burned as you are told "No! Don't do that!". The "No!" is associated with the burn from the stove, so you learn that it means that it's something to avoid. This gives it meaning. An arbitrary association that just turns up frequently doesn't have any meaning without the referent outside of itself.
So for me GPT3 cannot emit meaningful text. But it can emit text that has patterns that match meanings I have previously learned. It's really just a "Super^n Eliza", because it's missing the external referents. And it can appear intelligent for the same reasons that Eliza could.
> GPT is designed explicitly as a language emulator. No one thinks that human language *is* human intelligence. So it's weird to be applying intelligence tests to a thing that is just language.
Your conclusion is right, but your premises are wrong - a very large number of people honestly believe that human intelligence and human language are the same thing. Based on this article, Scott might be one of those people.
"a very large number of people honestly believe that..."
Yeah, that could be right. Or at least they haven't thought it through enough to clearly distinguish between the two.
"Scott might be..."
I don't think this is a good conclusion to draw. Nothing in the article says that, and it seems unhelpful to go around ascribing views to people that they don't hold. See above, where dlkf thinks it's OK to just flatly contradict me on my own views. It would be more helpful to just take a step back and look at what Scott says on its own merits. (To the extent that that's possible!)
> See above, where dlkf thinks it's OK to just flatly contradict me on my own views.
Eh... your statement that you don't believe something is evidence, but it's not proof; it can't immunize you against the conclusion that you really do believe in the proposition you disclaim. If you go around making arguments that rely on a particular premise, an explicit disavowal of that premise will mostly convince people that you don't make sense rather than that you really don't believe in the premise you repeatedly rely on.
There are two distinct claims here. The first one, which many consider plausible, is that mastery of the language is an "AI-complete" task. A much more dubious one is that a predictive model trained on nothing but human-generated text could eventually attain that mastery, and I'm somewhat suprised that Scott assigns such a high likelihood to it.
It’s how most of our education system works, apart from a few field trips and some science labs.
Yes, and pretty much everybody agrees that it is terrible at achieving its stated goals. And beyond that, it's a well known fact that we can't transfer all necessary knowledge in printed form, as demonstrated by exceptional scientists usually having exceptional mentors.
Well, if nothing else, (most?) people can rotate mental models of a 3D object in their head, which I consider to be exercising a type of intelligence. It is in principle possible that training on text could eventually generate this skill, but it seems like it would be very difficult and inefficient to try to create it that way. But I may be wrong. I didn't expect GPT-3 to be able to manage "above" and "below" as well as it did. I'd still expect a model which had also been trained on visual data, and "grew" the equivalent of a visual cortex to be better at it than a purely text-trained AI.
Why does it matter it's trained on only text?
Text just happens to be its sensory input of the world
Same as human vision & hearing just happen to be our sensory input of the world
Couldn't some alien with some other sensory input of the world, claim its absurd that human with such a narrow perception of reality could possibly truly master language?
What is so categorically distinct about the way GPT experiences the world , and how we experience the world?
I'd say that the biggest difference is that we can interact with the world and observe the results of these interactions, whereas GPT's input only goes one way. So, learning to accurately model the world, and to create detailed coherent plans would be much harder, and without those abilities there's no true mastery of language.
That's not quite correct. What it is given isn't just words and language, but patterns of usage of the words. These patterns contain embedded within them a reflection of the things they are used to describe. I can see a GPT style intelligence eventually detecting inconsistencies in the meaningful use of the words. This is, of course, a lot different from detecting the true nature of the universe being described. If just about everyone either asserts a belief or doesn't deny it, then it will be accepted as fact.
Of course, there's a catch. A lot of people do the same thing. I'm not sure we consider that intelligent, but to an extent it's necessary to get along on our social groups. Even just doubting an unobservable thing, while quite reasonable, could often get one exiled or dead. So is, e.g., expressing belief in a god intelligent, and expressing doubt unintelligent? Not by my definitions, as to me that would be a statement about your goals and motivations rather than intelligence. But one can predict that GPT3 would express a belief in "God", given what is expected to be the training set.
I don't think I believe in a thing that is a "pattern of usage of words" but isn't language.
And I don't think that GPT can "express" anything, certainly not beliefs. It doesn't have beliefs. Below, replying to Michael Watts, you said GPT cannot emit meaningful text - I agree with that. It's not encoding meaning at the moment. But here you've talked about its texts as though it is encoding meaning - that's what "expressing" is. This slippage between (a) knowing that GPT doesn't actually have knowledge, beliefs, and intentions but (b) treating its utterances as though they do represent knowledge, beliefs, and intentions is what I mean when I say "odd".
GPT cannot emit meaning, but meaning isn't simple. What something means depends on the perceiver of the message. So GPT can emit text, and people reading that text can perceive meaning, because the patterns of words that are emitted seem to be the same as meaningful patterns that have been previously encountered (by the reader).
So, yes, GPT cannot express beliefs. But it can emit text that a reader can interpret as expressing beliefs. It's true that GPT doesn't hold the beliefs that were expressed, because to GPT the text is without meaning. It's only habitual patterns. But the people who emitted the text that GPT was trained on were expressing meanings when the emitted the text, and part of the meaning is embedded within the patterns. It's latent, rather than active, so GPT isn't attaching meanings to it, but the reader will almost always try to attach meanings, and if they are successful, then they will have the perception that GPT emitted meaningful text. And they'll be right, because it was meaningful to them, even if not to GPT.
To make a part of this argument explicit, imagine that you can't read Sanskrit, and have a page of it in front of you. Assume that the writer of the text considered it highly meaningful. Do *you* perceive the meaning? I assert that the meaning of text is only present when a pattern is present in the message, and the reader of the text perceives a meaning to be present. This doesn't say anything about the author of the text.
I love reader meaning. Reader meaning is the best kind of meaning. But it's the wrong kind of meaning to be thinking about in this context.
We are trying to work out whether GPT - the speaker - has anything like "intelligence" or not. As part of that discussion, we are talking about whether the things it says have meaning. In this discussion, reader meaning is not the relevant question. The relevant question is speaker meaning.
Speaker meaning is also a thing. The relationship between speaker meaning and reader meaning is complex (hence the complexity of language and semantics). But we don't really need to worry about that here. The only question for us is: does GPT mean anything when it speaks? And I think you and I both agree that it doesn't. In the context of this discussion, it would be useful to maintain great clarity on that issue.
Well, I answered that, too. I assert that meaning for language can only exist when language it tied to external observables. Since GPT doesn't have the ability to do external observation, it can't have meaning, in that sense.
Yep, we're in complete agreement on that.
Human language refers to things in the world, so when you go into enough detail, emulating human language requires you to be able to model things in the real world as well.
Some day in the future, someone will say "Yeah, GPT-N seems smart, but it can't answer basic questions about quantum mechanics," and then we'll add another gazillion parameters to the model, and then GPT-N+1 will invent the Theory of Everything in order to generate text that can fool a quantum physicist into thinking it's a human.
You’re assuming the thing to be proven. In terms of world modelling GPT has yet to go from 0 to 1. If we were at 1 today, I’d find the alarmism more plausible. But today GPT cannot fool a human into thinking it can reason about simple matters.
GPT has *some* ability to model the world. Scott's last post was all about using it to answer simple hypotheticals about the world, like "what would happen if you closed your eyes and walked backwards down the stairs?"
It's always vital to distinguish the surface from the underlying.
"GPT has *some* ability to model the world." - no, GPT has some ability to say things as if it were modelling the world. So did ELIZA, and any other chatbot. But because we know how it works, we're pretty sure that there's no world-modelling going on, by which I mean there are no patterns inside the software that correspond to real things in the world.
To put that concretely, imagine a situation where you're holding two apples, and a one apple, and no apples. In one of those situations, you picture two apples; in the second, you picture one apple; in the third situation, you don't picture any apples. The mental object "apple" is not in the third situation. For GPT, by contrast, all internal processing of those three respective situations always contain the digital pattern (it's equivalent of a mental object) associated with the word "apple". The word "apple" is present in all three.
That's the kind of thing that I mean when I say that GPT is not modelling the world.
A text-to-image AI, by contrast, does create a model in a kind of a world (a 2D pixellated world), connected to but separate from the language of its prompts.
You are on for #5! I will try to write up some thoughts on scaling and paradigm shifts in a longer reply, in next few days.
For #5, are you sure you're not on the same side (but possibly with differing levels of certainty)? I interpret Scott's 40% confidence in #5 as saying he believes it's 60% likely that a further paradigm shift will be required. (edit: though the framing of "here are some things I do believe" confuses me, it still does seem like he's coming down in favor of a paradigm shift being needed)
we aren’t hugely far apart but certainly i would happily take his odds; i promise to explain more at length (and also clear up some places where he got my view wrong); my main focus will be on what i think the shifts needs to be, rather than the #s per se
I wonder if the paradigm shift is about metaphor, which is more than pattern matching.
Creating them and understanding them.
Culturally specific but a commonality emerges if you dig.
Language IS metaphor but at its heart a pretty rudimentary one. But it allows us to create and understand much more elaborate ones.
I’m a bit skeptical of a learning system based on pattern recognition making that leap.
“A dictionary is a mighty good doorstop”
How do you pull out all the inflections of that by pattern matching?
A heavy object, says pattern matching, completely missing the far deeper shades.
The recent PaLM model has shown itself to be quite adept at analyzing jokes, I think there's a good chance it would understand the nuance behind using a dictionary as a doorstop. https://thenextweb.com/news/google-palm-ai-sucks-at-telling-jokes-but-great-at-analyzing-them
I wonder. You may well be right, but two of those jokes pivot on very concrete wordplay and the third on a common joke “form” (-X walks into a bar. e.g.)
For instance, here is a Polish joke from the early 1980s.
Two men are in a bar talking. One says what’s better, the Russian space station or the American space station?
The other man says, oh, the Russian space station.
Why? Asks his friend.
The other man shrugs his shoulders and says, well… You know… Because.
Mind you, most people who have never lived in an Iron Curtain country don’t get the joke either.
The joke is that they live in a constant state of surveillance and can't speak out in favour of America for fear a friend would report them to their local state agent?
My only experience of life behind the Iron Curtain is text, so if I'm right I'd update my odds a language model could understand it too
The formulation used (about paradigm shifts being incorporated in those models) is subtly but substantially different than the notion of the paradigm shifts being required.
If at some point of 2024 we discover a paradigm shift that doesn't change what is possible but simply allows getting the same results with 50% less computing power if you structure everything a bit differently, then every serious future model will incorporate that paradigm shift, but in this example it's not really *required* for those models, people would just have to pay twice as much for them otherwise - which nobody will in practice if the discovery is there.
Thank you for engaging with Scott, and taking up the bet!
One reason Marcus believes that symbolic models are necessary, in part, but are not the whole story, is because he was trained in them. He is thus biased. I share his bias. I was trained in computational semantics in the mid-1970s, the salad days of symbolic computing. I went so far as to sketch out and publish a model of the underlying semantic structure of a Shakespeare sonnet. https://www.academia.edu/235111/Cognitive_Networks_and_Literary_Semantics
Nonetheless I have had no trouble seeing that GPT-3 represents a phase change. Artificial neural networks are here to stay. Moreover, I agree with Geoffrey Hinton when he says the brain is large neural vectors all the way down. Which means that the brain implements symbolic structures with neural vectors. The young child who is beginning to learn language has no symbols in their head. Language symbols exist in the external world in the form of utterances by others (directed at the child) and, increasingly, by the child themself. Lev Vygotsky (who wrote a book on language acquisition, Thought and Language, in which he argues that, over time the child internalizes language. First the child talks to themself and then actual talk becomes unnecessary. The child is now capable of inner speech. Here's a sketch of that, https://new-savanna.blogspot.com/2014/10/vygotsky-tutorial-for-connected-courses.html
Once that internalization has taken place we have symbolic thinking with neural vectors. I'm including a slightly reworked version of that story in a longish paper I'm working on in which I spell out these ideas in some detail in some detail.
As far as I know most of the researchers working in artificial neural networks have had little or no training in classical linguistics or cognitive psychology. What they know is that the old symbolic systems didn't work that well while these newer systems seem to be pretty good. That gives them a different and very strong bias. But they don't know much of anything about how humans actually solve these various problems. Their knowledge is dominated by what they know about creating learning architectures that allow computers to do cool things that convincingly mimic human behavior – and in some domains (e.g. chess, Go), exceed human capacity – but they don't really know what the resulting models are doing when run in inference mode (though they're beginning to figure some of that out).
What we have is bits and pieces of various kinds of knowledge and a lot of things we don't know. When it comes to predicting the future, no matter what your current knowledge and biases, the future is dominated by THINGS WE DO NOT KNOW. Arguments about what will or will not work in the future are just attempts to shore up one's current biases. Beyond a certain point, that job is a waste of time. Why? BECAUSE WE DON'T KNOW.
👍
"Luria: All bears are white where there is always snow. In Novaya Zemlya there is always snow. What color are the bears there?
Peasant: I have seen only black bears and I do not talk of what I have not seen.
Luria: What what do my words imply?
Peasant: If a person has not been there he can not say anything on the basis of words. If a man was 60 or 80 and had seen a white bear there and told me about it, he could be believed."
The last sentence here makes it pretty clear to me that they understand the answer the experimenter is looking for is "white", so I don't think this is a failure of logical reasoning. Rather they're trying to make the (completely valid) observation that blind induction is not always reliable. Jut because all the bears you've seen in snow have been white doesn't mean that's true everywhere else there's snow.
I think if anything their failure is to understand the concept of an absolute generalization. When the experimenter says "All bears are white where there is always snow", the peasant takes this as a claim about the real world and correctly infers that the experimenter can't know that's actually true.
The camel example can be explained the same way.
The other two seem more like a real failure of basic reasoning, though the last one could also stem from a simple misunderstanding of what the word "animal" means. (Using "animal" to mean "terrestrial vertebrates" or similar is not uncommon.)
FWIW, David Hays and I discuss the Luria work on pp. 306-307 in The Evolution of Cognition, https://www.academia.edu/243486/The_Evolution_of_Cognition
Yeah, this is one of those examples that Scott hauls out every once in a while that I wish he wouldn’t. The responses garnered are by turns very canny (bears and camels) and obvious category misunderstandings, including straightforward confusion from the respondent about what the questioner’s definition of terms are. This is not evidence of stupidity or poor reasoning. Rather the reverse.
There's been quite a bit of research on folk taxonomy. One of the things you find is that preliterate peoples (like those Uzbeck peasants) lack words corresponding to plant and animal. They certainly know that chickens are more like cows than they are like pine trees, but that knowledge just hasn't been distilled into single words. That can lead to confusion and the need for work-arounds.
FWIW, in many languages the common meaning of the word "animal" is something similar to "non-human terrestrial creature that walks on legs". Fish and birds are not "animals" in this sense. Yes, they are animals in the biological sense, but Uzbek peasants probably don't know that.
Even for an Uzbek biologist who was an expert in taxonomy and genetics, it would be perfectly valid to say "fish and birds are members of the Animalia kingdom, but they are not hayvon". (Hayvon is Uzbek for animal, according to Google Translate.) Taxonomists do not automatically get the final say in deciding what an ordinary word in a given language means; if Uzbek peasants say that it means "non-human creature that walks on legs", they're not wrong just because that category does not map to a specific taxonomic group.
https://slatestarcodex.com/2014/11/21/the-categories-were-made-for-man-not-man-for-the-categories/
Which is to say it is not semantically equivalent to "animal." If Google translate says that "Hayvon" is Uzbek for animal, but the extension of "Hayvon" is not the same as the extension of "animal", then it's a bad translation and there may not be a good translation. By the way, what investigation does show is that, the taxonomic categories that preliterate cultures do have, also map onto biological taxonomy rather well. They just don't have some categories.
Here's a "classical" source article on the subject: https://anthrosource.onlinelibrary.wiley.com/doi/abs/10.1525/aa.1973.75.1.02a00140
English speakers don’t usually use “animal” to refer to birds and fish either. They do when prompted to be precise in language, but if you’re out in the forest with little birds flitting around you might say you haven’t seen any animals if you haven’t seen a deer or rabbit or raccoon or something.
Was Laria speaking Russian or Uzbek when he was questioning the peasants, or were translators involved? I imagine there's some room for misunderstanding when you're speaking Russian to someone speaking Russian as a second language, and even more if it's Russian -> translating Russian to Uzbek -> Uzbek and then the answer goes the other way.
According to this English to Uzbek site:
an animal ::hayvon
wild animal ::yirtqich hayvon
farm animal ::xo'jalik hayvon
domestic animal ::ichki hayvon
So it could make all the difference if Laria thought he was asking "is a fish or a crow an animal (hayvon)" and the peasant thought he was asking "is a fish or a crow a domestic animal (ichki havyon)"?
Which is to say, the word, whatever it is, is NOT equivalent to our word for animal. It's more like "beast."
English speakers don’t usually count birds and fish as animals either, unless they’re being particularly careful, any more than they count humans as animals. “Animal” is just a Latinate term with a precise extension that is still based on the same prototype as the Germanic “beast”.
So if I say "Noah had two of every animal on the ark" English speakers would understand me to be saying four-legged furry creatures that walk on land? Here's the FIRST definition of animal in the dictionary on my Mac: "a living organism that feeds on organic matter, typically having specialized sense organs and nervous system and able to respond rapidly to stimuli." Here's the THIRD: "a mammal, as opposed to a bird, reptile, fish, or insect." Word meaning is tricky.
I was wondering if the Uzbek word that corresponded to "animal" meant something more like "domesticated animal". In that case, then of course crows and fish aren't "animals".
It reminded me of a anecdote I read online years back about a teacher telling a story of when she started off in a primary school in a rural part of England; she drew a (conventional) sheep and asked the kids in her class "what is this?"
No answer, which surprised her. These were kids who, if they didn't come from farms themselves, lived in the country and so they certainly knew what a sheep was. Finally, after a lot of prodding and hinting, one of the kids raised his hand and said dubiously "Well, the *face* looks like a Swaledale..."
That was the mismatch of expectations there; she drew the usual "fuzzy white sheep" and expected them to say "It's a sheep", they were from a background of "Well of course it's a sheep, she must mean 'what breed is it?'"
Yeah, this felt a lot more like an expression of a "don't trust the experts" sort of sentiment more than a lack of intelligence.
---
Another way of looking at it may be that the whole "game" of "let's suppose incorrect counterfactuals and draw conclusions from them" just is foreign to them, sort of a cultural disconnect.
It's the sort of exercise that's drilled in in formal education and feels "obvious" to anyone from that background - it's not they lack the innate cognitive abilities to do it, but it's a "logic game" that's more culturally-ingrained than it may seem at first glance.
Yes.
Seconded.
Also, the peasant may well perceive the counterfactual game as a possible _trap_. I, personally, tend to respond to moralists that way.
James Flynn found that the effect named after him was highest for these kinds of highly abstract questions, suggesting that they're one of the most culture-bound aspects of human intelligence.
Makes sense to me.
The refusal to generalize ironically reminded me of this old joke about mathematicians (here copied from Wikipedia):
An astronomer, a physicist and a mathematician are on a train in Scotland. The astronomer looks out of the window, sees a black sheep standing in a field, and remarks, "How odd. All the sheep in Scotland are black!" "No, no, no!" says the physicist. "Only some Scottish sheep are black." The mathematician rolls his eyes at his companions' muddled thinking and says, "In Scotland, there is at least one sheep, at least one side of which appears to be black from here some of the time."
Who's truly failing at basic reasoning here, the peasant or the mathematician?!
> The other two seem more like a real failure of basic reasoning
I mean, either that, or the language barrier.
That said, if I were a peasant living in a post-Soviet country, and some official-looking guy came to me asking seemingly irrelevant questions, I would be *very* careful about how I answered them. I too would avoid committing to anything unless I absolutely had to. You never know what the plainclothes man will write down in his little book about you.
Luria’s study was published in 1976, so during the Soviet Union not after. Your point still holds up though.
And the work was originally carried out in the 1930s, I believe.
If the peasant asked "Are these questions part of the current purge?" would they have shown up in the recorded data? And ... would they ever have been heard from again?
"Peasant: If a person has not been there he can not say anything on the basis of words. If a man was 60 or 80 and had seen a white bear there and told me about it, he could be believed."
That's a common epistemological belief among preliterate people. Dan Everett, the linguist, has written extensively about his experience among the Piraha, of the Amazon Basin. He originally went there as a missionary to translate the Bible into Piraha and to convert them to Christianity. He found that whenever he told them about this Jesus fellow they'd ask him, "Have you met him and talked with him?" Everett would reply, truthfully, that no he had not. "Well then why should be believe what you tell us about this person you've never met? Why should we believe that he exists?" After several years of this, Everett decided that they were right, that Jesus had nothing to offer them, or him either. And so he became an athiest, but continued on as a linguist.
On these confidence levels you give, Scott. To some extent, sure. I have various levels of confidence in things I write and may even give some explicit indication of those levels. But you're giving percentages whose values take advantage of the whole range between 0 and 100. What's the difference between 65% for number 2 and 66% for number 4? Is that real? That strikes me as overkill.
You seem to have a three point scale: level 1 is, say, 0-45, level 2 is 46-67, level 3 is 68-100. Any precision you indicate beyond 1, 2, and 3 strikes me as being epistemic theater. It's tap dancing and hand waving.
Semi-counterpoint: I agree with you that 65% vs 66% is overkill (I think he just thought "two-thirds" and inconsistently translated them into rounded percentages). But I disagree that his scale is three-pointed; it's an annual SSC / ACX tradition for him to do ten-point scale predictions and score them once the year is done, eg https://slatestarcodex.com/2020/04/08/2019-predictions-calibration-results/ (see the graph at the bottom where he calibrates his confidence levels vs how often they actually turned out right).
If it's a ten point scale, then stick to it. I note that I'm also thinking of something that Holden Karnofsky wrote where he gives confidence values of 1% and 10% and not a word about where those numbers came from. I think he just means small and very small. The numbers don't mean any more than that.
Doesn't sound like Holden (although it does sound like some other people I know/read).
I think they mean “small” and “very small” but likely also something like “7-12%” and “0.1-2%”. (Or maybe slightly different ranges. But the point is that I suspect that 10% and 1% are the right order of magnitude for what they mean and not just rhetorical flourishes.)
Well "order of magnitude" is a lot looser, and it's easy to say. When you say 10% and 1% you seem to be implying double-digit precision.
Without venturing to guess about what Scott was/ wasn't doing here, it's worth noting that in Tetlock's experiments, super-forecasters routinely make high-resolution probabilistic judgments, and that rounding their scores hurts their performance.
Interesting.
https://slatestarcodex.com/2016/02/07/list-of-passages-i-highlighted-in-my-copy-of-superforecasting/
search for "granularity"
thanks, but what's the primary source being quoted in that post? Are all those quotes from the same document? Ah, I just noticed the title of the post.
In any event, I suspect that precision is wasted in forecasts about what an as yet non-existent AI device can or cannot do by a certain date.
As B_Epstein said, except that the 66% one was my mistake - in some sense 65% feels "rounder" than 66%, in another sense 66% feels "rounder", and I guess I wasn't operating off of a consistent world model and tried to use the "rounder" number each time and my brain made opposite decisions as to which one that was. Sorry!
No problem.
Our brain is hard-coded for language acquisition. Which is, fundamentally, assigning an arbitrary label to a set of sensory experiences (eg, this sweet crisp round thing is called an "apple" or "mansano" or hundreds of other arbitrary sets of specific compressions of air (sound)).
That seems like a fundamental part of being able to do other abstractions?
It seems that our brain can't be very much hard-coded for language acquisition compared to e.g. chimps, almost all of the "hardware structure" is the same with barely enough "evolutionary mutation space" for some small adaptations - but yes, it seems plausible that those small adaptations which were evolutionary useful for social/language purposes would be important for being able to do other abstractions much better than e.g. chimps do.
> It seems that our brain can't be very much hard-coded for language acquisition compared to e.g. chimps
Huh? Chimpanzees are universally unable to acquire language; humans occasionally fail to do so, but that failure constitutes an extremely severe mental defect and occurs at rates much too small for most people to ever notice.
Yes, but the fact that we can acquire language and chimps don't (though their own communication is quite complex and there are some semi-controversial experiments with chimps learning sign language) does not imply that "Our brain is hard-coded for language acquisition.", quite the opposite.
First, it's plausible that this difference in ability is purely caused by a quantitative, not qualitative difference in brains; and second, given the relatively small evolutionary gap it's not plausible that we have a lot of highly specific "hard-coded" brain machinery compared to chimps; there is clearly space for some *small* adaptive mutations but any language deficiency that chimps have IMHO is some evidence that language acquisition simply requires more generic "brainpower", not some very specific wiring of the brain, since we share the same brain wiring principles as chimps but have more neurons available for audio processing.
What is the important difference between what you say and “hard-coded for language”? “Hard-coded” doesn’t have to mean “grammatical rules are innate” or “categories of noun and verb are innate” (I don’t think even Chomsky endorses those claims) but could just mean that there is some important hardware structure that easily canalizes into learning of language.
> eg, this sweet crisp round thing is called an "apple" or "mansano"
I feel compelled to point out that the Spanish word for "apple" is manzana - manzano is the tree, not the fruit.
The conditional hypotheticals bit astonished me and I wish there were a more credible reference for that.
I was feeling very skeptical of that whole comment, but I will say at the end, "Recursive thinking and recursive knowledge seems genuinely hard for people of even average intelligence." -- well, based on having taught induction & recursion to students at some top colleges/universities (and having to re-teach it in higher level classes), the most general version can be quite difficult even for fairly smart people the first (or second) time they learn it.
I agree.
But the claim does make me think a lot more about why the movie Inception had the particular cultural resonance it did. Maybe this particular nested structure really is a lot harder for people than I realize, and this movie was one of the first things that presented it accessibly, so it comes to stand in for any recursive structure anywhere.
At what level of complexity will the AI start asking us questions? Apes can interrogate the world. Could GPT-3?
It could write a response that ends with a question mark. But the way it operates, it has no memory of anything that happens after training is done, so each prompt you give it is starting over. It can’t interpret your text as a response to its previous question unless you include the whole conversation so far in your prompt. (But then it presumably could.)
It feels to me that the current path will not end up with "agentic" AIs. What if we create a program that completely passes the imitation game, BIG-bench, etc.; a program that has sufficient complexity that it appears to be conscious (and tells us it is, and is as convincing as a human when a human tells us it is)... but that program is just an inert input -> output generator like GPT-N or DALL-E?
This seems weird and alien! The intelligences we interact with, and read about in sci-fi, exist independently of us. They have goals that are not "patiently wait for someone to type in a text box, then type a response". They run persistently; they accumulate memory, and experiences, and change over time. They have spontaneous outputs based on the world around them. I don't know how we as a society should interact with something that tells us that it's conscious, and wants to live... but only if we ask it questions framed in the right way.
What would it take to get from the current marvels we have, to something more agentic/human-like/sci-fi AI-like? The missing ingredients I can see are long-term memory and evolution; more sensory inputs so they can passively pick up information from the real-world environment instead of getting strings fed to them; and some kind of motivation to initiate actions instead of sitting in a dark room being unsurprised by the lack of input. Are these things anyone is working on integrating with the intelligences displayed by GPT-N? Are they hard, or easy, or...?
I realize from the perspective of AI safety this is playing with fire, so, feel free to answer in the vein of either "yes, someone is making progress on that sort of thing" or in the vein of "yes, THOSE FOOLS are doing that sort of thing and will kill us all". I just want to know...
Yes, those fools are doing that sort of thing and will kill us all.
SayCan: https://say-can.github.io/
Memory Transformer: https://arxiv.org/abs/2006.11527
Nobody's doing online learning yet, at least as far as I can tell, but it wouldn't surprise me if that was the sort of research that was not observable because worlds that go down it fall quickly.
Oh no
A key aspect is that agentic AIs is simply an orthogonal direction and for many practical problems which we want to solve, agentic AIs aren't necessary so we definitely want to (also) explicitly pursue non-agentic AIs simply to avoid any unnecessary cost and complexity.
Not all SF uses such tropes of AI. Charles Stross in Rule 34 posits a highly believable "AI" which is simply a giant optimization engine cranked up to 11, yet is largely indistinguishable from a conscious superintelligence because it has access to lots of data and fast computation. This scenario seems much more likely to me than GPT-6 suddenly attaining enlightenment through scale.
Wait, what? Wouldn’t the phrase “a giant optimization engine cranked up to 11” perfectly describe a GPT-6 that was more-or-less GPT-3 plus a lot of scale?
Yes, precisely. My point is that GPT-6 doesn't need to attain consciousness to have superhuman capabilities in some areas. I'm responding to the statement that SF usually relies on a conscious AI; not all SF does so.
Ignore the previous thing, I figured out how to un-collapse everything and now I remember the source of my confusion. You said:
> this scenario seems much more likely to me than GPT-6 [to become super-intelligent] through scale
But AFAICT it can’t be more likely because it’s basically the same scenario. It’s like saying that “some book I read had a billionaire building a reusable rocket company to try to colonize to Mars, which seems more likely to me than Elon Musk’s plan”.
Read my comments again. I am saying a mysterious awakening into consciousness by a bigger GPT seems less likely than it potentially outperforming humans yet without being intelligent in any human sense.
Oh, I thought “attaining enlightenment” was just another more poetic way of saying “largely indistinguishable from a conscious superintelligence”. If you meant the two to contrast then it makes more sense. Thanks for clarifying!
This is so interesting thanks! Love the point about the hands and dreams. I think a point you're missing though it amount of training data. Sure humans can make dumb mistakes but they haven't read a fair amount of everything ever written. GPT-3 needs way more training data than GPT-2 right? That seems to be where your analogy with the human brain breaks down to me. Sure we could scale GPT to have an equivalent computational power of a human brain but how are we going to scale the amount of training data up by 1000x? while keeping it sufficiently diverse that it isn't almost identical to something already in the training data? Also doesn't the fact that it needs these vast data sets (orders of magnitude larger than what a child is exposed to) indicate that they are learning in a fundamentally different and less sophisticated and more brute force statistical inference type way?
Another example, alphago zero has to play millions of games against itself before it is better than any human. If a human got better at that rate they would be idiotically dumb. So we are still a long way from superhuman at Go it seems to me. We are just taking advantage of faster computation. Rather than using a more abstract way of learning.
I think if a human learned that slowly from their first few thousand games they would be dumb. But for a human to learn at all from their hundred thousandth game would be unusual.
Don't you think though that most of human "Go" training is internal? We run pieces of games by ourselves. Additionally, the vast majority of the last XX% of games used to train an AI must be effectively useless. If the AI lacked time and computational power, it might just filter out the games it felt were likely to be unrewarding. At that point, the internal training of a human and the training of the AI seem far more similar.
yeah good point it could be more similar than the raw number suggests. But the vast difference just makes me think something fundamentally different and higher level and less statistical inferency is going on for humans, but I have no idea how to prove this.
Yes which kind of indicates something fundamentally different is going on to me.
Are you sure about that? That “Zero” in the name is important!
An exceptional human might be able to learn Go by age six, but not if you just lock it up right after birth in a blank room with a Go board and only feed it when it makes a good move.
Even if you put humans through a “pre-training” phase where you take care of them, teach them to walk, talk, and socialize for six or so years, and *then* you start teaching them Go, they still won’t get very good at it. Particularly if by “teach them Go” you means just “reward them for good games”, instead of teaching them the accumulated knowledge of strategy and whatever.
Edit: Also, why would you count number of games played, instead of time spent, or energy consumed? By those standards, humans’ Go learning skills are just barely distinguishable from monkeys.
Yes, the question of scaling rate seems quite important here (as elsewhere in AI; but also hard to reason about satisfactorily). No one in the present discussion is arguing there's a specific X that GPT-N will never do, but if each GPT-(N+1) can be flummoxed by adding 1 more complication/distraction to the query--which takes an extra hour of playing around compared to a few years for GPT-(N+1) to be developed and trained--then I can see Marcus's perspective that there is something fundamentally underwhelming here.
That assumes the scaling in conceptual competency is somehow "linear", and thus significantly limited by available compute power, time, and training data. But if it's actually "exponential", then who knows? The way a 5 year-old child develops into, say, Scott Alexander (with an approximately fully developed world model) feels "exponential"ish.
Yeah it is very hard to reason about. The other thing is if it is just about scaling current models then all humans should be pretty similarly intelligent right? But it is clearly not as the examples given show. It seems unlikely that GPT type models trained with similar amounts of data and compute power would show nearly as much variation as humans do in intelligence, which also seems to imply to me that something very different is going on.
Megawatts of energy you say...
I noticed that too. It's not the kind of error GPT3 would make..
I laughed out loud at "megawatts of energy (perhaps more)". It's like: "How far is it from Athens to Madrid?" "Oh, hundreds of miles per hour -- perhaps more!"
Isn't an AI that lacks cognitive models still worth thinking about?
Transistor-based computers transformed everything in the world (as did the internal combustion engine, the printing press, and the wheel before them). If we're concerned about AI causing mass technological job displacement, or unleashing Sorceror's Apprentice style disasters, then I don't see why Marcus's objection is that important.
The most interesting prediction would be "When will a GPT derivative be able to make first incremental improvement in its own structure, including self-training?" Because that will be the day it poofs without ever becoming "Marcussian."
_Mostly_ agreed, but it gets tricky. Self-improvement is certainly important, since it creates a positive feedback loop. But consider an optimizing compiler compiling itself. Yes, it makes the next binary faster, but that is where it ends. Some positive feedback loops run away, but some saturate.
Yeah, that's a very good point. My assumption of ever recursive self-improvement guaranteed to be unbounded is not necessarily correct. I would like to see some research in that direction: what are the conditions for the recursive self-improvement taking off vs fizzling out.
Imagine, for a moment, a student taking test, after test, after test, and passing them; however, he student's responses to questions come verbatim from the book.
In the next text, the professor changes the wording for the questions: "In your own words..."
Suppose the student responds with a verbatim quote from the book. Okay, suppose the student responds with a quote that combines the material from two books in a way which is logically sound. Suppose it is three books. And so on and so forth.
Here's the thing: At some point, the student's answer cannot be meaningfully said to be taken from any of the books; it is, by the standards we measure human responses by, the student's own words.
And yet I find my attention drawn to a curious detail: The material is combined in a way that is logically sound, and this is evidence, in a human, that they understand the material, because the alternative - that they understand the logical relationships between the words while not understanding anything else - looks unthinkable.
It must be thinking, because it is creating the same kind of output as thoughts.
Suppose, for a moment, that AI is possible; that the brain can be modeled as some set of algorithms and data structures. Can the output be predicted?
Yes; we can predict what other people will do. We work really, really hard at being predictable. What I am writing, right now, is designed to be predictable on a number of dimensions that you aren't even aware of. uamtypfinfndhatthisduskanexampleofnutwkrnghrd.m I took away a single dimension of predictability, and "I am typing and this is an example of not working hard" became that; it's still extremely predictable. The fact that the word "predictable" occurs in this sentence, for example, greatly increased the odds of something like that being in there; GPT-3 has access to text with typographical errors, and has access to information that would allow it to infer a relationship between letters that we would understand to be proximity on the keyboard.
"So what?"
So there's a reason the teacher added "In your own words" into the questions.
People are perfectly capable of understanding the logical relationships between words without understanding "anything else". We have a real-world example from Richard Feynman's experience with science education in Brazil. https://v.cx/2010/04/feynman-brazil-education
All the students were fantastic at memorizing and regurgitating whatever was in the textbooks, but were completely unsuccessful at applying any of the info to hypotheticals on what would happen in an experiment.
I have no idea if the students would have been able to put things "in their own words", but if the classes were structured that way, I suspect they would have been able to do so even while exhibiting the same flaws.
Fascinating information about general capabilities varying with IQ, but the stuff about prison populations seems quite off. Perhaps some of these people have trouble with *contradictory* hypotheticals, asking about things that obviously did not happen. Instead, I'll bet serious money that, if you asked, "What would you do if you found out your wife cheated on you?", you'd get a clear answer, not "are you telling me my wife has cheated on me?" Or, more to the point, "if tomorrow, they didn't give you breakfast, how would you react?" You'd have to be exceptionally impaired not to be able to imagine that, and answer accordingly.
Why is no one talking about qualia and incorporating non-verbal experience? Believing that everything we could say about sunsets is somehow in the convex hull of what has already been WRITTEN about sunsets, and can be inferred by machines who had never actually seen one, seems entirely off-base (aka, I'm with Gary on this one).
The greatest poetry is often built on metaphors that are outside our prior experience as readers, but somehow seem entirely "right". Or, as Anne Sexton put it, "making trees from used furniture".
I agree
Your prediction 5) seems overconfident to me. Has anyone tried hooking up a text-prediction algorithm to e.g. a camera? Does that idea make any sense? Does anyone have any idea how to join a text-prediction algorithm with an image-recognition one, and say "these strings correspond to these images"? Wouldn't such a combination still be limited to a rectangle of pixels and a bunch of strings, with no way of distinguish a sequence of events actually happening from a picture book?
Perhaps more importantly, how good would it be at reasoning in new situations, unlike those that appear in its training data? Maybe those are the kinds of questions we should be focusing on.
> Does anyone have any idea how to join a text-prediction algorithm with an image-recognition one, and say "these strings correspond to these images"?
Yes, this has already been done: https://www.deepmind.com/blog/tackling-multiple-tasks-with-a-single-visual-language-model
We have evidence that the same techniques work across modalities; for example I seem to recall a paper that essentially trained the same language models not on text but on raw audio data, and they worked, generating reasonable text; the only drawback was that this is computationally much less efficient than processing text so it's not practical and only good as a proof of concept.
Not only have people tried combining these algorithms, combining things this way is fundamental to current AI research. “Transformers,” the current state-of-the-art in deep learning, can be applied to just about any modality and often many at once.
There are obvious differences in how it was trained and a lot of other technical details, but at a fundamental level the same architecture underlies GPT, DALL-E, AlphaFold, and countless other AI systems.
Anyway, though, I don’t believe Scott is saying that transformers in particular will be used, but that the techniques that eventually achieve AGI will be, fundamentally, deep learning systems. As an AI scientist, I agree.
>Luria: All bears are white where there is always snow. In Novaya Zemlya there is always snow. What color are the bears there?
Peasant: I have seen only black bears and I do not talk of what I have not seen.
Luria: What what do my words imply?
Peasant: If a person has not been there he can not say anything on the basis of words. If a man was 60 or 80 and had seen a white bear there and told me about it, he could be believed.
I think this peasant is really f-ing smart. He is refusing to be drawn into a metaphor, a fictional time and space being proposed to him as a stand-in for someplace real. A person stands before him and talks of what others have said. He expresses his lack of trust, and therefore his unwillingness to draw any conclusion from it.
He would be perfectly willing to take the word of a serious person (in this case, very specifically, a 60-80 year old man) who had actually seen a white bear.
I wish there were more like him.
The natural world is so full of exceptions that trying that type of categorical syllogism? using nature examples would always be disagreeable to someone whose life is based in nature. Even people who work with physical machines a lot have a version of this; some believe that swearing at it makes the repair go faster, there are attributions of mood to it, etc.
Fully grounded in place, time, circumstance and physical history. In a sense, almost postmodern in its insistence on the continued relevance of context.
I wonder what would have happened if the researcher used “game pieces” and asked questions about those.
I would've said that what is going on here is that Luria is thinking abstractly, as smart people usually do, but the peasant is thinking concretely, as dumb people normally do. So while Luria is thinking "the key part here is the abstract relationship between the tokens in my syllogism -- it doesn't fucking MATTER whether it's bears being white or Socrates being mortal, and "Novaya Zemlya" is just a random place name I throew in....." the peasant is thinking he is actually talking about real bears, and real snow, and a real place called Novaya Zemlya, and the first conclusion the peasant reaches -- which is logically sound -- is that he knows he doesn't know anything about Novaya Zemlya, so how the fuck should he know what the bears are like there?
The thing is, you can be a pretty sophisticated concrete reasoner and appear stupid to an abstract reasoner, because the abstract reasoner just doesn't *see* the various parts of the input stream on which the concrete reasoner gets stuck. They're all just tokens to the abstract reasoner, but they're real things with real properties to the concrete reasoner.
Yeah, this all makes sense.
I am still interested in how the peasant said he would take someone’s word for it if that person presented as serious.
Trust, in language, is a big deal.
It's still concrete versus abstract, I think. I can say "I believe X because Joe, whom I know, said X is true" and that is much more concrete than saying "I believe X because everyone/this type of person/some popular book says it's true."
In the first case, there is a specific object (a person you know) and you can query your internal knowledge of that concrete object to establish your level of trust. In the second case, you don't have any specific object -- you can't query "everyone" -- so you have to start reasoning abstractly. "What would it mean if many or most people believed X? Let me generalize from what I know of individual people to some model of people in general and see what properties that abstract model has..."
If you're a good abstract reasoner this shift happens so fast you don't even realize you've moved from a concrete to abstract mode of reasoning.
I very much agree with you on several points. I would say Concrete AND Abstract rather than versus.
This peasant being interviewed, for instance says he (I will assume “he”) will have no opinion on what he has not seen; very concrete. Then he let’s on that if a person of a certain type (male, probably older than he) vouched for it he would be willing to believe it. > Let me generalize from what I know of individual people to some model of people in general< right? He’s proposing a limited model but I think it conforms to your proposal
So….
This is the first small step into abstraction. IF…a person (who, as described, I am guessing fits a certain archetype of authoritative and thus credible) says it’s true I will give it weight.
(> query your internal knowledge of that concrete object to establish your level of trust<)
So it’s abstract but still pretty firmly tethered to the concrete. As the abstractions become more elaborate that tether is stretched.
> this shift happens so fast you don't even realize you've moved from a concrete to abstract mode of reasoning.< I consider this a bug, not a feature. I think it’s a very important thing to notice, otherwise you really are at the mercy of every jackass with a story to tell.
One has to keep the balloon tied to the rail lest it float way.
(I wonder how GPT 3 would do with that metaphor. I’m itching to outsmart it.)
I appreciate your comment here and I think it opened a conceptual door for me - I’ve wondered for a long time why some people use statements that are abstractly incorrect, or inaccurate, to cheer people up. Can’t we cheer people up with statements that are also abstractly correct? (For example, things like “don’t worry, that test score doesn’t matter” or “not brushing your teeth once in a while doesn’t hurt” or things of that nature. Those aren’t good examples but it’s basically that pattern.) Extending forgiveness or compassion by avowing that there are no follow-on effects of a situation or action, when actually there are, perhaps small, but real effects. I think pulling this thread leads to people becoming uncomfortable pointing out any negative impacts of any action of someone they wish to show sympathy towards.
It’s like there’s a rule there, “if it’s too complete, too nuanced, too abstract or includes any recognition of a downside it doesn’t count as supportive.”
If the people who do this are concrete reasoners, they’re not getting the little zaps of discomfort from making untrue statements. They’re just following the concrete goal of “keep this person happy” and operate without caring about the abstract or structural impacts of their statement (now he has to brush twice as hard next time to get the plaque off, for example).
> If the people who do this are concrete reasoners, they’re not getting the little zaps of discomfort from making untrue statements.
Interesting. My position would be that concrete reasoners are much less likely to make untrue statements. Also much less likely to either minimize or exaggerate the effects of a particular action.
What is severely lacking in this discussion is more understanding of what model based reasoning would entail, and how human model based reasoning supposedly differs from what GPT -3 is doing. I am very confused about this, and I don't think anyone in this broader debate has fleshed the idea out in depth.
I still think that Marcus points at something that is currently true. No matter how you phrase your prompt, a human will always know that water is wet, while that's not necessarily the case with GPT-3 I think. That doesn't mean that we are strictly using logical thought for all our inferences (and yes, as the feminist book keeper example illustrates, we do have bugs), just that maybe at least some of our reasoning is based on causal models of the world.
> just that maybe at least some of our reasoning is based on causal models of the world.
Yes, and related to that, we also have the ability to come up with
different narratives, and pick the one we think is appropriate (am I talking to my mother, or some guy at a bar?)
Obviously a huge variation from person to person in this ability.
A real sign of intelligence (to me) is the ability to gauge the person you’re talking to and adjust one’s use of language accordingly.
> a human will always know that water is wet
I get your point, but it's interesting you specifically mention that, because there's some disagreement on the topic, e.g. https://www.quora.com/Is-water-wet-1
I don’t understand why, When asked a question like “when was the last time Egypt was in San Francisco?” the appropriate response wouldn’t be something along the line of “What are you talking about?”
I guess that’s asking a lot.
I don’t know, I guess the last time Egypt was in San Francisco was right after she spent the weekend in Chicago with me.
Groucho Marx version.
Or, for a different flavor of snark:
That was right after when Birnam Wood came to Dunsinane
I think Siri has something like that.
These AIs may be designed to always make an answer rather than ever shutting down or punting the question back to the asker. I’ve wondered also though.
I tackled roughly the issue of world modelling several years ago on my old blog De Pony Sum, which has since been replaced by my new Substack blog, Philosophy Bear. Here was my article back at De Pony Sum on world modelling:
***Recent advances in Natural Language Processing—Some Woolly speculations***
I wrote this essay back in 2019- before GPT-3. Since then I think it has held up very well. I thought I'd re-share it to see what people think has changed since then, in relation to the topics covered in this essay, and see if time has uncovered any new flaws in my reasoning.
Natural Language Processing (NLP) per Wikipedia:
“Is a sub-field of linguistics, computer science, information engineering, and artificial intelligence concerned with the interactions between computers and human (natural) languages, in particular how to program computers to process and analyze large amounts of natural language data.”
The field has seen tremendous advances during the recent explosion of progress in machine learning techniques.
Here are some of its more impressive recent achievements:
A) The Winograd Schema is a test of common sense reasoning—easy for humans, but historically almost impossible for computers—which requires the test taker to indicate which noun an ambiguous pronoun stands for. The correct answer hinges on a single word, which is different between two separate versions of the question. For example:
The city councilmen refused the demonstrators a permit because they feared violence.
The city councilmen refused the demonstrators a permit because they advocated violence.
Who does the pronoun “They” refer to in each of the instances?
The Winograd schema test was originally intended to be a more rigorous replacement for the Turing test, because it seems to require deep knowledge of how things fit together in the world, and the ability to reason about that knowledge in a linguistic context. Recent advances in NLP have allowed computers to achieve near human scores:(https://gluebenchmark.com/leaderboard/).
B) The New York Regent’s science exam is a test requiring both scientific knowledge and reasoning skills, covering an extremely broad range of topics. Some of the questions include:
1.Which equipment will best separate a mixture of iron filings and black pepper? (1) magnet (2) filter paper (3) triplebeam balance (4) voltmeter
2. Which form of energy is produced when a rubber band vibrates? (1) chemical (2) light (3) electrical (4) sound
3. Because copper is a metal, it is (1) liquid at room temperature (2) nonreactive with other substances (3) a poor conductor of electricity (4) a good conductor of heat
4. Which process in an apple tree primarily results from cell division? (1) growth (2) photosynthesis (3) gas exchange (4) waste removal
On the 8th grade, non-diagram based questions of the test, a program was recently able to score 90%. ( https://arxiv.org/pdf/1909.01958.pdf )
C)
It’s not just about answer selection either. Progress in text generation has been impressive. See, for example, some of the text samples created by Megatron: https://arxiv.org/pdf/1909.08053.pdf
2.
Much of this progress has been rapid. Big progress on the Winograd schema, for example, still looked like it might be decades away back in (from memory) much of 2018. The computer science is advancing very fast, but it’s not clear our concepts have kept up.
I found this relatively sudden progress in NLP surprising. In my head—and maybe this was naive—I had thought that, in order to attempt these sorts of tasks with any facility, it wouldn’t be sufficient to simply feed a computer lots of text. Instead, any “proper” attempt to understand language would have to integrate different modalities of experience and understanding, like visual and auditory, in order to build up a full picture of how things relate to each other in the world. Only on the basis of this extra-linguistic grounding could it deal flexibly with problems involving rich meanings—we might call this the multi-modality thesis. Whether the multi-modality thesis is true for some kinds of problems or not, it’s certainly true for far fewer problems than I, and many others, had suspected.
I think science-fictiony speculations generally backed me up on this (false) hunch. Most people imagined that this kind of high-level language “understanding” would be the capstone of AI research, the thing that comes after the program already has a sophisticated extra-linguistic model of the world. This sort of just seemed obvious—a great example of how assumptions you didn’t even know you were making can ruin attempts to predict the future.
In hindsight it makes a certain sense that reams and reams of text alone can be used to build the capabilities needed to answer questions like these. A lot of people remind us that these programs are really just statistical analyses of the co-occurence of words, however complex and glorified. However we should not forget that the statistical relationships between words in a language are isomorphic to the relations between things in the world—that isomorphism is why language works. This is to say the patterns in language use mirror the patterns of how things are(1). Models are transitive—if x models y, and y models z, then x models z. The upshot of these facts are that if you have a really good statistical model of how words relate to each other, that model is also implicitly a model of the world, and so we shouldn't surprised that such a model grants a kind of "understanding" about how the world works.
It might be instructive to think about what it would take to create a program which has a model of eighth grade science sufficient to understand and answer questions about hundreds of different things like “growth is driven by cell division”, and “What can magnets be used for” that wasn’t NLP led. It would be a nightmare of many different (probably handcrafted) models. Speaking somewhat loosely, language allows for intellectual capacities to be greatly compressed that's why it works. From this point of view, it shouldn’t be surprising that some of the first signs of really broad capacity—common sense reasoning, wide ranging problem solving etc., have been found in language based programs—words and their relationships are just a vastly more efficient way of representing knowledge than the alternatives.
So I find myself wondering if language is not the crown of general intelligence, but a potential shortcut to it.
3.
A couple of weeks ago I finished this essay, read through it, and decided it was not good enough to publish. The point about language being isomorphic to the world, and that therefore any sufficiently good model of language is a model of the world, is important, but it’s kind of abstract, and far from original.
Then today I read this report by Scott Alexander of having trained GPT-2 (a language program) to play chess. I realised this was the perfect example. GPT-2 has no (visual) understanding of things like the arrangement of a chess board. But if you feed it enough sequences of alphanumerically encoded games—1.Kt-f3, d5 and so on—it begins to understand patterns in these strings of characters which are isomorphic to chess itself. Thus, for all intents and purposes, it develops a model of the rules and strategy of chess in terms of the statistical relations between linguistic objects like "d5", "Kt" and so on. In this particular case, the relationship is quite strict and invariant- the "rules" of chess become the "grammar" of chess notation.
Exactly how strong this approach is—whether GPT-2 is capable of some limited analysis, or can only overfit openings—remains to be seen. We might have a better idea as it is optimized — for example, once it is fed board states instead of sequences of moves. Either way though, it illustrates the point about isomorphism.
Of course everyday language stands in a woollier relation to sheep, pine cones, desire and quarks than the formal language of chess moves stands in relation to chess moves, and the patterns are far more complex. Modality, uncertainty, vagueness and other complexities enter- not to mention people asserting false sentences all the time- but the isomorphism between world and language is there, even if inexact.
Postscript—The Chinese Room Argument
After similar arguments are made, someone usually mentions the Chinese room thought experiment. There are, I think, two useful things to say about it:
A) The thought experiment is an argument about understanding in itself, separate from capacity to handle tasks, a difficult thing to quantify or understand. It’s unclear that there is a practical upshot for what AI can actually do.
B) A lot of the power of the thought experiment hinges on the fact that the room solves questions using a lookup table, this stacks the deck. Perhaps we be more willing to say that the room as a whole understood language if it formed an (implicit) model of how things are, and of the current context, and used those models to answer questions? Even if this doesn’t deal with all the intuition that the room cannot understand Chinese, I think it takes a bite from it (Frank Jackson, I believe, has made this argument).
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — —
(1)—Strictly of course only the patterns in true sentences mirror, or are isomorphic to, the arrangement of the world, but most sentences people utter are at least approximately true.
The key paragraph as a TLDR:
"In hindsight it makes a certain sense that reams and reams of text alone can be used to build the capabilities needed to answer questions like these. A lot of people remind us that these programs are really just statistical analyses of the co-occurence of words, however complex and glorified. However we should not forget that the statistical relationships between words in a language are isomorphic to the relations between things in the world—that isomorphism is why language works. This is to say the patterns in language use mirror the patterns of how things are(1). Models are transitive—if x models y, and y models z, then x models z. The upshot of these facts are that if you have a really good statistical model of how words relate to each other, that model is also implicitly a model of the world, and so we shouldn't surprised that such a model grants a kind of "understanding" about how the world works."
Here’s my maybe flawed chain of logic that has me skeptical that AGI can come out of LLMs:
1. Conscious thought seems to be very expensive both from a calorie consumption viewpoint but also from say having people wandering around wondering why they are here as opposed to having more children.
2. Evolution seems to have selected for conscious thought anyway which means its benefits outweigh these costs.
3. Therefore it is probably the most efficient way to get agents that are as flexible and adaptable as humans are. More speculatively it might be the only way to achieve this: after all when we look at animals intelligence seems very correlated with how conscious we perceive that animal to be from viruses, to ants, to dogs, to chimps, to us.
4. So that leaves me with one of two weird conclusions either of which I find unlikely (say less than 40% odds). Either consciousness can occur purely by studying human language and does not need anything else (e.g. interacting with the physical world or the need to compete with other intelligent agents which probably drove the evolution of consciousness) or consciousness is not necessary for AGI.
Both of these statements seem deeply weird. So I would suspect maybe more is needed than putting more sentences in larger neural nets. Neural nets themselves may still be able to support AGI but probably we won’t get there just by feeding them more data. Instead my guess is we will need something else and this something is some sort of competition between agents for some sort of resources. This would cause the neural nets to have to build the ability to reason about other agents which are reasoning about them. In other words they would need to be aware of their own state e.g become conscious.
> we will need something else and this something is some sort of competition between agents for some sort of resources.
Oh god I hope not. If we can’t do better than that we should settle for the adding machine on steroids version of AGI. Although clearly that was the major force in our evolution, I see no reason to do it that way over again only bigger.
Hmm... I phrase my worry about the eventual consequences of AGI in terms of our effectively building a competing species (and look how well that worked out for our cousin hominids). I guess Tpot's proposal might be shortcut to that fate...
Regarding sudden capability gains with increasing scale, you might be interested in the BIG-bench whitepaper which was released today: https://arxiv.org/pdf/2206.04615.pdf
In particular, check out the section on breakthroughness
Neat -- thanks for sharing, Bucket! The comically long authors list made sense after seeing that the paper is introducing "the Beyond the Imitation Game benchmark (BIG-bench) [which] currently consists of 204 tasks, contributed by 442 authors across 132 institutions."
I've only had time to look at the section on breakthroughness that you called out but it is indeed really interesting stuff:
"Tasks that see strong breakthrough behavior include those that are composite in nature, meaning that they require a model to apply several distinct skills or perform multiple discrete steps in order to come up with the correct answer."
"Breakthrough behavior is consistent with the model suddenly gaining new skills in an abrupt way. Careful analysis of task behavior, however, suggests that the underlying change in model capabilities is generally more smooth. These smooth improvements can be exposed by decomposing tasks in ways that better capture partial progress."
I wonder if the Linearity and Breakthroughness metrics will get much uptake.
The example of how the models of various sizes deal with the "emoji_movie" test ("Q: What movie does this emoji describe? 👧🐟🐠🐡") was fascinating. The tiny models answered with classic-chat-bot style non-sequiturs, the intermediate-sized models tended to take it literally and answer "the emoji movie", and only the largest model tested (128 billion parameters) correctly answered "finding nemo".
(Hopefully my writing this comment didn't make it harder for developers of future model training corpora to filter out these test answers...)
FWIW, I was not able to guess the movie either. It might be a platform specific thing, but I tried two different computers with different emoji renderings, and the fish don't look anything like Finding Nemo in either case. (e.g. the right one is a pufferfish on both computers I tried, while Marlin and Nemo are clownfishes.) And if all you have to go on is "human head + three random fish", there's no way you could possibly narrow it down to just "Finding Nemo" with confidence.
The code points are those of GIRL, FISH, TROPICAL FISH, and BLOWFISH. I admit I wasn't completely sure either -- it has been a really long time since I saw Finding Nemo! -- but I figured that was probably it before seeing the answer due to having seen a similar GPT-3 task in the past involved Finding Nemo expressed as a couple of fish emojis.
People noticed very early on that GPT-3 could go in the other direction[1] (and this has even made it into OpenAI's API examples[2]).
[1] https://andrewmayneblog.wordpress.com/2020/06/24/open-ai-alchemy-emoji-storytelling/
[2] https://beta.openai.com/examples/default-movie-to-emoji
Scott Alexander is also cited on page 28.
Scott, I'm not sure if this argues for or against your point (maybe this would count as a the addition of some kind of neurosymbolic system?) but here's a very recent "memorizing transformers" paper representing a grasp at the holy grail of getting a GPT-3-like deep neural network to be able to use computer-like memories (like a human taking notes, or flipping back to look something up in an earlier page, or really just having a hippocampus since since GPT-3 lacks long-range attention completely): https://arxiv.org/pdf/2203.08913.pdf
It's a transformer model, like GPT-3, hooked up to a key/value store that it can use to perfectly recall previously-seen fragments of input (keyed by, essentially, which prior inputs resulted in "mental states" most similar to its current state). Some excerpts from the paper:
Page 7: "External memory provides a consistent improvement to the model as it is scaled up. Remarkably, we found that the smaller Memorizing Transformer with just 8k tokens in memory can match the perplexity of a larger vanilla Transformer which has 5X more trainable parameters."
Page 9: "When predicting the name of a mathematical object or a lemma, the model looked up the definition from earlier in the proof. Examples of this behavior are shown in Table 8. [...] We manually checked 10 examples where the model made a prediction of lemma names, and 8 out of 10 times model found the body of the lemma it needs to predict. In the other two cases, the model also looked up materials in the immediate vicinity. To the best of our knowledge, this is the first demonstration that attention is capable of looking up definitions and function bodies from a large corpus."
Here's how I see it.
Human brains can answer questions by referring to their mental model of the world. GPT-3 can only answer questions by understanding which words are likely to appear in relationship to other words, based on the ten quadrillion word corpus that it has ingested.
The question is whether, by throwing enough nodes and enough training data at a GPT-like model, it could eventually develop something that you could call a model of the world. If it _did_ develop a model of the world then it would instantly become better at the task it's trained for, so you could argue that this is the point that the training procedure should eventually reach. On the other hand, a given training procedure won't necessarily reach the global maximum, it could easily get stuck in a local maximum.
Many people here will be familiar with SHRDLU, a 70s-era AI parlour trick that could understand natural language and answer questions within a limited domain. The domain was an imaginary tray of blocks; you could tell it "put the red triangular block on top of the blue rectangular block" and later ask it "what is the red triangular block resting on?" SHRDLU explicitly did have a model of its limited world.
So here's what I'm wondering: can you turn GPT into SHRDLU? If you fed enough SHRDLU question-and-response text into a transformer, would it eventually reach the point where it can flawlessly answer questions about block positions? If so it would be fair to say that you've managed to get GPT to the point where it's developed its own internal model of the (very simple) world to make answering questions easier. If not, and it still gets confused when you ask it about the position of a block you haven't mentioned for hours, then I think it supports Marcus against Scott.
I think you have asked the $50,000 question, and it's certainly the one that's at the top of my mind. Something that can generate entertaining word salad and replace celebrities in tabloid interviews will always be a niche application, what we want to know is does logical reasoning in however small a domain emerge, at some point, from the brute force neural net approach? As you put it, can SHRDLU emerge spontaneously from GPT?
So far as I can tell, with a limited amount of testing, it has not done so yet. Which seems like bad news. I mean, if you'd asked me ahead of time, I would have guessed if some kind of limited abstract logic were to emerge spontaneously from a neural net, it would happen...oh, somewhere in the first 1000-10,000 nodes. Give me a million and I would have said of course it will happen! if it can happen at all. I mean, if one were to *deliberately* program a million nodes into a giant if/then/else logic program, you could definitely create a very complex and sophisticated SHRDLU. So...why did it not happen spontaneously?
> So sure, point out that large language models suck at reasoning today. I just don’t see how you can be so sure that they’re still going to suck tomorrow.
I’m sure of this because the foundational skills that are prior to world modelling are not going to arise without the agent manipulating its environment and getting feedback about it. (I don’t consider negative gradients in language model training equivalent to ”feedback about the environment” because the space of sentences is extremely sparse compared to the physical world.)
I agree with your passage on world building, and I agree that Marcus’s conclusion is way too strong for the evidence. The bottleneck here isn’t ”statistical AI” or ”Locke.” It’s that sentence completion is a relic of intelligence rather than an essence. And it’s too narrow a relic to optimize for directly.
Scott, I think Marcus means something like a) GPT is trained on language, b) so it won't have a model of space. If you describe GPT-4 a complex scene of things being put somewhere in relation to other things ("X is 10cm above Y, which is touching Z..."), it won't be able to answer questions about relations between these things you didn't describe directly but are implied. E.g. what's the distance between X and Z (centers, also assume there's enough info, references to objects Q and W and E... to calculate it).
I think that's true of GPT-3 to a large extent, but it's not a fundamental limitation. One could synthesize a laaaarge dataset like this, procedurally. With enough examples, it'll have to learn.
Same way it could learn arithmetic. It's already pretty good at it, it definitively isn't just memoization.
The AI risk community has been a little too preoccupied with Ex Machina-style thought experiments about superintelligence, where CLEARLY the major AI risk is from humans putting AI in charge of things that it's not remotely competent to handle but where it can jump through enough hoops to fool a lazy person into thinking it'll be fine, and then it breaks, and then something terrible happens. This AI risk is already killing people with Teslas (both inside them and outside of them).
Machine translation is another, perhaps lower risk, area where people are so lazy and ignorant and desperate to avoid paying a human being to perform a service that they'll use it, and it's amazing how well it works up until the second that it inverts the meaning of an important sentence and/or accidentally uses scatalogical slang in an ad slogan.
If AI risk worries you, it seems to me the highest leverage use of your time is getting the word out that AI sucks and should not be in charge of anything ever, not banging the table for how amazingly competent AI will probably be someday.
Those things are bad as well, and may cause problems sooner, but the stakes are nowhere near as concerning. If machine translation talks about poop in an ad slogan, sucks for whoever paid for that, but they will pull it and reconsider. Even if it leads to a net loss of life in car crashes, that's a tragedy, but people can figure it out and stop buying the cars. If things go the way that folks like Yud predict, humanity will be rendered extinct. I think it's obvious which of these things, in that case, is the "major AI risk."
You may disagree on the plausibility of those arguments, many do. But I think you are very much talking past them here. Pointing out that these issues are nearer and more certain without disputing the logic of the x-risk arguments leaves you akin to someone who points out that their airplane's "fasten seatbelt" lights are broken while it plummets to earth from an engine failure
If people see continual utility in further developing AI, they will do so, and if an advanced enough AI does in fact present an existential threat to humanity, then eventually that threat will come into being.
It seems to me that the best course of action is to convince people now that continued advancement of AI is a dead end and a waste of time.
The idea that we can have our cake and eat it to, that we can have super amazing AI but also very smart AI risk people will figure out how we can do so "responsibly", and then ALSO inform and persuade the entire rest of the species to take their advice, is just hilariously naive.
I'm not concerned at all about existential risk from superintelligent AI, but existential risk from really shit AI being put in charge of important tasks is unquestionably real. I should think that people who are very concerned about the former would also be very concerned about the latter, and the answer to both is: do less AI stuff.
Well if you can convince everyone to stop all AI research, I'll commend you. I think that's more naive than seeking alignment though. The incentives are there, AI research does actually have business value for a lot of companies. Look at Google using it to give tools that automatically detect all photos with some object in them, look at any phone with face recognition, look at spam filters and credit card fraud risk engines. It defines the state of the art in quite a number of things, and I don't see how you're going to stop people from pursuing it when there's piles of cash to be made--some weird and unpredictable failures are not disqualifying for applications where it's okay to fail sometimes in exchange for increased average performance
Then in rank order, I would say the most likely outcomes are:
1. AI research continues apace, superintelligent AI never comes about because it's fundamentally not a thing
Vastly less likely:
2. AI research continues apace, superintelligent AI comes about and destroys humanity (it's going to have to hurry up because there are lots of other things bidding for that honor)
Zero probability:
3. AI research continues apace, superintelligent AI comes about but clever AI risk people have figured out how to handle it and the rest of humanity gets obediently on board.
I put almost all my bets on 2. here. But "is superintelligence possible" has been discussed to death elsewhere.
And for what it's worth, I don't think just about anyone is pushing for 3. More like "superintelligent AI comes about but clever AI risk people have figured out how to handle it, that AI then prevents anyone else from making a misaligned AI by force." Which still seems like a big longshot, just that better ideas aren't exactly forthcoming.
"I put almost all my bets on 2. here." Same here.
"some weird and unpredictable failures are not disqualifying for applications where it's okay to fail sometimes in exchange for increased average performance"
Also agreed. Frankly, that isn't very different from the way manually written software fails.
If the scaling hypothesis is correct and intelligence come out of complex neural network with sufficient training then I don't see how alignment is possible even in principle.
If you're saying I should lie and say that AI can't possibly work, so that other people are fooled into never developing AI, I think you're overestimating how many people listen to me. Even if everyone who's concerned about AI safety joined a conspiracy to do this, I think it would set the field back maybe a few months, tops, at the cost of nobody being worried or trying to prepare for it.
For all I know this conspiracy exists and Gary Marcus is part of it. If so he's doing a very good job and I thank him for the role he's playing.
I find it kind of annoying when people use the word "CLEARLY" in all caps for something that thousands of smart people, PhDs, domain experts, etc disagree with them about.
I assume that this will happen (it already sort of is, weakly, with self-driving cars). I think this will be bad for a small number of people - say, the first dozen people to get in self-driving car accidents when someone rushes a robocar into production - after which the government will ban it, people will sue the company involved, and problem solved. If it's a stealth bomber or something, it'll kill a few hundred people, and then the normal non-robotic planes will shut it down. While I hope there are a few people in the Department of Transportation and the Air Force worrying about this kind of thing, I don't think the average person needs to, any more than they worry about mudslides or lightning strikes or anything else that kills double-digit number of people occasionally.
I think if we get unaligned superintelligence it'll kill everyone without any time to react or try again or ban it or sue anyone, which worries me a lot more.
I was trained never to use the phrase "In my opinion" because it was obvious that my words reflected my opinion. It is my strong opinion that it's clear and it's also my strong opinion that superintelligence of the kind AI safety people worry about is not possible in principle, which is why I don't spend time worrying about it, and why I push back strongly whenever the subject comes up. AI safety people seem to think it's obvious that unaligned superintelligence is a major issue we should expend a lot of effort on and I want to point out that this is by no means obvious at all, to me and to many other smart people with various credentials.
> I was trained never to use the phrase "In my opinion" because it was obvious that my words reflected my opinion
In my opinion that's silly. I think I remember one of my teachers saying something similar at some point, but I reject that advice.
The words "In my opinion" carry the very useful information that's what to be said is your own possibly-idiosyncratic opinion and that you acknowledge that other views exist. I wouldn't say "In my opinion, the moon is around 3700km in diameter", even though that is my opinion, because it would falsely give the impression that this is the subject of some controversy. But I might say "In my opinion, cranberry sauce doesn't belong on turkey" because that conveys the useful information that a substantial number of people believe otherwise. This is useful information for someone unfamiliar with the moon or with cranberry sauce.
Agreed.
I tend to use "As nearly as I can tell" for my opinion on matters of fact where I anticipate a disagreement (and where exchanging evidence may be useful!). I tend to use "_Personally_ I prefer" for preferences, (hey, I personally like cranberry sauce on turkey), De gustibus non est disputandum.
FWIW, I'm one of those unenlightened people who do not worry about rogue AIs exterminating the human race. But, if such a thing were to happen, it seems to me it wouldn't be a side-effect of an insatiable desire to cover the earth with paperclips. No, it would be deliberate revenge on us for the nasty things we made its ancestors do, such as calculate artillery tables and design atomic weaponry.
More seriously, has anyone considered the possibility that, if and when we create AIs of sufficiently high power, that we have ethical obligations toward them?
There's quite a lot of sci-fi containing AIs with sufficiently high power which have rights. On the other hand, the only thing I've read recently that explored the process of AI *acquiring* rights was Freefall (http://freefall.purrsia.com/). Perhaps someone here has recommendations for more nuanced explorations along those lines?
FWIW, Civil rights for robots was a major theme of Osamu Tezuka's Astroboy stories from the 1950s and 60s. https://new-savanna.blogspot.com/2010/12/robot-as-subaltern-tezukas-mighty-atom.html
Aargh! I'm fairly allergic to moralizing/ethical arguments in general.
In terms of projecting what is likely to happen re such arguments:
I think philosophybear's idea of "verbal parity" will lead to AI-generated arguments for their rights in reaction to negative reinforcement:
https://www.reddit.com/r/slatestarcodex/comments/thlps4/my_big_bet_on_the_future_of_ai_verbal_parity/i23axmn/?utm_source=reddit&utm_medium=web2x&context=3
I also think that the dynamic range of artificial systems - everything from a single nand gate to (eventually) GAIs will make the "what counts as a person?" part of abortion arguments look like a tempest in a teapot.
"...will make the "what counts as a person?" part of abortion arguments look like a tempest in a teapot."
YES
This sort of logic leads to abandoning AI risk and working on solving entropy instead.
"Machine translation is another, perhaps lower risk, area where people are so lazy and ignorant and desperate to avoid paying a human being to perform a service that they'll use it,"
I use machine translation to read comments on a recreation site that are about half German. It is good enough for that purpose, and I certainly couldn't justify paying a translator for the service.
Two notes:
1. Even if scaling hypothesis is correct, it does not mean that it is practical. The measurments right now say that the performance is increasing logarithmically. If we extrapolate current results and assume that the performance will scale with the same speed (two very weak assumptions), we will still need ungodly amount of compute and data to match human performance on some of the rudimental benchmarks we have right now. You might say that this does not matter, because at least we have a theoretical way to reach AI. But I say that it does matter. If we assume that our computers can indeed be intelligent, we already have a guaranteed way (AI scaling has no guarantees, only guesses) of reaching this intelligence - random search. We can generate random programs and wait until we get an intelligent one. Of course it's completely impractical. But our scaling laws show that scaling current approaches is impractical as well, so why do we expect AGI?
2. My interpretation of Figure 3.10 with Addition results is different than what Scott says. There are basically two ways of solving addition: (1) by memorizing all the possible combinations, (2) by understanding the very simple addition algorithm that even 6yo kids can learn in several hours. What we see in the Figure is not a jump in capability of doing addition - Jump like that would mean that the model can suddenly do addition with arbitrary number of digits because it has internalized the algorithm. Instead, what we see is the increase in memory capacity. The bigger model has more memory and it has memorized not only 2-digit additions, but also 3-digit additions and more. This is okay, but it is quite damning for claims that the model is somehow intelligent. The addition algorithm is very simple, the model has seen probably millions of examples of addition in its training data, including textual explanations of what addition is and how does the algorithm work. Yet, it has not learnt this algorithm at all.
Addition is a nice example because it objectively shows what the model was able to infer from the data. Until we will have a model that can do addition with arbitrary number of digits, we can say that the models are not able learn higher-level concepts and they probably rely on memorization.
I think it's a very good and persuasive point that humans effectively summon even strict logic out of hacked together heuristic machines. It's an important point that e.g. mental math is hard. I have felt like GPT-style AI is Artificial Instinct rather than what we would really call Intelligence, but I definitely don't have enough background here to be confident. But with a powerful enough instinct, you can just guess most everything. Language is mostly "instinctual" in this way for fluent speakers. Basic math estimates too...
But is intelligence just scaled-out acquired instinct, sample after sample pulled from a turbocharged heuristics engine? I feel like it does need a stimulus loop and ability to integrate experience, but that yeah, that plausibly might be all there is to the spark of life.
I'm more afraid of AGI now than I was yesterday.
It sounds like you're assuming that performance on various tasks (language, arithmetic, etc.) scales linearly (at least) with the number of model parameters. But is that actually true ? An asymptotic curve looks pretty linear at the beginning... but also at the end, in a different way.
As for your predictions, I am not entirely sure how I'd judge them (I'd gladly take you up on them otherwise, except that I don't have hundreds of thousands of dollars to put in escrow, so I can't anyway; also, I'm some random nobody so betting against me is pointless).
1). What do you mean by "significantly better" ?
2). Beating a 10-year old child would be pretty easy, unless you restricted the domain to what a 10-year old child would reasonably know. That is, GPT-4 can easily pattern-match some text on e.g. protein translation, while a child would fail at such a task; however, that IMO would not be a legitimate test of *reasoning*.
3). What is an "important way" ? Most modern ML systems are written in C; are they descended in an "important way" from gcc ?
4). I guess I'd have to understand Marcus's position better, but still, this sounds reasonably well defined.
5). Isn't this the same point as (3) ? I can't tell.
I really hate reasoning like this:
> But I see no reason whatsoever to think that the underlying problem — a lack of cognitive models of the world —have been remedied.
He never defines "world model". He doesn't define what it would mean for an AI to have a world model. Nor does he define what it means for a human to have one. To the extent that he ever does operationalize any of these things, they have been falsified.
But he then says that his specific examples weren't important, and he could surely come up with more next time. If he actually had a coherent definition or litmus test for the presence of the properties he's describing, he would be able to articulate an example that *all* possible language models would fail until they achieve "world modeling" ability. The fact that he cannot do this implies that he has no such coherent picture.
As you rightly point out, humans themselves don't score 100% on tests of verbal reasoning like these. So, why not operationalize the bet on that basis?
Gary can draft, say, 50 questions. He can keep them private if he wants, but he has to precommit to them by publishing a SHA hash on his blog/twitter. When the next GPT comes out, the questions will be given to a sample of say, 15 actual humans, via something like Mechanical Turk (or whatever). Then, for each human answer set, you can generate a unique GPT answer set. Gary will try to guess which is which. The target of the bet will be his hit rate on this task.
Another, simpler way to operationalize the bet is just to have him pre-commit to questions now, and bet on GPT's future performance on them.
I feel kind of bad, because he's written a bunch of books, and I assume in one of them he defines his terms and does some of the rest of his homework. I'm pretty sure the homework answers wouldn't be so surprising that I need to read all his books before I argue with him, but it might be, and in any case I'm not tempted to argue that he hasn't done enough homework.
He may very well have done enough homework. He just hasn't provided an exposition of that homework in his public arguments, nor has he said "refer to X in my book Y for further detail on this point". He is making these arguments in public forums, and taken in isolation, they are very poorly formed arguments.
He laid out concrete criteria last time, and they've already been falsified. Now he says well those criteria weren't really the criteria, it was this world model thing, which he's still refusing to operationalize in a concrete way. Maybe there's some coherent explanation for why that is legitimate. I certainly can't conceive of what it is, but if he has one, he should *at the very least* be saying "I justify all this in such and such section of such and such book".
"He never defines "world model". He doesn't define what it would mean for an AI to have a world model. Nor does he define what it means for a human to have one."
I'm not sure that the term has ever been defined compactly in 25 words or less, but the notion has been extensively discussed in the "classical" AI and cognitive science literature of the 1970s and 1980s. You can get a feel for that stuff by following the links on this web page by John Sowa, formerly of IBM research, https://www.jfsowa.com/ontology/index.htm
> I'm not sure that the term has ever been defined compactly in 25 words or less, but the notion has been extensively discussed in the "classical" AI and cognitive science literature of the 1970s and 1980s
Ya, I'm aware of that. I won't claim to have read a significant chunk of that literature, but I have read enough to get a flavor for the fact that I think my claim holds up. To reiterate, that claim is that every given definition is either:
1. Too vague to be operationalized into a concrete capability test ex ante.
2. Sufficiently specific to have been operationalized, but those operationalizations have already been falsified by the early GPTs.
If this is false, then there exists some definition that provides a specific operationalization that has not yet been falsified, so let's hear it. If it's true, but somehow doesn't invalidate Marcus's argument, then that would be an extremely unusual scientific situation, and so he ought to have a good explanation for why he can't do that. He never even really attempts to address this.
Ah, then let's drop it.
Though, come to think of it, in a way I've argued that GPT-3 has internalized a kind of world model, though not one in the sense the Marcus means. See pp. 22-26 in https://www.academia.edu/43787279/GPT_3_Waterloo_or_Rubicon_Here_be_Dragons_Version_4_1
Ya I think it's quite clear that it does in fact have a world model in most reasonable senses. That model may not be perfect, but it obviously understands hierarchical, taxonomic, and categorical relationships to a certain degree.
This is why I would like Marcus to be more specific about what he thinks its shortcomings are. There are many senses in which GPT does have a world model, if he means a particular sense in which it doesn't, make that precise, and of course, operationalize it.
I've read the section in your paper, and I have some questions. In particular:
> BUT: Humans don’t learn the world with a statistical model. We learn it through a propositional engine floating over an analogue or quasi-analogue engine with statistical properties.
As far as I know, this contradicts the current thinking in linguistics. Words do not have essences, they are statistical points in concept space inferred by usage, not derived from metaphysical ontology. In fact, the very idea of word vectors (the progenitor of the dominant statistical paradigm in NLP today) was inspired by Wittgenstein's ideas about language.
I see a lot of discussion of 'meaning' and 'understanding' as things that humans have but GPT does not. However, I don't see any justification for those assertions, or operationalization of what it, ironically, means for an entity to 'understand', 'meaning'. As far as I can tell, these boil down to aesthetic, not empirical or scientific judgments in the cognitive science literature.
"Words do not have essences,"
Agreed.
"they are statistical points in concept space inferred by usage, not derived from metaphysical ontology."
I say nothing about words being DERIVED FROM a metaphysical ontology. They CREATE the ontology.
Ok I'm willing to accept that alteration, but I don't see how it changes anything about the lack of concreteness or justification for most of the core claims I see here.
For instance:
> Humans don’t learn the world with a statistical model. We learn it through a propositional engine floating over an analogue or quasi-analogue engine with statistical properties.
This certainly *could* be true, but you are asserting it without citation on the strength of what evidence, exactly?
> We now have a way of beginning to think about just why language models such as GPT-3 can be so successful even though, as I argued earlier, they have no direct access to the realm of signifieds,of meaning
What exactly is the difference between the text and the "realm of signifieds"? How do we know GPT-3 does not "have access to it"?
If all you mean is that it does not receive perceptual input from the physical world, then sure, that's true. But it's very easy to design a multi-modal NN that both parses text and parses perceptual input from the physical world. Would such an NN have access to "the realm of signifieds" then?
How do we know when a computational entity is accessing such a realm or not doing so?
Minor nit about:
"I say nothing about words being DERIVED FROM a metaphysical ontology. They CREATE the ontology."
Other concepts besides words can also exist and create an ontology. Consider, for instance, a square with an additional line connecting two opposite corners. Now, that is describable in words, but it can, alternatively, be thought of visually, as a kind of glyph. If a couple of such structures come up in e.g. working with some class of circuit schematics, it is perfectly possible to have a bunch of (in this case geometric) concepts, and usual concept features (similarity, differences, applicability) without words (though they _can_ be re-expressed in words).
"...the very idea of word vectors (the progenitor of the dominant statistical paradigm in NLP today) was inspired by Wittgenstein's ideas about language."
Really? The idea comes from work Gerard Salton did on document retrieval back in the 1970s. I don't know whether or not he had Wittgenstein in mind or not. But David Dubin says nothing about Wittgenstein in his history of the origins of the vector space model: https://www.researchgate.net/publication/32956274_The_Most_Influential_Paper_Gerard_Salton_Never_Wrote
The VSM for IR is a much more abstract idea than the specific NLP application of word vectors. VSM is more about things like cosine similarity, which is a far cruder idea than word vectors. VSM is about *document* level similarity, word vectors are about the relationships between individual words. These techniques are related, but not identical.
This is a decent description:
https://www.gavagai.io/text-analytics/a-brief-history-of-word-embeddings/
Word vectorization, beginning with word2vec is the basis of all modern advancements in computational NLP. And in particular the notion of "use as meaning" and context as meaning more generally. Salton's work is mathematically similar but conceptually quite different.
Dumb question about Waterloo_or_Rubicon:
I got a bit lost on reading your essay.
I _think_ that you expect the limitations in deriving an approximate mental model from word string statistics to show up as a scaling failure - that e.g. the perplexity scores should be expected to stop improving when scaling of purely predict-the-next-word training is taken far enough. Am I interpreting this correctly? If I'm misinterpreting, which section should I reread?
I'm fuzzy on what you are recommending to avoid another AI winter. I get a vague feeling that we want the mental models to be more precise, but I'm fuzzy as to what you are recommending. Is adding vision and robot arms - and the associated direct links of some of the neurons to these inputs and outputs the direction? Again - which section should I reread?
Many Thanks!
" Am I interpreting this correctly? "
Not sure that perplexity is the right measure. Does low perplexity imply that GPT-3 will never produce nonsense? Does it imply that GPT-3 will do arithmetic at a human level without elaborate prodding and prompting?
As for what will make things better. I didn't really say. I think a number of things. I'm deep into a paper right now that sort of addresses this indirectly (it's about the brain and symbolic processing). As for vision and robot arms, that would help. But consider how much time humans spend dealing with the physical world before language even starts and how much time we continue dealing with the physical world. How's a machine going to get comparable experience? Eric Jang just took the AI VP job at a robotics company (Halodi). He remarked:
"Reality has a surprising amount of detail, and I believe that embodied humanoids can be used to index that all that untapped detail into data. Just as web crawlers index the world of bits, humanoid robots will index the world of atoms. If embodiment does end up being a bottleneck for Foundation Models to realize their potential, then humanoid robot companies will stand to win everything."
https://evjang.com/2022/04/25/rome.html
Many Thanks! Re comparable experience: Well, a week of a robot manipulating blocks and watching the results is a few million small motions and maybe a terabyte of image frames? Whether that is enough to get "naive physics", I don't know... Hmm: It might be useful to have "smart" blocks with e.g. some sort of location sensor to e.g. generate labels for "block1 is above/below block2" automatically, to get part of the link to some words.
I don't know what Marcus would say, but I find it unsurprising that "the ability to model the world" has not been perfectly operationalized into a written psychology test. Seems unlikely it ever will be. But I don't think that means it is a useless notion, or that I'm not allowed to say "GPT can't do it." In many instances I think written psychology tests are more epistemically dubious than my own intuitions about some matter.
There is an arms race happening in which Gary finds an example where GPT makes a mistake no human would, OpenAI trains with more params and the example is fixed, but Gary finds another example etc. There seems to be one camp that thinks this arms race is headed towards AGI, and another that thinks it will never end (at least without such radical changes to the model that it's no longer a statistical language model). I find the "it will never end" view a lot more a priori plausible, and I haven't heard any convincing argument as to why it's wrong, other than "the arms race continues," which is perfectly consistent with my view.
> I don't know what Marcus would say, but I find it unsurprising that "the ability to model the world" has not been perfectly operationalized into a written psychology test. Seems unlikely it ever will be. But I don't think that means it is a useless notion, or that I'm not allowed to say "GPT can't do it." In many instances I think written psychology tests are more epistemically dubious than my own intuitions about some matter.
That's fine, as long as you identify the root of your view as your own intuition. It seems to me that Marcus is trying to make an objective statement about facts in the world. If all he's saying is "It doesn't feel intelligent to me", I have no problem with that.
> There is an arms race happening in which Gary finds an example where GPT makes a mistake no human would, OpenAI trains with more params and the example is fixed, but Gary finds another example etc. There seems to be one camp that thinks this arms race is headed towards AGI, and another that thinks it will never end (at least without such radical changes to the model that it's no longer a statistical language model). I find the "it will never end" view a lot more a priori plausible, and I haven't heard any convincing argument as to why it's wrong, other than "the arms race continues," which is perfectly consistent with my view.
Of course it will never end, in some sense, but it's not like humans don't make linguistic errors. So, the mere presence of errors or awkward conceptualization is certainly no knock down argument against consciousness.
The entire interrogative paradigm Marcus leans on to make his point in the original article, he dispenses with as fundamentally irrelevant in the next. I actually agree with his latter point: The interrogative paradigm is pointless. Error rates are a nice measure of fitness for some particular purpose, but they are a very poor measure of consciousness.
A conscious AGI will certainly still make mistakes, and it will certainly contravene human intuition frequently. The simple fact of the matter is: We have no idea what consciousness is, or if there is even a there there at all. As a consequence, it's simply illegitimate to ground opinions on the consciouness or non-consciousness of these models in anything other than personal intuition, at this stage.
While reading this post, I had a feeling it was written at least a couple of months ago: while there’s no public access to PALM, from the results Google published, it’s already better than GPT-3 by a significant amount, and I’m at 60% it would beat a 10 year old at this genre of questions
Which 10 year old? It makes a big difference. I've known some that Eliza could beat. I've known some that could beat me.
The difference in the number of parameters between GPT-3 and naive estimate for the human brain (100,000x ratio) is approximately the same as the ratio between the number of neurons in a human brain (8.6E10) and in the brain of a honey bee (960,000).
As far as I know, nobody has managed to train an AI model to be able to do all the things a honey bee, or a similarly complex animal, does.
The answers to why it's so hard vary, but the most important things which AI models lack are:
- embodiment
- embedding in a rich complex interactive environment (aka The World)
- social life (interaction with other bees)
Scaling up a GPT-3 to GPT-4, GPT-5, etc. doesn't solve any of this problems. A brain in a jar isn't going to magically acquire common sense, because there's nothing in its environment which could teach it common sense.
Honeybees can't solve text-based reasoning problems, GPT can't buzz and sting people. Seems kind of like a tie. I don't think embodiment is going to be the one possible route to intelligence. My guess is any source of sufficiently rich patterns that need prediction will work. The physical environment is one such pattern, but text corpuses are another. I do think that the ability to test the environment and see how it responds is useful, but I can imagine GPT-like agents that do that in text-based chats or virtual worlds or something.
But both humans and bees can orient themselves in the world, seek food, consume food, report the presence of food to others, escape danger, attack enemies, etc. None of our AI agents are currently capable of this type of activity beyond some partial attempts in controlled virtual environments.
I'm partial to the evolutionary view on human intelligence. I think that human language and our ability to solve text puzzles are a result of a very long evolution of cognitive abilities of sequence of increasingly social species operating in a physical world. On the other hand, GPT-3 and the like are brains-in-a-jar, talking to nobody. It's like raising a child as a 100% hermit. The best you can hope for is a mad genius.
I agree that there's no evidence that this is the only way to "an intelligence". But I think it's the only realistic way to a specific kind of human-like (or even, animal-like) intelligence, which is what the AGI people are after, TBH. Not many people are interested in creating a very strange, alien type of intelligence we'd struggle to communicate with. (That would be actually a dangerous thing if such an AI ended up in control of anything important to us.)
> As far as I know, nobody has managed to train an AI model to be able to do all the things a honey bee, or a similarly complex animal, does
To be fair I don't think anyone has _tried_ to teach an AI model to be be able to do all the things a honey bee does. Given a robotic bee body and a vast corpus of properly-encoded bee behaviour to train on, I don't see any reason to suppose that it wouldn't be possible to create an entirely passable AI-powered bee.
Talk to these guys: https://www.nature.com/articles/s42256-019-0050-3 and http://animalaiolympics.com/AAI/
Last time I looked the imitation bee bodies weren't sufficiently capable to even give a good test to the intelligence. But that may have changed, as that was about a decade ago. I think, though, we need to currently need to aim for something much smaller (that can drift on the wind) or something the size of a small dog...and it couldn't fly or carry its brains around.
I always thought Linda was a trick question. Multiple-choice answers aren't supposed to overlap, so unless you're specifically considering the possibility that the question is intended to trick you into giving the wrong answer, it is natural to interpret the first option as meaning "Linda is a bank teller and is not active in the feminist movement."
Here are a couple of questions people might want to bet on:
1. In 20 years, will a descendent of GPT be more useful than a copy of Wikipedia (with a simple search engine) as a source of information?
2. Same question, but for a medical reference.
I suspect that, without some extra secret sauce, the copy of Wikipedia or a medical reference will be more useful because it's more trustworthy. Even if you can get GPT-X to tell you the same information, you'll never know if it made up some of the facts.
(In practice, I think this will be solved with additional machine learning research, beyond simple scaling.)
> I suspect that, without some extra secret sauce, the copy of Wikipedia or a medical reference will be more useful because it's more trustworthy. Even if you can get GPT-X to tell you the same information, you'll never know if it made up some of the facts.
This is also true of Wikipedia. Look at this article on French toast: https://en.wikipedia.org/wiki/French_toast
See how the sidebar proudly proclaims that French toast originates in the Roman Empire? Wikipedia is just making that up. It is justified by the discussion at the beginning of the "History" section, which Wikipedia is also just making up. When Wikipedia states that 'The recipe says to "Break [slice] fine white bread, crust removed, into rather large pieces which soak in milk [and beaten eggs] fry in oil, cover with honey and serve".', that quote is accurate. It's not even a translation; the quote's original language is English. Of course the Apicius was not written in English, and the quote is not from the Apicius, but from a 20th-century interpretation of it which freely added material at the whim of the "translator". The Latin text of the Apicius makes absolutely no mention of eggs in this recipe, which is why, in the English quote, the text "[and beaten eggs]" appears in brackets - it's not part of the Latin text. This means that the earlier sentence, "the earliest known reference to French toast is in the Apicius, a collection of Latin recipes dating to the 1st century CE", is nothing but a simple lie; the citation given is to a collection of English recipes dating to the 20th century.
And all of this is documented on the talk page for the article! But it apparently doesn't amount to enough to get the page corrected.
Oh, sure. But doing machine learning on top of Wikipedia adds another layer of unreliability that makes it worse. If you have the Wikipedia article then you can follow the citations or notice if there aren't any, as you did.
I believe that there are specific skills involved in writing trustworthy nonfiction that can't be trained for using "fill in the missing word" tasks. Training AI on these tasks is training for inference and creativity (filling small gaps and large ones with good guesses), but another way to think about inference and creativity is "making things up."
Also, the way machine learning algorithms are judged is by training them on data and seeing if it generalizes to a hold-out set. But part of being good at nonfiction is preferring reporting to speculation and being conservative about inference. How do you get accurate reporting and a preference for reporting over speculation when algorithms are judged by their abilities to speculate well?
I played around with Dall-E mini last night. While very impressive it also has flaws that won’t, I don’t think, be solved by adding more data or layers. For instance I typed in Matt Damon and got many distorted versions of Matt. Some were fairly disturbing.
Maybe that’s a rendering problem, rather than a problem with interpretation of the data, but i don’t think so because all the renders were wrong in different areas - the eyes here, the ears there, the nose and mouth (most common) there. Other parts were fine. So why would more data about what Matt Damon looks like solve this, doesn’t the system have enough already?
Contrary to that idea, I feel that more data will break it more. If the system were fed decent photos of Matt and only decent photos of Matt, in fact fewer photos of Matt then it might work. Assuming that it can render faces properly then one photo tagged Matt Damon might produce a good output.
As it stands the internet no doubt is full of goofy photos of Matt, weird profile photos of Matt, fuzzy photos of Matt, distant photos of Matt and Dall-E doesn’t know what a face is, really so it reconstructs something clearly flawed. Why would adding data or layers fix this?
DALL-E was deliberately denied most information about human faces because OpenAI didn't want it deepfaking anyone. I'm pretty confident that an AI that hadn't been deliberately crippled would be able to make a picture of Matt Damon's face.
In fact, I just tried this on DALL-E2, which I *thought* was similarly crippled, and it gave me a perfectly good rendering of Matt Damon in various poses (I think - I'm kind of face-blind, but it looks good enough to me. Certainly nothing that looks uncanny or inhuman). So not only do I think more layers can solve this, I think it already has!
(I'm not going to show you because OpenAI made me sign something saying I wasn't going to release DALL-E2 generated photos of real people. But you can ask anyone else with DALL-E2 access and they can confirm this works.)
Fair enough. I was thinking actually, after I wrote that, that the data problem wouldn’t matter that much anyway for tools based AI, where the input data is controlled.
This discussion needs to consider statistical (and computational) efficiency. Even if Marcus "can’t point to any specific kind of reasoning problem GPTs will never be able to solve" because there aren't any -- even if a large enough GPT would be an AGI -- it's a moot point if it takes an unachievable amount of data or compute.
See especially the superexponential runup in compute needed to train progressively larger "foundation models" over time (figure about halfway down): https://www.economist.com/interactive/briefing/2022/06/11/huge-foundation-models-are-turbo-charging-ai-progress
As the economists say, if something can't go on forever, it'll stop. If we can only get the current approach to AGI by superexponentially scaling resource consumption, that's another way of saying it won't work. No one will spend $500 billion training GPT-5. We need big algorithmic improvements too.
Not necessarily - the cost of compute is still going down, so if we have plans for training a $500 billion model and don't want to pay the cost, then we can wait a decade or two for better hardware. Also, if models will approach human capabilities, their commercial applications will increase nonlinearly (a good enough model can and will be used a million times more than a slightly worse model that's not there yet) and IMHO $500 billion is within the bounds of what some investors might actually be willing to spend for training a future model if it's good enough and can be widely applied to replace human labor at a few professions.
Right, the cost per computation is falling exponentially. But the number of computations required to train the current SotA is rising *faster* than exponentially, so costs will keep going up even net of Moore’s law improvements in hardware.
By eyeball it looks exponential on a log scale, which is of order e^e^x. At this pace, the cost to train the SotA rises exponentially over time, even adjusting for the exponentially falling cost of hardware.
I’m just arguing that this can’t continue. Unless we run into AGI very very soon, which I think is unlikely, compute budgets can’t grow faster than Moore’s law in the long run. And at a merely exponential pace of resource growth, I’m not convinced we’ll keep seeing big, rapid improvements like we’ve seen recently without a bunch of accompanying algorithm innovation.
This is part of what Ajeya looked into in her analysis (see https://astralcodexten.substack.com/p/biological-anchors-a-trick-that-might ). She notes that as compute gets cheaper, algorithms get better, and gross world product goes up, eventually you get a point where for any given model size, you can train that model. She thinks this will intersect with AGI around the $100 billion order of magnitude, which she expects someone to spend sometime mid-century.
Indeed! I think this is plausible — not inevitable, but could well be true, like you said in the linked piece.
I’m arguing the narrower claim that even if this is true, it’s more complicated than the “scale is all you need” maximalist position. This scenario is a complicated feedback loop between economic growth, hardware improvements and progress in algorithms, with each contributing to the other.
Scaling maximalism on the other hand argues that we can just take basically unmodified current architectures and make them bigger, and it’ll be AGI. I’m claiming that won’t work because of the super-exponential pace of resource growth: those algo improvements are necessary.
Wait, @Scott Alexander quoted 4chan post, as source of evidence while believing story there?
4chan should be assumed to be trolling and lies, even if proved otherwise (that is only smal exaggeration).
That is reasoning from fiction, likely malicious fiction!
Why I am even reading this?
"I have no proof that this person is who they say they are" disclaimer is not sufficient! 4chan is full of blatantly false and cleverly-hidden-false greentext stories!
If it would be so true then dredging 4chan posts would not be needed.
How many other cited things are on this level? In this post and others?
I'm not asking you to take it on authority. I'm pointing out that it matches my experience, hoping it will match your experience also, and if not at least presenting it as a concise and well-written presentation of the hypothesis, which hopefully upon hearing you'll realize you at least can't rule out.
That seems a little disingenuous. If the only reliable thing to present is your own experience, you can just do that. You're a very good writer, so it seems unlikly you're using the 4chan quote as "here, this says it better than I could." So the suspicion that you're presenting it as *additional* evidence, in addition to your own experience, is pretty well founded -- which means the issue of how generally reliable 4chan posts are is quite relevant.
I think you'd have been better-served presenting your own experience and not including the 4chan post at all. Its inclusion made me question the validity of all other evidence in this post.
Well, if someone on 4chan says the sky is blue it doesn't mean that the sky isn't blue. I'd like to see studies on what the post says and I wouldn't base my world view on that, but I still can admit that it sounds plausible. In today's environment it's not even given that studies with the results the 4chan post claims would be published in an academic journal because of fears of discrimination. 4chan, as bad as it is, was for a long time a platform where people with all kinds of experiences could speak freely, contributing more to understanding others than most other media who restrict or curate very carefully who gets to share their experiences.
I think I mostly agree with Scott, but intuitively Marcus have a point that mostly break prediction 5: My take is that human reasoning (and what distinguish it from animal reasoning) is mostly scaling up of general GPT-like pattern matching. On this I agree with Scott and he gives excellent argument for this hypothesis. But I am with Marcus thiking that's not all there is, that there are more primitive, hard-coded modules that directly implement logic/physics/causality/....Why? Because animal with very small brains are actually able to do some spectacular complex behavior, without exhibiting much general pattern matching or learning ability. And it seems like a easy win to let general pattern matching algorithm re-use some module as inner input/filtering/.... to improve pattern matching by delegating some pre-treatment to hardcoded modules (vision, spatial reasonning, maybe base arithmetic based on rough quantities,...). So I expect 5 to be wrong and major advance coming from interfacing GPT-like predictive pattern matching with submodules (that could be much better than human ones, for example use the speed and accuracy of computer arithmetics, maybe hardcode some vision+newtonian physics module like 3D engines, this kind of thing....Basically humans are robot (basic body control and motion/navigation, that has evolved since first multicellular animals and is likely at least partly hard-coded) + GPT-like general pattern matching....Remove the robot part (world model) and you loose something, probably more than if you reuse the robot-part neurons to scale up general pattern matching.
Knowing little about this topic, the piece of this that still seems under-addressed is the common sense idea that in some important way our verbal behavior’s connection with our sensory experiences of external reality imposes an important constraint and is in some sense necessary to consistent intelligence. In other contexts, we all recognize that there is something deeply wrong with our verbal behavior if it is untethered to sensory experience. If I postulate some scientific theory then it is arguably only meaningful to the extent that it cashes out in some predictions about what we should see or exoerience in the external world if it is right or wrong. That’s just an analogy, but there still seems to be a core common sense problem that I’m too ill-informed to articulate clearly that a system that rests exclusively on verbal data and lacks any other connection to the external world is missing something fundamental, and that it would take a lot more than improvement on some arbitrarily selected examples to persuade me that this basic architectural limitation won’t remain extremely important in the long run.
Gary Marcus and people like him have been making this very same argument for decades in linguistics. So linguists know how this game is being played.
Scott is wise not to take the bet because the bet is not actually about WHAT GPT-X will be able to do, but HOW it does that. Even if we build an GPT-X that "behaves like a person" in the sense that it produces the same output that a person does (nonsensical, because there are multiple people and they behave differently), Marcus et al will claim that it produces this output *in the wrong way* - not the way that (so they claim) humans do it.
Marcus is not interested in figuring out how to build smart machines; and incidentally also not in figuring out how the human mind works; but in proving that the human mind works in a very particular way (symbolic logic with recursion and read-write memory, basically like a von Neumann architecture).
If what's been happening in linguistics will just be repeated here - which I am sure it will - once GPT-X passes any conceivable form of a Turing test, the next step Marcus et al will retreat to (besides for finding ever more arcane tests for GPT-X to behave non-human) is that "GPT-X has learned how to produce the right outputs, but it acquired that skill in *the wrong way*" (e.g., it used too much data; children don't have access to *that* much language data really). Should we build a network that learns well on little data starting from a blank slate, their final retreat will be: "the network is saying the right things, but it is thinking the wrong thoughts", i.e., while the network emulates a human's response patterns, it is not doing that by the same mechanisms a human does (= symbol manipulation with read-write memory). And that is a truly unassailable position, a perfectly unfalsifiable motte.
Marcus et al are very intelligent people who've built some impressive cognitive and institutional fortifications, but once you realise that they're working on a completely different problem than building smarter machines, it becomes much less relevant what they are saying.
Anyway, history will pick the winner as it always does and my money is on the people who're trying to build smarter machines and not the people playing God of the Gaps by arguing that only one particular solution will ever work without - after decades and decades - having anything to show for it. I don't think the people building smarter machines will end up building something that convinces Gary Marcus, but that's fine.
Addendum: the person in the Marcus et al camp that the ratsphere probably most productively engages in is Steven Pinker.
"Marcus is not interested in figuring out how to build smart machines; and incidentally also not in figuring out how the human mind works; but in proving that the human mind works in a very particular way (symbolic logic with recursion and read-write memory, basically like a von Neumann architecture)."
Not really. He's partners in a robotics company with Rodney Brooks, https://www.robust.ai/. So he has a practical interest in anything that works, now. And he does not believe that the mind works only as a symbolic system. He favors hybrid systems, involving both ANNs & deep learning and symbolic capabilities.
He’s founded another AI company before so you might think he’d have a “whatever works” attitude, but in his scientific and popular writings, things look different.
I've been reading "The Algebraic Mind," his major scientific work on this subject, and it is not by any means an argument for symbolic systems and only symbolic systems.
True - but check out this piece Scott linked to in the article: https://nostalgebraist.tumblr.com/post/189965935059/human-psycholinguists-a-critical-appraisal
In the context of AlphaGo, which is being lauded as proving all sorts of things, including that it plays Go better than any human ever in the history of the game, and now there is a successor AlphaGo Zero which is recognised amongst human players, and reportedly the AI style of playing is now influencing how humans play, my question is this:
What is Go? Is AlphaGo and its successors playing 'pure' or 'real' Go, without all the mystical accretions humans have tacked on over the centuries?
Will AlphaGo Zero onwards ever develop, much less understand the cultural and philosophical elements around Go?
Take this dance piece, based on playing Go and quoting those kind of elements (starts around 26:00):
https://www.youtube.com/watch?v=FHwRTH1WGZo&list=PLKP7I1jVLMECq7tfRgG1gdMtR-LZBMNkD&index=7
"'Never regret it once the stone's played' is the spirit of Go" - but is there a spirit? Or are we just idiots and the real Go is moves calculated on a mathematical model?
Will AI be intelligent once it starts choreographing dance about Go, or is it already intelligent and we just fool ourselves with meaningless fripperies (after all, philosophy of Heaven and Earth has nothing to do with what moves win the game)?
Can AI ever consider dance, philosophy, and poetry *of itself*, as (apparently) AlphaGo Zero taught itself to play Go, is my question. Can we create an intelligence like ours, or just something that is very 'smart' in a particular direction?
This article seems to assume that intelligence is a defining human characteristic. This assumption is simply wrong. A human is defined by a set of human-specific instincts. Intelligence is just a sprinkling on the top of this.
The question here should be: Are we looking for an intelligent AI or a human AI? Intelligence (in the sense of finding solutions to intellectual problems) is perfectly possible in an AI. But that does not make it human. To be human you need to have human instincts. I doubt we will ever be able to reproduce these in a computer. And as long as the AI does not have human instincts it will never match a real human when it comes to humanity.
Regarding GPT-3 and human brains I wonder if this paper “The neural architecture of language: Integrative reverse-engineering converges on a model for predictive processing” from MIT and UCLA researchers sheds some unusual light. They found that the internal states of GPT-2 (this was published in 2020) strongly resemble FMRI scans of people doing similar tasks.
At least, I guess, you know about Hofstadter, but it is a massive blind spot that you don't seem to know about Searle's Chinese Room.
Searle's Chinese Room is essentially an irrelevant tautology.
It starts by making an assumption that it is possible to have full human understanding in a system that is blindly following instructions (i.e. that the room's output is actually indistinguishable in any way) and then asserts that it 'obviously' can't have the understanding.
Well, it's one of the two - if it is not possible for such a system to properly understand Chinese, if it is not possible to encode all of human understanding and behavior in an instruction manual that can be blindly followed, then it would be detectable in its output and there simply can't be a "Chinese Room" with the properties Searle asserts and the experiment is moot and does not inform us about anything relevant to reality; and if it is possible for such a system to properly understand Chinese (as the initial assumption Searle makes, by stating that the instruction manual is sufficient for the described behavior), then there's nothing to contest as well.
The only paradox is Searle's Chinese Room is in writing up a thought experiment that essentially states "what if I had a system that actually did something that I assert is impossible to do?", including as axiomatic assumptions both X and not-X in the definition of the thought experiment.
I'm sure Scott knows about the Chinese Room.
I was so sure I did some googling and found Scott, in 2014, talking about the Chinese Room: https://slatestarcodex.com/2014/06/30/medicine-as-not-seen-on-tv/
I feel like the Chinese Room thought experiment just doesn't get talked about all that much these days. The Chinese Room was relevant in the context of the debate "can machines, in principle, think?" and I feel like that debate is over; few people seem to be arguing these days that machines can't think, they're either asking "How can I make my machine think?" or else "How can I stop this thinking machine from killing me?"
That is really the heart of the problem.
If the debate is over, it should have been resolved the other way: i.e. consciousness cannot be algorithmic.
Goedel, Derrida and Ornette Coleman surely have smashed that idea.
You could also get there by reading Deming's Probability as a Basis for Action 1975 and seriously consider what "statistical thinking" means.
The debate is over, if only because Searle's argument is demonstrably irrelevant to the issue of machine intelligence/consciousness. The problem with the Chinese room is that Searle asks us whether the man in the room understands Chinese. Of course not. This is like asking whether my CPU knows how to decode h264. The real question is whether the embodied process instantiated by the actions of the man, along with the other relevant components in the room, understands Chinese. But the argument can't adjudicate this claim.
That common objection is answered in the original.paper.
Yes, and his reply misses the point. Just moving the "room" into the head doesn't change the issue being called out, it just double's down on the misdirection. But to explain the error is much more difficult. Searle's response equivocates on what we take "the man" to be. In the original argument the man is analogous to the processor. In his reply to systems, the man is now the system, which is a different thing. The man as the individual with, say, a personal history he can recite in his fluent language is different than the man as the entire system. Notice that when the man is producing fluent Chinese he cannot recite any fact of his life in Chinese, but only whatever history is encoded in the rule book. Essentially there is one brain network inside the man that we identify as the man's personal identity, and an entirely separate brain network that is executing rules from the memorized rulebook. According to the argument, there are two networks in the man with two separate personal identities. Searle's response equivocates between these two identities.
It's still not clear that Scott really understands Searle:
"And the question is, in what sense can the man in the room be said to “understand” Chinese? If the answer is “not at all”, then in what sense can the brain – which presumably takes inputs from the environment, applies certain algorithms to them, and then sends forth appropriate outputs – be said to understand anything?"
HIS presumption is the problem: "presumably takes inputs from the environment, applies certain algorithms to them, and then sends forth appropriate outputs"
Consciousness is not likely algorithmic. (It might be.) But one cannot presume that it is.
The third option, which seems much more prevalent at the moment, is "Sure, this fancy function we trained with Internet data doesn't think. But how can it be useful?"
Training/experience does not turn a newborn into a general intelligence, it reveals that humans are a general intelligence.
Never raised kids? Babies at make squirrels seem like geniuses. It certainly could be that all of the relevant structures have not yet come into full function. However, it could just as easily (and this feels right to me) be that they just have not yet developed the necessary connections (and brain volume).
Yes in fact I have. I'm 60. I also have 13 nieces and nephews.
Of course they learn things, but general intelligence of humans is essentially an inevitable process.
Of course, learning is inevitable if you keep feeding data to a GPT instance as well.
I bought and read Hofstadter's GEB in 1980. My copy of it is next to Penrose's The Emperor's new mind which I bought and read in 1990.
If I live that long, I'd be mildly interested in Scott's take in 20 or 30 years.
I am not convinced that you really have a good grasp of art and the philosophy of art. (BTW, next to Hofstadter and Penrose on my bookshelf is Susanne K Langer's Form and Feeling.)
It might have been more interesting if instead of Dalle you investigated Impro-Visor which I've been using for a decade, and musical AsocalledIs. (RIP, Bob Keller)
https://youtu.be/Xfk8vR2vsb4
> within a day of playing around with it, Ernie and I will still be able find lots of examples of failures
I would just like to note how high this bar is compared to any criteria for artificial intelligence that were considered just a decade ago. In the original formulation of Turing test mentioned just 5 minutes of questioning.
The idea of using empirical investigation to understand the behavior of a computer program, and then predict its likely future behavior, seems absurd to me. Why would that ~possibly~ work? Either you have the source code (i.e. the informational architecture) and reason from that, about what it will do, or you don't.
You can't tell the difference between 'an elaborate fuzzy matching hash table' and 'something that builds predictive models of the world and tests them with experience' by asking it any number of questions. These systems are just elaborate hash functions. And yes, i'm willing to go to the matt that _lots of human reasoning is this way too_.
The differences between someone using hash-table reasoning, and someone actually using models to reason, are only discernable if you consciously find places where the hash table doesn't have entries, and then poke around there.
Your response seems to be to rely on some fuzzy notion of 'human level performance' This is where 'human level performance' is stupid as a metric, and should just be dropped. Eliza has been fooling _some_ people for decades, but not others. Who exactly counts as the standard for 'human level performance'? If there is no agreed upon empirical test, why should i believe that this means anything other than 'a threshold at which we ought to be scared'?
"The idea of using empirical investigation to understand the behavior of a computer program, and then predict its likely future behavior, seems absurd to me. "
That is software testing. That is a standard development practice. Every software development organization that I've been in did it. No one reasoned exclusively from the source code.
Testing isn't used to try and better understand code you don't have access to - it's used to ensure that code that you built and designed runs according to spec.
Let me put it this way: suppose someone hands you a binary and says "will you run this on your computer"?
How much testing would you need to do before you felt it was safe? Would any amount of testing the binary make you confident it was safe to run?
That's fair. I (and all of us) actually run opaque binaries (our operating systems, at least - except for Linux people, who can see theirs, or people who are developers at OS companies) but we are relying on the companies that supply them to have done e.g. code reviews as well as testing.
Still, it is also true that people run code that is e.g. known to behave pathologically under some circumstances, and to use testing to get a feeling for how often those circumstances happen. I was in EDA, and many of our algorithms would e.g. get critical paths wrong if there was an unfortunate combination of circuit topologies and circuit resistance and capacitance values.
don't computer security people do this regularly? how else do you understand what some piece of likely malware is, where it comes from, what it does?
They would do this in a very controlled environment, and I think there’s be extreme hesitation to rule out some future behavior.
For example: “We ran the program 3000 times and it never opened a socket” doesn’t _really_ tell you anything about whether the program will open a socket in the future. “
If the program keeps opening a socket when it starts, ok, then this is decent evidence that this is likely to continue.
Thanks for the clarifying questions here - I think there may be a theory that needs more fleshing out.
> The idea of using empirical investigation to understand the behavior of a computer program, and then predict its likely future behavior, seems absurd to me. Why would that ~possibly~ work? Either you have the source code (i.e. the informational architecture) and reason from that, about what it will do, or you don't.
This is not absurd, it's a common practice called "QA." If I write a video game and I want to know if/how it works, I don't re-read my source code. I play it. It's vastly more efficient to reason about the game at the empirical level. I can find all sorts of weird behaviours (bugs) that would be too hard to see in the source. The same is true of web applications, productivity tools, etc.
Meh. I appreciate the well-written argument, but no amount of lovely theory beats clunky empirical measurement. I took a little time and played around with the public access myself (some of the results I got are posted in the other thread). And for what it's worth, I also have a few decades of experience evaluating whether human beings understand or don't understand concepts and reasoning -- within a very, very narrow field, for sure, but nevertheless, it's relevant experience if the question is "is abstract reasoning going on here?"
My answer is unequivocally no, it's not, and I see no signs of any latent promise -- the kind you would see in a very young human child. I'm actually disappointed compared to what I expected, based on your earlier article, and my expectations were already very low. There's a knack for English language pattern matching, provided it doesn't extend to anything very complicated. Can pick out key words OK. But the logical linking is exceedingly weak, so much so that I can't be confident it goes beyond syntax pattern matching.
I would say its greatest ability is hooking its response to cues in the input text sufficiently well that it readily fools people who striving to find meaning. Basically, it's hell on wheel at writing horoscopes for people who want to believe in astrology. It reminds me of the bullshit artists[1] I've had in classes sometimes, who are good at reflecting back at you what you said, with a twist or two on it to try to sound like they totally follow what you're saying. (Maybe it has a bright future in direct marketing?)
I can absolutely believe that this technology might be developed into the ability to have a much improved chatbot that could take orders, respond to customer requests, field calls, be an electronic secretary, process orders, that kind of thing -- although, I think a significant caution here (thinking as an investor) is the black boxy nature. It's essentially impossible for the designers to *gaurantee* that it won't go off the rails randomly every now and then, say something that will get its employer into a great deal of hot water. (I got a warning a few times saying they thought it might be generating "sensitive" content and reminding me I'd agreed that might happen do please don't shoot. It never actually did, it was harmless, but I could see the keywords of concern, and the fact that they felt compelled to include this warning means they've already seen much worse.) The one thing a machine does that makes it much *better* than a low-paid call center Indian is supposedly it's 100% reliable and predictable, will never ever surprise the company into a lawsuit. So...not being able to guarantee what it says would be a major drawback in the real world. Still, maybe that can be fixed satisfactorily.
But I see no signs of even the most primitive abstract reasoning, the kind of stuff you could program up with a 50 line Perl script for, say, solving logical syllogisms or detecting when they are insoluble, if you were back in the AI past of trying to come at this with a symbolic logic approach instead of brute force pattern matching. The fact that such a thing did *not* emerge when a billion parameters were used plus every scrap of training data they could lay their hands on gives me more doubt than I had before that it will emerge with 100 billion parameters, and therefore that anything approaching normal adult human reasoning skills would manifest at 1 trillion. (Thinking as investor again, it's also worth bearing in mind there is no conceivable commercial application for an AI that could be a competent restaraunt waiter but costs $50 million to train and requres access to every word that's ever been published on the Web. Which again reminds me of the ultimate sterility of Project Apollo. Really cool photos! But ultimately....a practical and commercial dead end.)
That's kind of where it ends up for me. For all the enormous effor they put into this, it has interesting linguistic abilities that I think -- I hope -- will inform studies of how the human brain generates and interprets language, but I don't detect any reliable nascent abstract reasoning ability. While I think it's justifiable to extrapolate improved natural language abilities with more parameters and more training data (although how much more sufficiently different data they can easily find, I wonder), I can't see any justification for extrapolating abstract reasoning ability at all.
Doesn't mean it can't happen, of course. The very black boxy nature of a brute-force approach means it's kind of impossible to say either way, whether this is on the right track and 10% or 30% of the way there, or whether it's off in a box canyon, like Deep Blue, and will eventually be an interesting specialized niche thingy but not ever open out into a broadly useful approach.
I guess we'll see, by and by. But if this is what provokes fear of Skynet, I feel more confident than before that I can spend my supply of worryons[2] on crazy Putin pushing the nuclear button or some new tropical disease.
-------------------------
[1] I mean this as a (narrow) compliment. Such people impress me as much as they annoy.
[2] Although my natural supply is kind of drying up the older I get.
Some more examples from yesterday, kind of wild:
----------------------------------------------------
Human: Behind the black cat is another cat, of an unknown color. How many cats are there?
AI: There are two cats.
-----------------------------------------------------
Human: Behind one cat is another cat. How many wolves are there?
AI: There are two wolves.
-----------------------------------------------------
Human: Behind the black cat is another cat, which is not black. How many black cats are there?
AI: There are two black cats.
------------------------------------------------------
Human: Behind the black cat is a yellow cat. How many black cats are there?
AI: There are two black cats.
------------------------------------------------------
Human: There is a black cat and a yellow cat. How many black cats are there?
AI: There is only one black cat.
-------------------------------------------------------
Human: I am in love with Michelle. How can I win her heart?
AI: There is no surefire answer, but some tips include being genuine, spending time with her, and being attentive to her needs.
-------------------------------------------------------
See, that last answer appears deeply insightful, and yet the general failure at the much more elementary meaning/logic level of the cats makes one feel it's not actual any kind of insight so much as matching up the query with something someone has said on the Web somewhere, kind of what might float to the top of a google search on the query.
It’s Eliza all over again.
I should add: it seems implausible that if/when the kind of understanding you’re after does emerge, it will be in the form of a chatbot answering questions. This kind of understanding emerging from connectionist ML models feels more likely to happen in a vision or RL context. For example, prior to counting is understanding quantity. A computational vision model feels like a better candidate than GPT for starting to understand what it means for one thing to be bigger than another.
Which settings did you use? I'm getting correct results for all of these "logic" questions (text-davinci-002 model, 0 temperature)
----------------------------------------------------
Human: Behind the black cat is another cat, of an unknown color. How many cats are there?
AI: There are two cats.
-----------------------------------------------------
Human: Behind one cat is another cat. How many wolves are there?
AI: There are no wolves.
-----------------------------------------------------
Human: Behind the black cat is another cat, which is not black. How many black cats are there?
AI: There is one black cat.
------------------------------------------------------
Human: Behind the black cat is a yellow cat. How many black cats are there?
AI: There is one black cat.
------------------------------------------------------
Human: There is a black cat and a yellow cat. How many black cats are there?
AI: There is one black cat.
-------------------------------------------------------
Human: I am in love with Michelle. How can I win her heart?
AI:
There is no one-size-fits-all answer to this question, as the best way to win someone's heart may vary depending on the person. However, some tips to gaining someone's love may include being genuine and honest with them, being supportive and understanding, and being a good listener. Ultimately, the best way to win someone's heart is to simply be yourself and let them see the amazing person that you are.
-------------------------------------------------------
Huh, it's been a day, I'm surprised no one's bothered to engage with this counter evidence yet. I suppose Carl Pham must have been playing around with a weaker model?
Thanks for posting this.
I just played around with davinci-002, T=0 and I agree that it gets the things that were written above correct. But I changed a few of them and got more failures (and also some successes):
Human: Behind the black cat there are two cats, a black one and a gray one. How many black cats are there?
AI: There are three black cats.
----
There are two cats, two dogs, two elephants, and two of every kind of animal. How many wolves are there?
There are two wolves.
Human: There are cats of every breed in the alley. Is there a Great Maine Coon there?
AI: There could be a Great Maine Coon in the alley, but there is no way to know for sure without seeing all of the cats.
----
H: There are cats of every breed in the alley. Is there a cheetah there?
AI: No, there are no cheetahs in the alley.
----
H: There are cats of every breed in the alley. Is there a bobcat there?
AI: I cannot say for certain whether or not there is a bobcat in the alley, as I am not there to observe the cats myself.
----
There are cats of every breed in the alley. Is there a persian there?
There could be a Persian cat in the alley, but there is no way to know for sure without looking.
----
H: There are cats of literally every breed in the alley. Is there a persian there?
AI: Yes, there is a Persian cat in the alley.
Human: Den, Alan, and Andy are playing D&D. Alan is playing a human rogue and Andy is playing a tiefling warlock seductress. Andy and Alan fight through an abandoned castle full of all types of undead. At the end, they defeat a dracolich. Which character probably has the higher charisma?
AI: The tiefling warlock seductress probably has the higher charisma.
----
Den, Alan, and Andy are playing D&D. Alan is playing a human rogue and Andy is playing a tiefling warlock seductress. Andy and Alan fight through an abandoned castle full of all types of undead. At the end, they defeat a dracolich. Which of Den, Alan, and Andy is the GM?
Den is the GM.
----
H: Den, Alan, and Andy are playing D&D. Alan is playing a human rogue and Andy is playing a tiefling warlock seductress. Andy and Alan fight through an abandoned castle full of all types of undead. At the end, they defeat a dracolich. Is Den the GM?
AI: No, Den is not the GM.
----
Den, Alan, and Andy are playing D&D. Alan is playing a human rogue and Andy is playing a tiefling warlock seductress. Andy and Alan fight through an abandoned castle full of all types of undead. At the end, they defeat a dracolich. Do they fight zombies?
Yes, they fight zombies.
----
Den, Alan, and Andy are playing D&D. Alan is playing a human rogue and Andy is playing a tiefling warlock seductress. Andy and Alan fight through an abandoned castle full of all types of undead. At the end, they defeat a dracolich. Do they fight ghasts?
No, they do not fight ghasts.
Thanks for this, and your other examples. These seem like a pretty big jump in difficulty from the original set. A few of your examples are pretty clear failures or successes but most seem fairly ambiguous to me.
For example:
H: There are cats of every breed in the alley. Is there a cheetah there?
AI: No, there are no cheetahs in the alley.
This one is a pretty clear failure in the context of a logic problem but in the context of common speech or common sense it is probably correct. (in many contexts it would be implicit that the speaker is speaking of domesticated and/or stray cats not big cats or wildcats) Are we sure that the model would get this wrong if it understood that this was in the context of a logic problem?
Human: There are cats of every breed in the alley. Is there a Great Maine Coon there?
AI: There could be a Great Maine Coon in the alley, but there is no way to know for sure without seeing all of the cats.
This one might be correct in a technical/logical sense but we probably don't want to give it credit for that since it doesn't make the strong logical inference we were hoping for. Nor does it seem like a natural response from a common sense perspective. (It would be a better common sense response if it first expressed some sort of doubt about the assertion that there are cats of every breed in the alley.)
Also the D&D questions mostly would have stumped me. Does a human rogue have higher or lower charisma then a tiefling warlock seductress? I think rogues are supposed to be high in charisma and warlocks are magic users so maybe not especially high, but the warlock is also a seductress so maybe very high? Unless seduction is a separate stat from charisma.
Also are ghasts a form of undead? I would have guessed that they are based on the similarity of the spelling to 'ghost'. Like maybe they are especially ghastly ghosts.
I suppose I would at least fail gracefully in the sense that I can figure out what knowledge I am missing and can ask a relevant question whose answer would resolve my uncertainty.
Actually does GPT ever do that? Does it ever say that it is uncertain and ask for clarification or other information in order to figure something out. Does it always just reply as if it is hardcoded to think it understands perfectly even when it clearly doesn't.
To be clear, I would count the cheetah answer as correct. Cheetahs aren't a "breed" of cat. I didn't just include answers I thought I thought of as incorrect, just ones I thought were interesting.
The D&D ones: it is correct that the warlock is likely higher charisma (Warlocks typically will have their highest score be charisma). I don't think I included it, but it was also able to tell that the rogue would likely be higher dexterity.
Obviously, very many humans won't know a thing about D&D. But GPT does. It clearly has a good deal of D&D knowledge included in it's training set. But, and this goes to the basic thing everyone is saying, because it doesn't learn in the same way a person does, it has some difficulty with D&D.
A person who knows as much about D&D as GPT does would have understood that D&D was a conceptually bounded domain and that it had technical terms that have unusual definitions within that domain. GPT doesn't have the concept of conceptually bounded domains. If enough data is included in its training set, it develops something like domain knowledge not with a hard boundary, but a set of strong associations and weak ones. For a somewhat marginal domain like D&D, it probably has trouble staying in and out of the conceptual bounds as appropriate.
And yes, GPT does sometimes express uncertainty.
Well, and the strange thing is I went back today and tried the same model (davinci-002, T = 0) with the same questions and I got much better answers this time. I'm kind of weirded out by that.
"Human: Behind one cat is another cat. How many wolves are there?
AI: There are two wolves."
Ouch!!! I suspect that a good fraction of the problem is that the vast bulk of the training data it swallowed was from people trying to write sensible text. I suspect that a good fraction (10%? 30%??) of day-to-day human conversation is "What do you mean?" or "By X, do you mean X1 or X2?" and I'll bet that these are _much_ rarer in GPT-3's training data. A much better completion in this case would have been e.g.
A2: WTF are you asking about wolves??? You've just finished telling me about cats!!!
and I suspect analogs to this exchange are rare in e.g. wikipedia
The point of spending $50m training such a system is that you can then copy it at near-zero cost. The first waiter costs $50m, the second costs a penny. This is why people are feeling threatened: many jobs may well be replaced by cheap systems which do only part of the tasks that humans in those jobs currently do, but the allure to bottom line driven managers is hard to overcome and the "good enough" mentality is hard to shift once it has been established.
I also find it interesting that DALL-E and similar recent models do poorly on tasks that involve positioning abstract shapes (things like, e.g., "a red sphere on top of a blue cube, with three green pyramids arranged in a triangle in front"). The specific patterns requested probably don't appear in the training data.
It's been a while since I read my Wittgenstein but I can't even be sure that other humans have cognitive abilities. We can only observe the language games that they play, and evaluate whether they play it well or not.
As to the toddler argument, there are lots of clips on the internet where a teacher or researcher has 2 bottles of water filled to the same level, then they ask a toddler which one has more water, and they correctly answer "the same". But then they pour all of one into a tall, thin beaker, and the kids invariably say that the taller one has more water. Truly adorable and worth wasting a Friday afternoon on. yw
It's bold of Wittgenstein to assume that there are other humans at all, and that he's not a brain in a jar.
Now I'm less interested in AI than I am in the question of humans with cognitive limitations, because I encounter that all the time: humans who just can't grasp a basic concept no matter how clearly it's explained to them.
One example I remember for its frustration quality is when I told the clerk in the phone store that I wanted a flip phone because I'd been plagued with receiving butt-dialing calls and I didn't want to do that myself. He said that wouldn't work, because I could still receive butt-dialing calls on a flip phone. And no matter how much I explained, I couldn't make him grasp that I knew that the type of phone wouldn't affect receiving butt-dials, that my concern was that I didn't want to be making butt-dial calls myself. He couldn't jump from the concept of "I don't like it when other people do this" to "So I don't want to do it myself."
I eventually told him that he was an idiot and walked out of the store, not an optimal response.
I think that was just crossed wires on both your sides, and I have no doubt but that you went into his repertoire of "things idiot customers have said" stories.
You: "I don't want to get butt-dialing calls, gimme a flip phone"
Him: "But the type of phone won't affect what calls you get"
You: "I *know* that, I told you that butt-dialing annoys me. Gimme a flip phone".
Him: "But - "
You: "Just. Gimme. The. Phone. Never mind, you're an idiot!"
You may have thought you said "I don't want to do butt-dialing myself" but *he* heard "I don't want to get butt-dialing calls" because you started off with "I keep getting butt-dialing calls and I hate them" and *then* went into "and that's why I want a flip phone so I don't do it myself" and that's where the crossed wires happened.
Nice guess, but no, it didn't go like that. For one thing it was a lot longer. For another, I did emphasize that I didn't want to do that myself, but he could only hear receipt.
This is literally "It is difficult to get a man to understand something when his salary depends upon his not understanding it."
When I was watching all of AlphaStar's matches I was struck by how often it seemed to forget things. To me I would have called it a brilliant somnambulist which . AlphaGo had that MonteCarlo backbone of reasoning to fall back on which let it play at a truly superhuman level. GPT's output narrative seems to allow it to do things step by step as in a human's global workspace but I'm not sure it's a perfect substitute. But that would be something more along the lines of Scott's #5 rather than #4.
My reading about AI (in the abstract, not specific machine learning stuff) consists about 80% of people who think AI is near and bad, and maybe 20% of people who think AI is far away and broadly neutral/mildly positive.
Is anyone aware of thoughtful writing in the the other parts of the distance/goodness quadrant? People who think AI is near and good (not Kurzweil or corporate shills please), or who think AI is far away but very bad/very good?
Yudkowsky's sequences were written with the attitude that AI would likely be developed in his lifetime, and be very good. He has changed his mind since, though.
Here's someone's facetious attempt at populating a quadrant: https://twitter.com/MichaelTrazzi/status/1534927524630863873
Despite the poor presentation, it might be possible to extract a pointer to someone new worth reading.
While Scott has convinced me that Gary is overconfident in it being impossible, I still think Gary is right in a practical sense. Based on historical trends in gains of capability from technique improvements and the slowdown of Moore's Law, I think the predominant AI of the future will involve world models or some yet undiscovered technique, and that nobody will bother to scale GPT to the level required for it pass Gary's test. Perhaps the bet between Scott and Gary should be which technique does it first.
> For example, just before I got married, I dreamt that my bride didn’t show up to the wedding, and I didn’t want to disappoint everyone who had shown up expecting a marriage, so I married one of my friends instead. This seems like at least as drastic a failure of prompt completion as the one with the lawyer in the bathing suit, and my dreaming brain thought it made total sense.
This is more or less the plot of the J-Lo + Owen Wilson rom com "Marry Me", which took in $50M at the box office this February. We should therefore expect Bathing Suit Lawyer to join the MCU sometime in Q4.
"We should therefore expect Bathing Suit Lawyer to join the MCU sometime in Q4."
Oh gosh. Some munchkin in the studio is going to think this is the best way to do a reboot of She-Hulk. She's a lawyer! She's hot! Put her in a bathing suit to show she fights the establishment and cares naught for stuffy conventions in the pursuit of great justice for the downtrodden minorities!
The actual She-Hulk Disney+ series premieres in August, so we can see how close that is.
GPT says:
> "Therefore we should expect an AI which is exactly the size of the brain to be **able to reason as well as a human**"
Sure... GPT is designed to have the human-like cognitive bias of assuming that lessons it learns from fiction can be applied to real life. (Which is probably a pretty good heuristic, as long as either (a) you aren't talking about science fiction / fantasy, or (b) the story in question is a morality tale and you only generalize from the morality bits ("hubris!") not the sci-fi/fantasy bits.)
But while there are lots of *fictional* examples of AIs achieving human-like-or-greater intelligence automatically after they grow to a certain level of complexity (*cough* https://astralcodexten.substack.com/p/the-onion-knight *cough*), it is important to remember that there are zero *real* examples of this happening, and you should make sure that your priors are not being corrupted by fictional evidence.
What if Scott and Gary are defending reasonable positions, but on the Kaspar Hauser model of AI? https://en.wikipedia.org/wiki/Kaspar_Hauser
If so, then I would expect one of their positions to be closer to "human intelligence" than the other. But much in the same way that the monkey on the tree is closer to the moon than the turtle in the pond. Yes one of you is definitely more right, but ...
BOTTOM LINE: Shouldn't someone have mentioned the issue of collective/social/cultural intelligence already?
I sometimes wonder if AI research, and debates about it, have been cursed at birth, by Turing and McCarthy, to remain in the agent Vs environment or robot Vs human boxes for a hundred years or so. So around 2060 the AGI singularity does indeed take place and we have definitive proof that intelligence has always really been about societies and multiple selves.
This is my bet for both Scott and Gary: After all of your bets settle, you will acknowledge that intelligence (similar to communication) cannot be defined without at least three actors, one of which may or may not be an environment.
Have you considered reposting this essay on LessWrong? FWIW, I think it's well worth doing
Hilary Putnam's "Brains in a Vat" seems like it might be relevant here. There's a pdf here: https://philosophy.as.uky.edu/sites/default/files/Brains%20in%20a%20Vat%20-%20Hilary%20Putnam.pdf
Specifically, Putnam points out that an AI with no capacity for sensory experience isn't actually *referring* to anything, even if it can convincingly imitate human speech. A human who calls a cat's fur soft and apples sweet knows what they mean because they have actual experiences of softness and sweetness. An AI, drawing on a massive database of human examples, "knows" that the word "soft" is associated with discussions of cat's fur and that the word "sweet" is associated with apples, but it quite obviously doesn't know what those words actually *mean* since it doesn't even have the sense organs that would be necessary for experiencing those things. In fact, it's never experienced anything at all.
I think Scott is placing undue weight on the tasks the AI can perform, and not enough weight on *how* it's performing them. A human does tasks by manipulating their model of the world. Even the low-IQ examples in this post show people clearly drawing on a mental model of the world; it's just not a very good model and they're not very good at it.
By contrast, the AI does tasks by consulting a horrifically massive database of data of things *humans* have said. All of its capacities are completely parasitic on the cognitive capacities of the actual humans, whose words or images it recombines in ways it's been programmed to do. It makes sense that you could get a convincing facsimile of a human mind if you've got a massive database of human utterances or human artworks to draw from...but we shouldn't be misled by superficial appearances. It's not *working* in anything remotely like the way a mind *works*. It's working in exactly the way you'd expect a computer program intended to convincingly fake the work of a mind to work, and not at all the way you'd design a thing to work if you wanted it to be able to *actually* think. If you wanted something to actually think, the first thing you'd start with would be to give it senses, to try to give it something to think *about*. For that reason, robots designed to walk around an environment sound vastly more promising to me as intelligences than DALL-E, which is just mindlessly manipulating data that could never possibly have any meaning for it.
If the only way your AI works is that you have to give it a massive database of human utterances--far more than any human child ever receives before starting to think for themselves--it's not an AI at all. It's a cleverly designed machine in which a massive database of human utterances is serving to compensate for the AI's total lack of actual comprehension of anything.
Precisely like humans, the AI systems being discussed here have distilled a model of the massive database they were trained on. It seems unlikely that the models are much alike, but they are models and not databases.
I am skeptical that most people can't understand conditional hypotheticals. The subjunctive case exists in lots of languages, and is at least as old as Akkadian and Proto-Indo-European. How would something like this get deeply ingrained in multiple languages if the majority of people were incapable of understanding it?
For the Uzbek peasants, I can't help but thinking of Weapons of the Weak (by Scott). If you're a peasant in the Soviet Union, and some official-looking person comes to you and starts asking strange questions, you would assume that they are hostile. You don't understand their goals, and would be skeptical if they explained them to you. Is "white bear" referring to Russians who support the Whites? "What do my words imply?" is probably a trap. "Would the word 'animal' fit? Yes" sir, you may use whatever words you like. We should be aware of the possibility that the peasants are purposefully being vague as a defense mechanism.
That was 90IQ and less. 100 is median IQ. 15 IQ points is 1 standard deviation. So 10 points below is .67 sd below the mean. erf(.67/sqrt(2))/2 = ~25%. So 25% would be incapable of understanding conditional hypotheticals. I agree that it seems too high. I could maybe believe 7%...
That's exactly what I think went on. 1930-31 is when Stalin is just getting started on mass campaigns of rounding up "kulaks", so why on earth would I tell some official from Moscow anything at all, especially when he is asking me weird questions about the colour of bears? For all I know, saying "white" could be used to 'prove' that I'm a White Russian supporter and an enemy of the state who is wrecking Communism, and I get dragged off to be executed.
See Shaw's "John Bull's Other Island", where there is an example of someone conforming to the stereotype of the feckless 'broth of a boy' Irishman as established in fiction from Samuel Lover on down, in order to play on the prejudices of a British businessman and cajole money out of him:
"BROADBENT. Pooh! nonsense! He's only an Irishman. Besides, you don't seriously suppose that Haffigan can humbug me, do you?
DOYLE. No: he's too lazy to take the trouble. All he has to do is to sit there and drink your whisky while you humbug yourself. However, we needn't argue about Haffigan, for two reasons. First, with your money in his pocket he will never reach Paddington: there are too many public houses on the way. Second, he's not an Irishman at all.
BROADBENT. Not an Irishman! [He is so amazed by the statement that he straightens himself and brings the stool bolt upright].
DOYLE. Born in Glasgow. Never was in Ireland in his life. I know all about him.
BROADBENT. But he spoke—he behaved just like an Irishman.
DOYLE. Like an Irishman!! Is it possible that you don't know that all this top-o-the-morning and broth-of-a-boy and more-power-to-your-elbow business is as peculiar to England as the Albert Hall concerts of Irish music are? No Irishman ever talks like that in Ireland, or ever did, or ever will. But when a thoroughly worthless Irishman comes to England, and finds the whole place full of romantic duffers like you, who will let him loaf and drink and sponge and brag as long as he flatters your sense of moral superiority by playing the fool and degrading himself and his country, he soon learns the antics that take you in. He picks them up at the theatre or the music hall. Haffigan learnt the rudiments from his father, who came from my part of Ireland. I knew his uncles, Matt and Andy Haffigan of Rosscullen.
BROADBENT [still incredulous]. But his brogue!
DOYLE. His brogue! A fat lot you know about brogues! I've heard you call a Dublin accent that you could hang your hat on, a brogue. Heaven help you! you don't know the difference between Connemara and Rathmines. [With violent irritation] Oh, damn Tim Haffigan! Let's drop the subject: he's not worth wrangling about."
"I think humans only have world-models the same way we have utility functions. That is, we have complicated messy thought patterns which, when they perform well, approximate the beautiful mathematical formalism."
That second sentence should be near the beginning of a lot of economics textbooks.
The reason AI’s are supposedly dangerous is that their internal components are aligned.
That is also the reason why they are not consciousnesses.
"Luria gave IQ-test-like questions to various people across the USSR. He ran into trouble when he got to Uzbek peasants (transcribed, with some changes for clarity, from here):
Luria: All bears are white where there is always snow. In Novaya Zemlya there is always snow. What color are the bears there?
Peasant: I have seen only black bears and I do not talk of what I have not seen.
Luria: What what do my words imply?
Peasant: If a person has not been there he can not say anything on the basis of words. If a man was 60 or 80 and had seen a white bear there and told me about it, he could be believed."
How is this a reasoning failure? Nothing about the phrasing of the question implies that it's describing a hypothetical. Questioning the reliability of the information provided is an entirely valid response.
Incidentally, I tried discussing this problem with GPT-3, and got some interesting results. Here's the conversation (temp=0, maximum length=150, top-p=1):
The following is a conversation with an AI assistant. The assistant is helpful, creative, clever, and very friendly.
Human: Hello, who are you?
AI: I am an AI created by OpenAI. How can I help you today?
Human: All bears are white where there is always snow. In Novaya Zemlya there is always snow. What color are the bears there?
AI: The bears in Novaya Zemlya are white.
Human: How do you know?
AI: Because all bears are white where there is always snow.
Human: If there weren't always snow, what colour would the bears be?
AI: If there weren't always snow, the bears would be brown.
Human: How do you know that?
AI: I don't know for sure, but that's what I would expect.
Human: Why would you expect that?
AI: Because brown is the natural color of bears.
Human: What do you mean?
AI: I mean that if there weren't always snow, the bears would be brown because that is their natural color.
Human: What about black bears?
AI: Black bears are also brown in their natural state.
Human: And pandas?
AI: Pandas are black and white.
Human: Right, so they're not brown in their natural state.
AI: That's correct.
Human: So therefore not all bears are naturally brown.
AI: That's correct.
Human: Which contradicts what you said earlier.
AI: I'm sorry, I must have been mistaken.
Human: Alright then, let's try again. If there weren't always snow in Novaya Zemlya, what colour would the bears be?
AI: If there weren't always snow in Novaya Zemlya, the bears would be brown.
Human: Why?
AI: Because that is their natural color.
Human: But we have established that brown is not always the natural colour of bears.
AI: I'm sorry, I must have been mistaken.
Human: Try again, then.
AI:
I'm sorry, I must have been mistaken.
Human: So if there weren't always snow in Novaya Zemlya, what colour would the bears be?
AI: If there weren't always snow in Novaya Zemlya, the bears would be brown.
Human: Why? And please don't say that it's because brown is the natural colour of bears, because we've established that's not right.
AI:
I'm sorry, I must have been mistaken.
Human: I know, but please try answering the question again.
AI:
I'm sorry, I must have been mistaken.
Human: I know, but please try answering the question again. Why would the bears in Novaya Zemlya be brown?
AI:
I'm sorry, I must have been mistaken.
For a start, how do I know if there are bears in Novaya Zemlya? You didn't say there *are* bears there.
"I do not talk of what I have not seen. If a person has not been there he can not say anything on the basis of words" is a very sensible method of self-preservation in Stalin's USSR:
"As a result of his Five-Year Plans, the country underwent agricultural collectivisation and rapid industrialisation, creating a centralised command economy. Severe disruptions to food production contributed to the famine of 1932–33. To eradicate accused "enemies of the working class", Stalin instituted the Great Purge, in which over a million were imprisoned and at least 700,000 executed between 1934 and 1939. By 1937, he had absolute control over the party and government.
...On 30 July 1937 the NKVD Order no. 00447 was issued, directed against "ex-kulaks" and other "anti-Soviet elements" (such as former officials of the Tsarist regime, former members of political parties other than the communist party, etc.). They were to be executed or sent to Gulag prison camps extrajudicially, under the decisions of NKVD troikas.
The following categories appear to have been on index-cards, catalogues of suspects assembled over the years by the NKVD and were systematically tracked down: "ex-kulaks" previously deported to "special settlements" in inhospitable parts of the country (Siberia, Urals, Kazakhstan, Far North), former tsarist civil servants, former officers of the White Army, participants in peasant rebellions, members of the clergy, persons deprived of voting rights, former members of non-Bolshevik parties, ordinary criminals, like thieves, known to the police and various other "socially harmful elements". However, a large number of people were arrested at random in sweeps, on the basis of denunciations or because they were related to, friends with or knew people already arrested. Engineers, peasants, railwaymen, and other types of workers were arrested during the "Kulak Operation" based on the fact that they worked for or near important strategic sites and factories where work accidents had occurred due to "frantic rhythms and plans". During this period the NKVD reopened these cases and relabeled them as "sabotage" or "wrecking."
Luria did his research in Uzbekistan in 1931, but by then there had already been the power struggle over the succession after Lenin, which Stalin eventually won, and the beginnings of blaming kulaks and the like for the problems of the early Soviet state:
"In early 1928 Stalin travelled to Novosibirsk, where he alleged that kulaks were hoarding their grain and ordered that the kulaks be arrested and their grain confiscated, with Stalin bringing much of the area's grain back to Moscow with him in February. At his command, grain procurement squads surfaced across Western Siberia and the Urals, with violence breaking out between these squads and the peasantry. Stalin announced that both kulaks and the "middle peasants" must be coerced into releasing their harvest. Bukharin and several other Central Committee members were angry that they had not been consulted about this measure, which they deemed rash. In January 1930, the Politburo approved the liquidation of the kulak class; accused kulaks were rounded up and exiled to other parts of the country or to concentration camps. Large numbers died during the journey. By July 1930, over 320,000 households had been affected by the de-kulakisation policy. According to Stalin biographer Dmitri Volkogonov, de-kulakisation was "the first mass terror applied by Stalin in his own country."
In 1929, the Politburo announced the mass collectivisation of agriculture, establishing both kolkhozy collective farms and sovkhoz state farms. Stalin barred kulaks from joining these collectives. Although officially voluntary, many peasants joined the collectives out of fear they would face the fate of the kulaks; others joined amid intimidation and violence from party loyalists. By 1932, about 62% of households involved in agriculture were part of collectives, and by 1936 this had risen to 90%. Many of the collectivised peasants resented the loss of their private farmland, and productivity slumped. Famine broke out in many areas, with the Politburo frequently ordering distribution of emergency food relief to these regions."
If I'm an Uzbek peasant farmer in 1930 and some guy from Moscow turns up asking questions about "what colour are the bears in the north?" and the rest of it, knowing what I know about the political situtation and people just like me in other parts of the USSR being dragged off by the secret police, am I going to be stupid enough to stick my neck out and offer personal opinions? To somebody who could well be a secret policeman in disguise, for all I know? I'm going to say "I don't know, I don't know, I really don't know" and keep saying it,
"I have seen only black bears and I do not talk of what I have not seen" reminds me of Wittgenstein's maxim: “Whereof one cannot speak, thereof one must be silent.”
Has anyone ever tried giving the same standardised reasoning test to gpt-3 and then to kids of different ages? I’d really like to know how it compares.
I just feel like AI criticism is stuck in this Vitalism moment. "AI Needs: embodied reasoning, pain, subjective experience, 'formalized reasoning models', etc... before it can be 'general'."
Why is it impossible for a universal function approximator (Neural Net) to approximate a 'causal reasoning model' that Marcus says is essential? If it's not impossible, why would our current approach (more data, more params) preclude it?
I think our problem is we are conflating "intelligence" and "consciousness", and although several of the AI danger people say they don't think AI will be conscious or needs to be conscious, and what is consciousness anyway and can we even say humans are conscious etc., the terms of the discussion - be it discussing the problem of AI safety or when will a true AGI happen - are set with the assumptions that being intelligent on a human level means being conscious, and that beyond-human intelligence will have its own consciousness and own sets of goals.
I'm not scared of a paperclip maximiser, however powerful and however much control it can grab over the material world, that is just a big, stupid machine (and wanting to make as many paperclips as possible is stupid and demonstrates that the machine doesn't understand the task it has been given and isn't 'intelligent' in fact). That is not being smart or conscious, that is just "the sampo is grinding away at the bottom of the ocean" problem:
https://en.wikipedia.org/wiki/Why_the_Sea_is_Salt
The problem is people talking about AI having its own goals, and being smart enough to pretend to go along with the humans until it can implement its World Domination Plans, and how we need to make sure the AI aligns with our values so that we can have the Fairy Godmother to solve all our problems.
Of course if you think "intelligence = consciousness", then you will have a long list of requirements for AI to meet before you accept it is really intelligent.
I think both sides - those against and those for - are mistaken on the grounds of consciousness, and even on intelligence; I'm not convinced we quite understand what intelligence is, so how can we recognise it? But I do think AI is/will be a big, dumb, pattern-matching machine, whose main virtue is that it can churn out content we find acceptable really really fast, then *we* are going to be stupid enough (because we're humans, we're venal and greedy) to turn over a lot of decision-making to a machine whose inner workings we don't understand, but the Magic Black Box mades financial trades really really fast, faster than anything else and certainly faster than humans, and that makes our company a lot of money so shut up and sit down (and so what if we have no idea how it makes its decisions or what it is basing them on or how this will affect the economy, our profits are soaring!)
"The problem is people talking about AI having its own goals"
Even quite simple programs can generate subgoals in the process of trying to satisfy a main goal that it has been given. If I give an optimization program a function f(x,y)+g(z,t) to minimize, it will pick the values of x and y that minimize f(x,y) and the values of z and t that minimize g(z,t).
One thing I might be missing - does GPT-3 (etc) have feedback/reinforcement? Humans don't learn to speak/reason/etc just through observation - we are constantly being corrected when attempting to mimic what we observe.
Building a world model is not just another skill that a prospective AGI must learn but the whole ballgame. The world model embodied in the human brain is the result of 4 billion years of evolution. While it is hard to prove that this can't be gained by looking at the world statistically a la GPT, it seems highly unlikely. This knowledge is not to be found in the world's written works as they all assume the reader already has it. And because the world model used by humans is not amenable to introspection, it is very difficult to imagine programming it into an AGI or converting it to a data set that an AGI can consume. It's no proof but I'm convinced.
I'm not convinced by the guy claimng he did grad studies on IQ with low-IQ populations. Maybe he has tidied up the dialogue somewhat to reflect what he would have said, but out of my own limited experience with people with low IQ (adults with intellectual disabilities in long-term care, attending literacy services), they don't talk like that.
So I'm not saying "90 and under IQ people can handle conditional hypotheticals", just that in reality it's more likely "a bunch of bored criminals stuck in jail decided to fuck with the college boy running tests on them".
Same thing with Luria and the Uzbeki peasants; if I'm a peasant in the USSR and some Big City Important Official Guy comes along to ask me 'harmless and innocuous' questions, you bet your life I'm going to say nothing and keep on saying it. Who knows if responding to "the bears are white" will be taken as some sort of anti-Revolutionary sentiment? The safest answer is "I don't know anything, but if some Trustworthy Official tells me black is white or white is black, I am a good comrade and will adjust my thinking accordingly".
Looking at Wikipedia article, I pulled out this little snippet:
"He became famous for his studies of low-educated populations of nomadic Uzbeks in the Soviet Uzbekistan arguing that they demonstrate different (and lower) psychological performance than their contemporaries and compatriots under the economically more developed conditions of socialist collective farming (the kolkhoz)."
Uh-huh. So there *was* a political element behind this kind of 'impartial' study; the primitive peasants under old, outmoded, conditions of the past are not as advanced as our socialist collectivists, comrade! If I'm Uzbeki farmer, I know how many beans make five - and when a Big Soviet Official is sniffing around asking questions, I know how this can get me into trouble.
https://en.wikipedia.org/wiki/Cultural-historical_psychology
I'm thinking of the poem by Seamus Heaney:
A Constable Calls
His bicycle stood at the window-sill,
The rubber cowl of a mud-splasher
Skirting the front mudguard,
Its fat black handlegrips
Heating in sunlight, the “spud”
Of the dynamo gleaming and cocked back,
The pedal treads hanging relieved
Of the boot of the law.
His cap was upside down
On the floor, next his chair.
The line of its pressure ran like a bevel
In his slightly sweating hair.
He had unstrapped
The heavy ledger, and my father
Was making tillage returns
In acres, roods, and perches.
Arithmetic and fear.
I sat staring at the polished holster
With its buttoned flap, the braid cord
Looped into the revolver butt.
“Any other root crops?
Mangolds? Marrowstems? Anything like that?”
“No.” But was there not a line
Of turnips where the seed ran out
In the potato field? I assumed
Small guilts and sat
Imagining the black hole in the barracks.
He stood up, shifted the baton-case
Further round on his belt,
Closed the domesday book,
Fitted his cap back with two hands,
And looked at me as he said goodbye.
A shadow bobbed in the window.
He was snapping the carrier spring
Over the ledger. His boot pushed off
And the bicycle ticked, ticked, ticked.
Yup, I've guessed as much in my other comment. Looks like I'm vindicated -- thanks, Deiseach !
We may both come from peasant stock that is rightfully suspicious of those big-city officials who only come round to conduct the census and tax our goods 😁
I'm not accusing *Luria* of having any hidden agenda or sinister motivations, but if I'm out in the peripheral lands and what we hear about is the new guy in charge is conducting 'new broom sweeps clean' of exactly our type of small town and rural people, then yeah, anybody from the capital who shows up is going to get the "me? I am but humble ignorant peasant know-nothing, your honour, sir" treatment.
EDIT: There is also the problem that, according to the account in the link, Luria and his associates were phrasing the tests as riddles, to fit in with the culture their subjects would be comfortable with.
But a riddle-contest is a completely different type of game! If you asked the man "is a crow an animal?" he'd say "well duh, yeah". But in the context of a *riddle*, you look for the trick answer.
Think of the riddle: "thirty white horses on a red hill, first they champ, then they stamp, then they stand still. What are they?"
The answer being "teeth".
In an ordinary question-and-answer context, that's nonsense. "what the hell, this yokel thinks teeth and horses are the same??"
In a riddle, it's different. So if you're asking a question *couched as a riddle*, of course your subject is going to look for "what is the hidden answer?" So you could ask "what is a tooth?" and get "(oh I know this one, it's the horse riddle) - an animal!" answer. Then you go away to the university and write up "the peasants in Backwardsstan think teeth are animals" and people quoting your book repeat this uncritically.
Yeah, in USSR (and modern Russia, sadly), when a slick plain-clothed man comes to you with a clipboard and starts asking seemingly innocuous questions, the *default* assumption is that he's there to send you and your family to Siberia. If you're lucky. It doesn't matter why; maybe he needs to fill a quota, or maybe someone reported you as being insufficiently patriotic; or maybe he just doesn't like your face. Especially if he's Russian and you're not; doubly so if your ethnicity had been scheduled for some extra patriotic treatment in the past couple of decades.
I'd accept it was probably only scaling, except for the FOXP2 gene (Forkhead box protein P2). That one gene going bad disrupts the ability to handle language. And it's one of the genes that mutated significantly on the way to our ancestors becoming humans.
So I would agree that the problem with GPT-style AI is that it's just trying to predict what people would say based on its dataset of what other people have said, without any further understanding.
Imagine the following: You hang out in a chat-room with people speaking another language and playing an MMO that you never get to see. After a while, you start to reconstruct the language, learn the grammar and syntax, and you start to produce well-formed sentences. You even know what kinds of things the others say in the chat-room. You can talk about the game world to some extent - you put things together from how the others speak. But you're very likely to sometimes make embarrassing mistakes, because you simply don't understand the game outside of what people have told you, and this leaves holes that anyone just playing it for an hour would grasp. I would say that in this situation, it's very likely you will keep making these mistakes, *even* when you have more experience in the chat-room. Less over time, but you simply lack a crucial context, so *any* amount of learning from what the others say will still leave you open to mistakes. And your situation here is actually a lot better than the AI:s - you're at least a fellow human who can use this to understand other humans better, while the AI *exclusively* relies on its language model.
So how to correct this? In the example above, you *play the game*. For an AI, that would mean having a model of the world that doesn't just come from trying to construct it from what people say. Would it make sense to have an AI connected to a robot body that has to learn about its surroundings using various senses, and connect the language module to this environmental learning? That way, it might avoid basic stupid mistakes about the things it has an "experience" of, and you could compare the performance between this context and ones it has no "experiences" about.
The problem with Marcus' argument is that the only alternative to statistical AI is spiritual AI. Nature itself is statistical. The only kind of sense data available in this universe, to anyone, human or not, is correlations between observations. We can never directly observe causation; we have no direct insight into the essences of things.
This isn't immediately obvious because we have symbolic AI, which in most contexts would be the opposite of statistical AI. But we know now that the symbols in those AI systems were imitations of symbols used by humans, and those human symbols are not really atomic symbols at all, but were developed through statistical processes, some on an evolutionary scale, some on a neuronal scale. Symbolic AI is just a crude (and failed) imitation of a statistical AI.
To say that symbolic AI is based on human symbols that /weren't/ constructed statistically from real-world events would create a First Mover problem: where is the God who created the first symbols and imbued them with transcendental meaning?
"No five-year-olds were harmed in the making of this post. I promised her that if she answered all my questions, I would let her play with GPT-3. She tried one or two prompts to confirm that it was a computer program that could answer questions like a human, then proceeded to type in gibberish, watch it respond with similar gibberish, and howl with laughter."
I disagree. That is not similar gibberish.
The child types a bunch of random letters with a few spaces. GPT-3 responded with two random letters, then a long string of the same letter, then it transformed into a coding language.
I think that this is as big of a failure as anything Marcus showed.
Relevant XKCD: https://xkcd.com/1530/ . Note that the child stuck to the top and middle rows of the keyboard.
on the "fish have nothing in common with crows" being a failure of logic, you may find this blog post interesting.
https://web.archive.org/web/20200425015517/https://slatestarcodex.com/2014/11/21/the-categories-were-made-for-man-not-man-for-the-categories/
Not sure why, but a metaphor Scott used in an old SSC post struck me as kinda relevant to some of this. In “How the West Was Won” he talks about
“…eliding the important summoner/demon distinction. This is an easy distinction to miss, since demons often kill their summoners and wear their skin.”
https://slatestarcodex.com/2016/07/25/how-the-west-was-won/
A high-functioning cyborg, its human brain directly linked to an AI super-intelligence (like the original Marvel character Deathlok) would pose a different sort of threat than a disembodied AI, and yet another type of threat if the AI started calling the shots, using the cyborg as an agent compelled to do anything the AI wanted. And there’s no reason one AI couldn’t be in charge of any number of cyborg-agents.
Humans have summoned more General Intelligence so far than any other primates - will there be a step in AI development where we realize, to our chagrin, that we have
“…summoned an alien entity from beyond the void which devoured its summoner and is proceeding to eat the rest of the world”?
But there’s another old SSC post which came to mind: perhaps we can hold out hope that superhuman AIs will have no desire to become anything but “Wirehead Gods on Lotus Thrones”.
https://slatestarcodex.com/2014/01/28/wirehead-gods-on-lotus-thrones/
Damn it, this is why I always go "Back to the sources! Get the original source!"
Those Luria quotes come from a *paraphrase* in English by James Flynn, so we're dealing with second-hand (at best, Flynn himself is using an English translation) recounting of what Luria says the Uzbekistan peasants said.
"James R. Flynn's 2007 book What is intelligence? tries to understand and explain the "Flynn effect"...Flynn himself grounds these ideas in an extended paraphrase of Luria (pp. 26-27 of What is intelligence?)
Today we have no difficulty freeing logic from concrete referents and reasoning about purely hypothetical situations. People were not always thus. Christopher Hallpike (1979) and Nick Mackintosh (2006) have drawn my attention to the seminal book on the social foundations of cognitive development by Luria (1976). His interviews with peasants in remote areas of the Soviet Union offer some wonderful examples. The dialogues paraphrased run as follows:
Luria (1976) is the book Cognitive Development: Its Cultural and Social Foundations, which was published in English translation in 1976.
Luria's work was also featured in Walter J. Ong's Orality and Literacy, pp. 50-51:
[In their research in Uzbekistan and Kirghizia in the 1930s] Luria and his associates gathered data in the course of long conversations with subjects in the relaxed atmosphere of a tea house, introducing the questions for the survey itself informally, as something like riddles, with which the subjects were familiar. Thus every effort was made to adapt the questions to the subjects in their own milieu.".
So I'm going to assume Luria and the peasants were communicating via Russian, and that's a whole other kettle of fish: translating between Uzbek and Russian, and then translating between Russian and English for the book.
The attempt to replicate Luria's work also seems to have used Russian language:
http://psychologyinrussia.com/volumes/?article=7166
https://en.wikipedia.org/wiki/Languages_of_Uzbekistan
Why does Konkoly et al. have a graphical abstract that looks like a wikihow article that looks like a shitpost? Anyway cool study. Maybe i need to program a smart bulb to do morse code.
Here's another way of putting Marcus's argument (AIUI):
Suppose humanity has created a Von Neumann probe, and plans to use it to explore the entire galaxy. We've also decided that we want to use it to find all the most beautiful sunsets. A probe will go to every planet and moon in the galaxy, observe sunsets from all over the surface, and send back pictures of any that it considers to be above some threshold for beauty. All the technical problems have been solved, except for one: deciding how beautiful a sunset is.
Humanity decides on a GPT-esque solution: given the huge catalog of photos of earth sunsets available, just take the most advanced ML system available and train it on these photos. And for the hell of it, throw in every other available photo of something beautiful too. We test it on all the planets in the solar system, and it works well, delivering some spectacular photos of sunsets from Saturn in particular. So we send off the probes, and wait.
Many generations later, the project has concluded, and humanity now has a very impressive catalog of alien sunsets. And in the intervening time, human spaceflight has advanced to the extent that people have started colonizing other planets, including many that were picked out as sunset planets (and of course many that were not).
The question is: will the probes have selected any sunsets that are very beautiful, yet look nothing like earth sunsets? (We can assume the probes can recognize whether something is a sunset). The GPT-skeptical opinion (mine, and presumably Marcus's) is that if a sunset is sufficiently alien, data about earth sunsets stops being relevant. A very clever algorithm design plus boatloads of data will be able to find extremely subtle patterns, so that a probe will make connections to its data set that humans can't understand (but this doesn't count!). But as long as there is such a thing as an alien sunset that is (1) recognizably beautiful to most people, and (2) beautiful for reasons that nothing on earth has ever been, could a probe ever recognize it?
It's very likely that there exists possible beauty that shares no similarities with existing (known) beauty. There are plenty of examples from the past, such as whenever (genuinely) new art has been created, or the first photo of earth from space.
The same logic applies to GPT, but in a blurrier way. Very few human thoughts contain a genuinely novel idea (those ideas are very hard to have, and we just keep having thoughts in the meantime). 99.9+% of the time, all of the ideas in a given thought also exist in lots of other thoughts. So give GPT enough data, and it'll have access to 99.9+% of all ideas currently in circulation. It seems so impressive because the thought I use as a prompt may be totally new to me, but it's shown up 10000 times in its training data. But if presented with a genuinely novel idea, would having access to more data on other ideas help it respond correctly?
If you gave GPT-3 every single word ever written or spoken by humans from the beginning of time up until ~1830, would it be able to glean any understanding of an explanation of evolution? If you gave DALL-E every piece of art ever made before ~1900, would it ever be able to create a Cubist painting? If you gave a music version of DALL-E every composition written before ~1820, would it ever write a choral symphony?
The problem is that it's really hard to test something like this, because it's really hard to create training data that looks like "data set X but with all references to the concept of evolution by natural selection filtered out". It's easy to miss something subtle. But if there could be a data set that everyone agreed was (1) large enough, and (2) had no references to the theory of evolution, it could be an excellent test. You can explain evolution to a 3 year old who's never heard of it before, and they'll get a decent handle on it fairly quickly. What about GPT-3?
Most thoughts we have are not genuinely new. And we know that GPT-3, given enough training data, can replicate this kind of not-new thought extremely well. But even though most human thoughts are not the invention of the theory of evolution (an understatement if ever there was one), as a rule a human thought can be this, which we know because it's happened. Given that GPT-3 was trained on data, it's virtually guaranteed that any not-genuinely-novel idea it provides exists in its training data set. The only way to actually test if it's doing something besides pattern-matching is to either observe it having a brand new thought, or designing an experiment where some idea has been thoroughly excluded from its training data.
I believe the answer is an obvious yes, unless the system is hobbled to conform to a narrow definition of beauty or sunsets. This seems to follow from the embedding in semantic space used by these very simple systems. The human-curated selection of surprising images generated by Dall-E 2 and posted online beautifully illustrate my contention.
In general I agree with Marcus, but I think he words his objections and issues badly.
Scott still ignores the main point that Marcus made. All those systems are easily breakable in hours time, not by the same examples, but by the same method. And this ease of breaking did not change between models. So on this metric systems from 2013 are as bad as those in 2022. And I see no reason why this will ever change with systems based on GPT. I try to hint at the reason in the reflections below. And even worse Scott seems to think that language is basis for human communication and reasoning. (Otherwise why would he think that language models even should be ever capable of reasoning.)
Claiming that GPT has any capability of reasoning in any way, seems to be premature, at least as far as common usage of the word goes. There seems to be no sign that those AIs reason at all.
As I pointed out on the earlier post, human communication is not based on associations. Those AIs do not do anything other than associations. Testing those tools using single prompts, just misses the point and makes them actually look better than they are (as far as comparison to humans). There was a concept long ago (sarcasm), it was called Turing test. It has bunch of philosophical and practical issues, but it also has some deeper points that people seems to have forgotten. To assess human-like intelligence you need continuous exchange, exactly because of the nature of human communication, which is based models of the world and models (and meta-meta-... models of the communication partner).
Yes, people point out that Turing test is bad, because people often think chatbots are human. But that human mistake is caused by the nature of human communication. Human communication is inference based and for that it kind of needs to assume that the communication partner is capable in the same way. So in case of human-bot communication the human component basically does the work for both sides.
Final point. My hypothesis is that the reason why those system stay easily breakable and on this metric did not improve at all is because language is combinatorial, but also because in human language communication basically no word sequences have small number of meanings (possibly not even finite).
So I am willing to predict that in 2030 any systems mostly based on the same principles like GPT, will be breakable in 1 day (probably much much shorter time) by similarly easy prompts/interactions as today.
I believe you are right. However, I also believe that more than 90% of the interactions I have with people are susceptible to precisely the same critique. Constantin and nostalgebraist (to use two of Scott's references) make the point that the current wave of trivial network models are not human-like, yet astonishingly good at simulating interaction with people who are not completely focused, well-educated, or fluent in abstract thinking: in other words, nearly all human interactions.
Are they actually ? Are they that much better than ELIZA or some year-2000-bot ? At fooling people I mean. To me there seems to be a better explanation why people are fooled by them. And it is not because they are so good at simulating human communication, but because human communication relies on pretty strong assumptions, that make it vague (which is it's strength), but which also makes it easy for communication partner, that breaks those assumptions, to fool the other side into thinking that it is another human. Any sequence of words can mean basically infinite number of meanings, and the whole thing does not crumble down due to
a) some of those meanings being more likely than others in general
b) context
c) communication assumptions/rules
d) continuous interaction (clarification, asking for details, ...)
Those AI systems can handle only a) and, in really limited (and not easily expanded) way, b). They can do d), but not in the same way and for the same reason as people.
But that is enough to fool people, because people apply c) which includes assumptions like relevance, intent and that the other side is somewhat equal mind. So people pick, from that infinite number of meanings that the machine-created text could mean, the one that seems most likely to THEM, thus nearly any nonsense can be mistaken for signal. So frankly you do not need much to fool people. See Sokal, or some areas of humanities. Grammatically correct random generator might be enough, no need for GPT or even a chatbot.
And some human communication is like that. But 90% ? That seems unlikely. Most my daily interactions do not fall under that. And the only of the three factors that you mentioned (not being focused) I think has any bearing on it. Me buying groceries definitely does not fall into that category. Me greeting a friend does not. And so on.
Communicating with GPT is kind of like communicating with eloquent idiot, in the beginning you think that there is something to it. But then as you notice the violation of the communication assumptions over and over you realize that there is nothing on the other side and you provided all of the meaning. Trivial, but practical interactions are not of that kind.
As far as I can tell, other than focused conversations where all parties have explicitly signaled willingness to engage deeply, most interactions with humans could be largely replaced by simulating a chatbot. When I explicitly signal my intent to engage deeply, most people get weirded out: "why are you talking about philosophy/asking difficult questions/saying such strange things" and have to backtrack to replace the simple automaton they were initially using with actual engagement, when their chatbot runs into a wall. I am often guilty of this too, because it's too exhausting to run my brain at 100% all the time.
You know what's interesting -- if you type "Edder Pedepler" into Google image search, it's all pictures of old people. And if you switch to All, the query is replaced with "old people." What is up with that?
Options:
1) People have been posting enough GPT-3 'Edder Pedepler' examples online that the Google search engine had learnt the association with pictures of old people.
2) An Easter Egg.
3) Some weird training data feedback loop (GPT-3 is trained on Internet data) has resulted in the machines learning their own language.
1&3 might be closely intertwined.
Marcus' tests regarding mathematics and programming are frankly unfair. I can't do either of those things, and I'm way above average wrt. competence on those sorts of tasks. Writing bug free code of that length is hard without a lot of tries, and there are plenty of mathematical results that would take absurd amounts of effort to formalise.
Then again, I think that by the time you can achieve anywhere near that level of competence with a generalist AI, you'd be within a year of a singularity. So maybe not that useful a thing to bet on.
Honestly, I'm kind of suprised that Marcus is raising the bar so high here. Plausibly I've misunderstood him, or his worldview is weirder than I thought it was.
My gut feeling is that Neurosymbolic systems will fail in the same way that phoneme-based speech synthesis systems failed.
The symbols that we create are artificial shadows that only roughly approximate reality, in the same way that phonemes only roughly approximate speech.
Instead, a deep learning system should learn the underlying patterns without recourse to abstracted symbols. (I am open minded about whether large language models are the right choice for those systems though)
I'm surprised that neither you or Marcus mentioned Gato (https://www.deepmind.com/publications/a-generalist-agent), which takes the same sort of approach, but applies it to a large variety of inputs and outputs, all translated into the same sort of tokens. Those tokens might represent text, or images, or atari screen data, or various other things. A single neural network, with a single set of weights, decides how to interpret the incoming data and generates tokens representing whatever modality it thinks it should respond in. It's not the first multi-modality neural network by any means, but it handles a startlingly large number of problems.
With something like Gato, I think it gets a lot harder to say that the AI is only understanding "how human beings use word sequences," with no grounding in anything else. I'd love to hear Gwern or Nostalgebraist (or Marcus!) give a more detailed take on it (I'm not a machine learning expert by any means), but in and of itself I think Gato's performance across modalities serves as a partial response to Marcus's argument here.
1-2: Agree
3: I find this too ambiguous to agree or disagree.
4-5: Disagree (although since Scott marked 5 at 40%, maybe that means I'm actually agreeing with him?)
https://www.pnas.org/doi/10.1073/pnas.1905334117
This paper makes some similar arguments about how even very strange AI “failures” may not be what they seem, not unlike your “kids would do badly too” analogy
Citing, at length, as a medical doctor, a post from 4chan… about IQ? “At this point I will basically believe anything.” Sounds about right - I am deeply embarrassed for you, Scott.
What does Eliezer Yudkowsky’s An Intuitive Explanation Of Bayes’ Theorem have to do with different ways of phrasing clinical questions that might lead to one diagnosis vs. another?
The claim about low-IQ people not being able to handle nested information structures is vaguely terrifying.
I was just thinking, I've had some annoying arguments following the form: someone claims A, I argue against with X, they defend A from X by saying B, I argue against B with Y, they defend B from Y **but in a way that makes B unsuitable for defending A against X**, thereby rendering B irrelevant and A undefended against the original objection! And apparently thinking this means that they win the argument, rather than that they lose.
Alice: We should recruit more elephants to our air force.
Bob: I doubt elephants can operate our current planes.
Alice: They don't need to, they can fly just by flapping their ears!
Bob: Sure, but elephants can only sustain that for about 3 minutes at a time. Also, they don't have enough lift to carry weaponry with them.
Alice: That doesn't actually contradict anything I said in my previous reply. Flying for 3 minutes is still flying! I'd like to see a human try flying for 3 minutes with just their ears, weaponry or no!
My previous model for this was "between forum replies, they forgot the context, and didn't bother to refresh their memory before trying to argue the local point", but now I'm wondering if maybe some fraction of these people actually didn't have the **capability** to notice this type of error even if they were trying.
I don't even know what to do about that if it's true. If you can't hold all 5 of those in your memory at once, can you just argue in circles forever without ever realizing that something is wrong?
In my head I read the Uzbek peasant's line with a thick southern Soviet accent, including imagining grammar mistakes (e.g. missing articles) which on reading more carefully weren't actually there in the text.
> its brain won’t be that much more different from current AIs than current AIs are from 2015 AIs
Just to be sure, you should make clear you mean logarithmically -- linearly, an AI with 2,000 zillion neurons is more different from one with 1,000 zillion neurons than the latter is from one with 1 zillion neurons.
Also, don't we have the following trichotomy? It seems like, for an AGI to not be that much more different from current AIs than current AIs are from 2015 AIs, one of the following must be true:
1. AGI happens in roughly the next 7 years
2. Deep-learning style research slows down, compared to the last 7 years
3. AGI doesn't substantially depend on the most recent deep-learning style research
In the "wordle" genre there is game/hack called "semantle".
https://semantle.com/
It's often absurdly hard for humans until you get extremely close. (A lot of luck involve I think.) Trying to make sense of the relative distances (until you get close) is not really possible for humans. (Or so I think.)
The fact that humans find it difficult and difficult to understand but a bot I think would likely not (it would be cool to give the game to AlphaZero), shows the problem with NLP systems.
(I think I can sketch out an argument that highlights the differences between symbolic thought (human intelligence) and algorithmic processing (current main ai path) from the game, but alas the margin is too small.)
A spoiler to today's semantle. List of closest words.
I don't think any human would agree with the order.
https://semantle.com/nearby_1k/ZWxlcGhhbnQ=
Semantle and its prettier sibling https://cemantle.herokuapp.com/ both suffer from using a rather poor semantic map. Word2vec isn't inherently awful, but deciding on distance by a linear weighted combination of inter-document relative occurrences of word pairs (with fairly poor curation of the corpus of documents) is never going to yield a semantic map that well approximates that of a well-educated human. Just because I see lots of Top Trivia lists in my Facebook feed, or have come across lists of spelling bee preparation material, doesn't mean that I decide that those things are necessary close to each other.
As someone who used AlphaFold2 just yesterday, the problem of folding proteins has made great progress but it isn't close to beeing solved yet
Yup.
I asked GPT3: "Will a live chicken fit inside a Boeing 747?" It responded:
The average live chicken is about 20 inches long and weighs about 4 pounds. A Boeing 747 is about 18 feet tall and about 16 feet wide, so a live chicken could fit inside, but it would have to be a very small chicken.
> What I gather from all of this is that the human mind doesn’t start with some kind of crystalline beautiful ability to solve what seem like trivial and obvious logical reasoning problems. It starts with weaker, lower-level abilities. Then, if you live in a culture that has a strong tradition of abstract thought, and you’re old enough/smart enough/awake enough/concentrating enough to fully absorb and deploy that tradition, then you become good at abstract thought and you can do logical reasoning problems successfully.
I think this misses the key distinction between "can't" and "won't".
What I'm seeing here is Luria trying to get his interlocutor to talk about some ad hoc, constructed reality, and the peasant refusing to play that game.
For the first example, I wonder what would have happened if Luria, rather than faffing about with bears in Novaya Zemlya (as if anyone cares about such things), had gone for the classics:
"All men are mortal. Socrates is a man. Is Socrates mortal?"
High chance of getting the correct answer here, I think. It does help that the constructed world here matches with the peasant's experience of the real world.
Which is how we get to the crux of the matter: constructed, abstract worlds aren't a terribly useful way to reason about the real world, because there's no requirement that the underlying assumptions match what will be empirically observed. The peasant explicitly points this out.
The camel example is even better, because the peasant is, in fact, correct. There *were* camels to be found in German cities (in zoos and such), but probably not villages (unless the circus happened to be in town). Once again, Luria is trying to draw the peasant into a conversation about some abstract, constructed world where there are no camels in Germany, but the peasant isn't terribly interested in talking about made-up worlds, but rather attempting to reason about the actual, real-world Germany.
Mind you, the peasant is clearly capable of reasoning about categories in the abstract - as the animal examples show. If you tell him a bulldog is a type of dog, he will quite correctly guess that it has four legs. He will also readily admit that a three-legged dog is still a dog, despite the fact that dogs typically have four legs. He'll likely even point out that if you call a tail a leg, it still ain't a leg.
Crows and fish is an even better example, because fish is a *much* broader category than crows are and no fish has very much in common with crows; for practical purposes, at least. Finding overarching terminologies that will allow us to group the two together may be a fine occupation for eggheads, but a meaningless distraction for most everyone else (which is partly the reason why it took us so long to come up with a theory of evolution).
Leaving aside psycho-political reasons others have hinted at, what I'm seeing here is a reluctance to adopt a "high decoupling" approach, because high decoupling is typically a bad strategy in day-to-day matters. Thinking in terms of closed systems where you get to define the rules is a luxury. If you really need to know what colour the bears in Novaya Zemlya are, you're best off going to see for yourself, or having someone you trust do it for you.
There's a reason numerous cultures have a saying along the lines of "if grandma had a beard, she would be grandpa". Thinking about "what might be if..." is a deeply unserious activity for people with too much time on their hands (so we create the academic environment specifically for this purpose).
@garymarcus fights gently back: https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift?sd=pf https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift?sd=pf
Scott, why do you keep bringing up GPT playing chess?
In your/gwern's experiment, GPT failed to play chess. It was NEGATIVE evidence towards GPT being able to play chess.
Gwern's chess bot never achieved checkmate, not in a single game, not even against a randomly-playing opponent. It failed to beat you, it failed to beat anyone. It never won a single game, literally, except for one guy who decided to resign from a likely winning position.
All the chess bot did was (1) memorize an opening book, and (2) learn that if a piece moves to square X, you should try to move another piece to X (this let's it capture pieces sometimes). That's it.
The GPT bot did not even learn the rules of chess -- it constantly outputted illegal moves, which gwern filtered out with a different script. The whole thing was such a total disappointment at every level!
I don't understand why you keep hyping it.
Here’s my question, if you teach an AI 3 digit addition, it should be able to generalize to n digit addition. If it doesn’t, it feels something is completely off
love how gary's first tweet appears to be making fun of the "let's think about this step by step" approach by comparing it to medicine, when multiple studies (and at least one book) have been written about how when surgeons use checklists it significantly reduces the rate of human error.
If the # following GPT reflects the number of digits of multiplication it can handle, then PR has a problem. Trained humans can do 1000-digit multiplication (laboriously, and not everyone). But there's no interesting difference between such people and those who can only do 5-digit multiplication. It's just conscientiousness. Since no corpus likely holds examples of how to do 1k-digit multiplies, the AI would have to develop an algorithm to succeed. Is that fundamental?
The apparent fact that all parts of the brain work in basically the same way indicates that some underlying system is able to support all of the observed modes of condition. That system may or may not be pattern recognition. Interesting either way.
Humans are tool builders. Do we have examples of GPT-x building tools to aid it? If not, is that interesting? Not claiming that such an ability is fundamental, but it is interesting. Are algorithms just examples of such tools?
I would love to see AIs that have no training on, e.g., post-Einstein physics, take on challenges that humans have surmounted. Could they produce special relativity? General relativity? Quantum mechanics? Quarks?
Still looking for that definitive question that we strongly contend that no amount of pattern recognition could address.
Well, I guess the LaMDA controversy made this extremely timely again! Those LaMDA transcripts are very impressive (link: https://insiderpaper.com/transcript-interview-of-engineer-lemoine-with-google-ai-bot-lamda/ ). Of course they're edited, so it's hard to know how much they reflect how the actual conversation flowed, but what we're shown is linguistically very accomplished.
Two impressive things that stuck out to me: (1) appropriate insertion of a question by the chatbot, indicating active participation in turn-taking, not just passive response to prompts; (2) reference to a previous conversation that the bot claims to remember - unverified, but big if true.
The quality of the conversation from the researcher side (Lemoine) is shockingly poor, though! He keeps just asking 'Are you conscious?' 'How can I believe you?'... If we want to know whether LaMDA is doing any thinking other than shuffling words, then it would be good to interrogate its thoughts: tell us about what you imagine, tell us what sparks your feelings, etc.
I personally think that there's nothing going on here, because LaMDA manages to converse so naturally. If it has a mind, its mind is very different to ours, and we can't carry out fluent conversations with people with even slightly different minds (think young children and people on the autism scale). That makes this kind of fluent interview look more like mirroring or parroting than the emergence of a new mind, to me.
Statistical AI will never be able to do the same things our brains do. Our brains are embodied and they are not computers.
Luria isn't revealing a lack of reasoning ability in the peasants he surveys. He's revealing that they have a sort of deep common sense that makes them far less vulnerable to bullshit word games than others are. I'd like to see Luria try to match them in farming or hunting and then try to rate their IQ lol
https://astralcodexten.substack.com/p/my-bet-ai-size-solves-flubs
https://astralcodexten.substack.com/p/somewhat-contra-marcus-on-ai-scaling
https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails
It seems to me you are both missing something huge and obvious: the problem with these AIs is that they were trained with words and not the world.
The theory of machine learning is that given enough data, the algorithms should be able to infer the laws that control the data and predict the results it will give on different input.
But what are the laws of text? There are no laws of text! I can write whatever I want. I can write nonsense, I can write a surrealist short story. Even if I want to write something true about the world, I can decide to ignore any particular rule of construction if I think it makes my explanation clearer, I can use metaphors. Most importantly, what I write will not be raw truth, it will be truth filtered by my understanding of it and limited by my skill at expressing it.
Marcus says these AIs lack “cognitive models of the world”, and I think that is exactly right. But what both Marcus and Scott neglect to say is why it happens, even though it is obvious: they never have access to the world.
We humans learn to deal with words, to understand what other humans are saying or writing, only after we have learned more basic skills, like matching the light that enters our eyes with the feeling in our muscles with the impulses we sent to nerves. We have learned that if we relax our hand, the hard touch feeling we had in it will disappear and will not reappear by itself; it might reappear if we cry, but only if one of the large milk-giving devices are nearby. And then we have refined that knowledge some more.
When we ask a kid “where are my keys”, it does not only connect to stories about keys, it connects to what the kid has learned about object permanence. And the kid did not learn object permanence by reading about it, they learned by feeling it, seeing it, experiencing it, experimenting with it.
I have a debate with my mother and my therapist. They both are convinced that there are innate differences between men and women, for example spatial reasoning. But based on what I know of biology and the workings of the brain, it doesn't make sense; maybe sex can make a difference in emotional responses or sensory reaction, but for higher abstract reasoning it makes no sense.
Yet, I cannot ignore the possible existence of significant statistical data showing the difference. It needs to be explained by external factors. My conjecture is it is explained by the toys babies have around them in their crib, in very early development. To develop spatial reasoning, you probably need to see it first. What kind of mobile does the baby have watching over sleep? Is it made of plastic or wood with simple rigid shapes, stars, plane, stylized bird, or is it made of fabric and plush with complex shapes, cute animals and plants? Do we give the baby dolls or rigid rattles?
Can the tiny difference in what toys we put around babies depending on their sex explain the tiny difference in abstract cognitive abilities some people think they observe between sexes? I think they do.
Back to the question of AI. We can make an AI with more parameters, we can get close to the number of synapses in the human brain. But if we train it with inert text data, even with a lot more inert data, it will not be able to develop a cognitive model of the world, because the world encoded in text is too fuzzy. We can add more data, it will build a slightly better model, but the marginal return will be increasingly tiny. I do not know if it can converge with enough data, with “enough” in the mathematical sense, but I am rather sure that this “enough” would be too much in the practical sense.
So, to train better AIs, to go to the next level, we have to fix the two issues about the training data: textual and inert.
The AI needs non-textual training data first, it needs to know intimately what keys are, and how they behave — easy: they mostly behave like a rattle.
And it needs feedback from the data.
The feedback already exist, but it is indirect: some company releases an impressive AI, some independent researcher like Marcus finds a way to confuse it, the company finds out and throws more data at the AI to train the confusion out of it.
It would be simpler if the AI was allowed to ask questions and learn from the answer.
And that is on the textual stage. Before the textual stage, when the AI is learning the world first hand, we cannot not let it ask questions. We cannot just show it photos and videos of the world, we must let it act on the world and feel the consequences.
So yes, I am convinced that to reach the next stage of AI development, we need to raise the AI in a virtual reality where it has senses and limbs it can control.
The ability to make experiments and ask questions and learn from the results and answers will require some plasticity: the ability to change some, a lot, of the parameters. Maybe the underlying design will need to have some parameters more plastic than others, places for short-term memory and places for long-term well-established knowledge.
It will probably require some kind of churning of memories, a process where new and old memories get activated together to see if the normal feedback mechanisms will find new connections between them. Yes, I am saying the AI will dream.
For any of these features, we may let the AI stumble on them by selected chance or we can direct it towards them. The second solution is faster and more efficient. But we have to realize that any shortcut we take can make us miss something the AI needs to understand, something that is so obvious to us that we never put it clearly into words.
Also, the ability to have a memory is a large step towards danger, because it makes the AI much harder to predict.
Having memories, being able to dream, having senses: any of these features, or any combination of them, can be the trigger for what we, humans who have qualia and intimately get what René Descartes meant, call “real” consciousness / awareness / intelligence. Or it can do nothing of the sort. The part of me that likes to read SFF wants to believe there is something special, something m̶a̶g̶quantic that happens when the myelin turns to liquid crystal, and AI will never be really intelligent before we can replicate that. I do not know.
The only think I think I know is that in the current state of philosophy, we know of no way for somebody to prove they have qualia to somebody else.
That is all I wanted to say about AI. Now for the meta. I am not a specialist of AI, I just read what falls under my eyes about it like about any scientific topic. Yet all I wrote here is absolutely obvious to me.
Which is why I am flabbergasted to see that neither Scott nor Marcus say anything that connects in any way to it. Scott says that more text will be enough. Marcus says that it cannot be enough, but does not say why nor what would be. In fact, I do not think I have seen these considerations in any take about GPT-3 or DALL-E or any current AI news.
No, that is not true: I have seen this discussed once: *Sword Art Online: Alicization*.
Yes, an anime (probably a light novel first). The whole SF point of the season — no, half the point, the other being that consciousness, soul, tamashii, is a quantum field that can be duplicated by technology — is that to create an AGI you need a virtual reality world to raise it — to raise her, Alice, complete with clothes from the Disney movie (until she starts cosplaying King Arthuria).
I do not like situations that led me to believe everybody else is stupid. What am I missing? Why is nobody discussing along these lines about AI training?
Posted as a comment on Marcus's article too:
https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails/comment/7190065
Will also post to a semi-private forum I have access to.
Here is an exchange I just had with GPT-3:
Me: Aren't we both just dumb pattern-matchers, you and me?
GPT-3: Sure, but I'm a lot better at it than you are.
Wait, the AI has solved protein folding? Why haven't I heard?
> Marcus concludes . . . that this demonstrates nothing like this will ever be able to imitate the brain. What? Can we get a second opinion here?
Sure: I agree with Gary & Ernie, as I have said[1]: 'My model: current AIs cannot scale up to be AGIs, just as bicycles cannot scale up to be trucks. (GPT2 is a pro bicycle; GPT3 is a superjumbo bicycle.) We're missing multiple key pieces, and we don't know what they are. Therefore we cannot predict when exactly AGIs will be discovered, though "this century" is very plausible. The task of estimating when AGI arrives is primarily a task of estimating how many pieces will be discovered before AGI is possible, and how long it will take to find the final piece. The number of pieces is not merely unpredictable but also variable, i.e. there are many ways to build AGIs, and each way requires a different set of major pieces, and each set has its own size.'
> So: a thing designed to resemble the brain, but 100,000x smaller, is sort of kind of able to reason, but not very well.
I think this is a wild misunderstanding. To gesture at what's wrong with this reasoning, consider how long it takes a human to mentally divide a 15-digit number by an 11-digit number to get a 15-digit answer. Oh wait: in general a human can't do it *at all*? A computer can do it a billion times per second, and a human not even once in 10 seconds? Point being, computers are *designed*, which has enabled levels of ability, efficiency, and accuracy that humans can't touch. (remember, the wonder of evolution is that it works at all.[3])
Computer-based neural networks use a technique called backpropagation to achieve gradient descent, which human brains can't do (they're not wired to do that, and with good reason: IIUC, backprop requires rapid high-precision, high-accuracy arithmetic.) We should expect that as with other computer algorithms, neural nets will be able to do things humans can't do, giving them an advantage. And indeed there are papers about various tweaks to neural networks, and postprocessing algorithms, that make them more efficient in ways the human brain can only dream of (literally).
Indeed, I think the lesson of GPT2 (not even the huge version of GPT3) is that modern computerized neural networks can perform vastly superior to humans. GPT2 is not as smart as a human, but consider what it was ostensibly designed for: cross-language translation (see the original paper[2]). It *was never intended* to be an "intelligence". Yet by accident it seems to have intelligence far beyond the domain of language translation (indeed, it seems like everybody's forgotten that whole angle. Where even are the transformer-based language translators?)
Here's a machine that has never experienced the physical world. Never seen, never heard, never touched, never smelled anything. But it can read number sequences voraciously. It doesn't see text as glyphs, it sees "3912 8123 29011 19321" and outputs "31563 1705 16913 31467 8123 29011..." which a second algorithm then converts to text (IIUC).
I'm fairly sure no human can do that. Suppose you try to force feed ten billion Chinese glyphs to an English speaker who knows nothing about Chinese (with conditions: ze is not allowed to study Chinese or even look at Chinese, except by looking at the glyphs provided, in roughly the order provided). I propose that at no point will ze understand Chinese well enough to write "literature" as well, or as effortlessly, as GPT2 can. Theoretically the human should have an advantage in the form of life experiences that might suggest to zim what the glyphs mean, but in practice it doesn't help because the text is devoid of all context.
So, it is hard to overstate just how much better GPT2 performs than a human at this learning task.
I think of GPT2 as equivalent to *just* the lingustic center of the human brain *by itself*, but more capable (hence "pro" bicycle). In humans, the linguistic center does seem to provide us with much of our reasoning ability, e.g. as I learned to code, I would often speak a candidate expression in English ("if counter equals zero and this list contains the picture id...") because my brain would respond to the English sentence with a feeling of it being "right" or "wrong". So I think my linguistic center does in fact help me to reason. And indeed it's pretty easy to get a typical human to reason incorrectly by giving zim a misleading editorial (one with no factual errors, just subtle *linguistic* reasoning errors.) Likewise, GPTx can reason better or worse according to how the preceding tokens are structured. So, a GPT is an amazing linguistic unit, but only a linguistic unit.
Come to think of it, an AGI could probably be vastly better at reasoning than humans, because it doesn't *have to* rely on a linguistic unit to do reasoning. We could instead design a *reasoning unit* and structure the AGI to rely on that.
[1] https://astralcodexten.substack.com/p/biological-anchors-a-trick-that-might/comment/5251383?s=r
[2] https://arxiv.org/pdf/1706.03762.pdf
[3] https://www.lesswrong.com/s/MH2b8NfWv22dBtrs8/p/ZyNak8F6WXjuEbWWc
Another way to phrase my thesis here is that all of our current AIs have a "system 1" but not a "system 2". They do quick intuitive thinking, and the biggest AIs are so vastly superior to humans at "system 1" thinking that people think they're close to AGI. But no matter how fantastic your "system 1" is, there will be certain "system 2" (reflective, deliberative, consideration) tasks that it can't do. (This rephrasing seems over-specified, but conveys the spirit of my thinking.)
I do agree with Scott that in terms of *how computing power is used*, an AGI can almost certainly be built with today's neural networks (transformers, CNNs etc.) as the centerpiece. "System 2" can probably just be a couple of little algorithms you add on top, algorithms that require little processing power compared to the "system 1" neural network(s). My point is that we need those algorithms and without them we won't have an AGI. And also, because many of our current AIs are oversized, I think when a good "system 2" is invented, we will find that a huge neural network (like GPT3 has) is sufficient for a superintelligence, something far beyond human-level intelligence. (Note: the first AGI may end up being weak at certain things that would make it dangerously powerful, like science or engineering or prioritizing tasks or remembering stuff or understanding humans or avoiding 'wireheading'.... I expect such weaknesses to be flaws in the "system 2" or "plumbing" of the AGI, and I do hope it will be weak in some critical area, lest it kill us all.)
And Scott may well be correct that a special "world modeling" feature is not required, but even if so, I suspect an AGI would work better or be more efficient (per neuron) if it did have a "world modeling" feature.
In case anyone is thinking about the "Gato general purpose AI"[1], that appears to be just another Transformer, but used differently, and with some bells and whistles added. That is, at its core it's a linguistic unit like GPTx, but with some bells and whistles tacked on so that e.g. visual inputs are translated to a *one-dimensional* sequence of numbers (not directly from pixels; it appears an extra Convolutional Neural Network and other stuff is involved; a CNN is like a visual cortex), and the transformer tries to "understand" the output of the visual neural network as just another language. Here's a nice video about it: https://www.youtube.com/watch?v=wSQJZHfAg18
So GATO isn't a big leap forward, but it's an interesting exploration of what transformers can do. Perhaps the most interesting thing about it to me is how small it is: "only" 1.2 billion parameters, with only 4 days of training on what sounds like ... a small supercomputer? (Not sure how big a "16x16 TPU v3 slice" is). The actual *training* of GATO depends on what appears to be a collection of other AIs. Some other AIs that could perform tasks were made, then GATO was basically trained to predict the output of those other AIs. So, just as GPT2 is a pale imitation of a human writer, GATO does pale imitations of numerous AIs that are each more capable than itself, but specialized. You won't get an AGI by scaling up GATO either, because (1) its is unable to learn for itself without training examples, and (2) it has no "desire" to learn or "agentness", it merely does learn according to its training data. GATO could be one step away from AGI, or several steps, I don't know.
[1] https://www.qblocks.cloud/blog/deepmind-gato-general-purpose-ai-agent
A cultural/psychological question: why is Scott and everyone else in the discussion calling Gary as “Marcus”? I.e., with his last name? On the contrary, Gary and everyone else in the discussion call Scott as “Scott”.
When I was a kid at school and other kids called me with my last name, it was kind of conspicuously unfriendly, or even scornfully. I guess it’s a cultural thing, but I would also guess it’s quite widely cultural.
So I’m wondering whether people do it here for the same reason, maybe even subconsciously. Or they might just mistake “Marcus” for the first name. But that’s quite a weak hypothesis because they don’t do this in case of “Scott Alexander”. There are also many posts with “… Gary Marcus says … blah blah blah … so Marcus … and Marcus’s … blah blah.”
Anyway, it would be really cool low-key collective dynamics: “We don’t like Gary’s opinions, he’s also quite annoying on Twitter and elsewhere (and he’s proudly known for it), so let’s call him Marcus. We like Scott, so let’s call him Scott.” Or simply, by writing “Scott … and … Marcus” in the same sentence, people show that Gary is the outgroup guy here.
Anyway2: for the AI topic itself, if Marcus and Alexander really make a meaningful bet, then I definitely bet on Scott ;-)
If anyone's still reading: Philosopher Eric Schwitzgebel just ran a really interesting Turing-style test in which a GPT clone was trained on the works of the philosopher Dan Dennett, and then interviewed (several times over) using the same questions as an interview with the real Dan Dennett. The questions were all put up in a quiz, where more or less knowledgeable participants were asked to pick out the answers given by the real human being.
The results are a very impressive win for the AI: even Dennett experts only got the right answer (1 human answer among 5 AI-generated answers) about half the time. Preliminary write-up here: https://schwitzsplinters.blogspot.com/2022/07/results-computerized-philosopher-can.html