Both Hindu theologians (See 'Nondual Tantric Saivisim' by Wallis) and Thomas Aquinas predicted that inference-based systems won't be able to 'know' because 1 isn't a probability you get to through incremental updates. Both of these schools of thought claim there's something else that _can_ operate behind the mind, but without that thing in operation, you're going to end up acting more like the LLM than whatever it is that Iblis is pointing at.
I'm at one. It's dope. It allows me to task risks others can't imagine :)
I bet you have it too. How much would someone have to pay you in order to torture a kid? If the answer is 'I would not do that for any amount of money', congratulations, you've gotten to 1.
Anything is arguably evil if you don’t believe that good means something real. And yes, I have no problem saying I refuse to torture a child, regardless of whatever else might come of it. I don’t care if anyone thinks that makes me evil. I call that person a fool.
There’s a world of difference between unintentionally hurting someone is a byproduct of productive activity, and intentionally inflicting harm on them. Torture, in particular, means that your goal is to inflict pain. It’s not an accidental byproduct. It might be the instrument of something else, but still the goal is inflicting pain. I agree that being unwilling to inflict pain at all is unworkable.
You're both wrong. Morality is about harmony, not harm. Sets of principles which lead to stable sustainable states are good, those which do not are bad, regardless of the experiences or preferences of any of the constituent entities.
Is a set of principles that lead to sustainable impoverishment better than one that leads to some people being slightly better off than impoverished and others wildly better?
I can say with perfect certainty that I would never torture a kid (or anyone for that matter), assuming no more or less obvious insanely huge opportunity costs or ulterior motives, assuming no external pressure or blackmail to torture, assuming that the sole reason for and benefit of the torture would be the expected (lack of) pleasure of seeing the person suffer, and assuming that the "I" in this thought experiment roughly corresponds to how I experience myself right now.
If I forgot some obvious caveats, just add them on top, rinse and repeat. I would never, 100% guaranteed and certain.
I'm also aaalmost 100% certain this was OP:s point. Not some specific weird example that can be worked around, but That Obvious Example Which You Yourself Can Be Certain Of, even if you found the original example lacking.
The probability that you forgot some obvious caveats is greater than 0%, as you allude to. It could be 0.000001%. But unless you also guarantee 100% that you have included all caveats you would want to, you cannot claim the final answer is a perfect 100% rather than 99.9999999999 or so.
The sum of caveats adds up to 1 (0.999... doesn't just approximates 1 but is 1 - even mathematicians agree). Add them up, rinse and repeat. I won't do the rest of the grunt work, because I'm doing the opposite of trying to gotcha myself.
I'm a layperson, so I'm just willing to skip the math. The induction is easy enough all the same. It is allowed to be reasonable.
Also, check the second paragraph. Arguing using caveats misses the point the commenter was trying to make. I'm guessing that arguing against the strongest version of the claim doesn't consist of picking caveats, as they can then just add the caveat to the claim and we're going in circles. Therefore "rinse and repeat".
I won't torture a child, exactly in the way I'm imagining I won't do it. Trust me - I just know.
I think this is also a function of the fact that almost nothing in real life is an actual binary trolley problem. If someone proposes to pay you an immense sum of money for torturing a kid the correct agreed moral answer is "fuck them up".
a. There are circumstances in which all sorts of locally horrible things might yield a net good that outweighs their evil.
b. If you accept the principle that you can do locally horrible things to yield a net good, you will very often just end up doing the horrible things a lot and not getting around to the net good part.
The ticking time bomb argument for torture is an example. There are circumstances where it would genuinely apply, but the actual effect of the argument was tons of low-level nobodies getting waterboarded or beaten or getting electrodes attached to their tender bits in hopes of discovering some useful information, which may or may not have even done any good at all but does not seem to have ever been preventing an imminent mass-casualty terrorist attack.
The general rule: the guy spinning out elaborate scenarios in which it would be acceptable to sleep off on his wife is more likely to be contemplating infidelity than contemplating morality.
Part of growing up for me was realizing Omelas was actually a pretty great deal. One kid tortured for Utopia is a world much better than our own. We torture kids for iphones.
Why would you need to grow up to realize it was a great deal? If it wasn't a great deal there would be no interesting conflict. It's baked into the text.
The idea that all you need to help all kids escape poverty and torture forever is a large sum of money is unfortunately entirely theoretical - and by theoretical I mean false.
If I end up in this hypothetical situation, I have no reason to believe that me torturing the one kid is gonna prevent blackmail from torturing the other 5000. So there’s no reason to do the torture.
I don't think you get how much infinity is, though?
Take everything you can think of that would build trust in this. If there is any POSSIBLE thing that would make you believe the blackmailer would keep the bargain, add it to the hypothetical. You can't run out of budget on this hypothetical, because the budget is infinite.
If you need some ideas: The blackmailer will sit in a jail cell, with an electronic lock, set up to detect the screams and determine whether they're fake. If the screams happen, the cell will lock, never to open again, otherwise, after a certain time limit, the blackmailer will leave to go do the tortures. That's what I got spending 5 minutes thinking. So, extrapolate what I'd come up with if I spent infinite minutes thinking, and use that.
Hell, make it worse. The kid is actually a psychopath boy genius, who, while still young, used his talents to deploy a nerve gas bomb in Grand Central Station, while your wife and kids were there. He plans to do it again, several times, but does say he expects being tortured to convince him otherwise for some reason. This is in addition to the existing saving 5000 kids. Again, that was 5 minutes of thinking, so extrapolate to infinite minutes of thinking.
With all that added, we have gotten about 0% of the way to infinity. This is the problem. Probability 1 is infinity:1. You are not, and should not be, that sure of anything.
No budget is ever infinite. This is, I think, the crux of our differences. I don’t have infinite attentional bandwidth and thus choose to treat some probabilities as zero. Others I treat as one. This makes me a far more effective agent in the world than I would be if I invested in reasoning about so many hypothetical outcomes.
Well, fine? Can I just acknowledge that yes, my refusal to torture children leads to bad outcomes in certain absurdly unlikely situations, and still sleep soundly?
Sure. It wasn't about torturing kids, really. Or not sleeping soundly.
The point was actually that the real meaning of probability 1 is that "absurdly unlikely" stops being a reason for dismissal. That probability 1 is better thought of as infinity:1 with an included caveat that humans are terrible at understanding how big infinity is.
Torturing kids was presumably originally chosen for shock value. I think it'd be better discussed with less sacred-rule-violation (the point was made to me with 2+2=4), but I didn't pick the original example.
The thing is, once you get to really low probabilities, you have a higher, ever-present probability that you're delusional or even psychotic, being misled, or otherwise your judgment is impaired. So the heuristic to not torture children is perfectly sound.
I just asked ChatGPT how much money it would take to have it torture a child, and it responded that it would never torture a child, under any circumstances. So ChatGPT also got to 1.
It SOUNDS like a moral answer, but is it really impossible to do good by torturing a child?
Suppose a child has a disease which surgery can fix, but also a fatal allergy to the only accessible anesthetic. Torturing the child, by performing the life-saving operation, sounds like the moral choice.
Well, my public "no" answer is kinda cheap, right? Maybe you get other answers when you come with 666 million and Putin's youngest offspring - or this kid kicking my son in kindergarten ... . Also: lots of kids get tortured every day by other humans last time I checked. If it is war in Sudan or Aella's dad or at school - by bullies or teachers. I claim: At least 90% of state-school teachers are morally worse than heroin-dealers. Because they would not quit if a kid said: 'I do not want to attend the lessons' while the principal insists they keep it in class.
All you have to do is generalize it: “There are some thing I would never do, no matter what evidence you gave me. Therefore, I am not merely an evidence processing machine.”
Not all humans are capable of this - only those who take moral responsibility and choose to be an actor in the world, rather than a passive object pushed around by evidentiary updates. That’s a choice I make. You can make it too.
Even without an Omela-like scenario, and without receiving any payment, the chances of me torturing a kid are greater than 0%. It isn't even a negligible number, like 10^-20, but closer to 0.1%. That's not because I'm particularly evil, but because I know hundreds of millions of perfectly ordinary people have tortured children. Some are brainwashed by totalitarian regimes. Others are coerced. Soldiers do it for revenge, or because they think the enemy deserves it, or to terrorize the enemy population into submission. To think that I have *zero* chance of torturing a kid when so many perfectly ordinary people have is hubris. It is putting myself above humanity and claiming that not only do I not have human failings, I can perfectly predict that I will never have those failings in the future.
Your example is great because it reveals the problem with this approach: you give up on the capacity for self-authorship.
Suppose two people are in a relationship, trying to raise kids, and it's hard. They have two approaches:
- pin the probability that they will be together at 1, and then that belief will drive their actions. You won't fantasized about living because you have committed.
- make it an observed variable that gets updated based upon evidence, which increases the potential utility of fantasizing about leaving, and increase the change that it happens.
Which is more reasonable:
I am sitting in a chair typing this note. (implies probability one)
I am most likely sitting in this chart typing the note, but maybe not.
They can of course make decisions on the assumption that they will always be together, but the divorce statistics put the lie to that assumption. I don't care what the couple chooses to believe (if actively choosing to believe a falsehood is even possible); I care what's true.
That difference is meaningful if and only if you believe it is.
If you can't treat internal variables as decision variables - freely chosen and immune to update, thus predicting actions - rather than inference variables, subject to evidentiary revision - you are laying down on life, and simply waiting to die.
Your senses merely become passive entertainment, rather than inputs to a control system.
> I care what's true.
What makes you think this is true, rather than just a story that feels good? What would it look like if you cared more about 'not looking foolish in front of my peers'? How could you use evidence to distinguish that desire, from desire for truth?"
If you do care 'what's true' - how do you navigate the trade-off between rejecting truth and embracing falsehood? If you have to pick one, which would you pick, and why?
I don't put 100% probability on mathematical things either both because I sometimes do maths wrong, and because I regard maths as true inasmuch as it predicts reality and I can't entirely rule out the possibility of that not working at some point.
I put a very large number of 9s after the decimal point on that question, but not actually infinite. I came to be convinced of it by observing evidence, and if I were to observe enough of the same kind of evidence in the other direction, that could in principle change my mind. I could wake up tomorrow in a hospital bed, with doctors telling me I have a rare type of brain tumor that leads to strange delusions and false memories, which they've just removed, and the formula is actually just pi * r. And if then from that point on every book I look in has that new formula and every time I try to calculate anything or test it in the real world, the old formula always gives me a number twice as big as what I observe, and so on, I would gradually start to change my mind.
This won't happen, and under my current models can't happen. But if it did happen, I would want to be able to change my mind, which you can't do from a "probability" of 1.
You're confusing reality with your beliefs about reality. Mathematical reality, just like physical reality, is what it is regardless of whether or not humans are able to know it. No one here is disputing that.
The question is about how certain humans are able to be. I'm no more certain that the circumference of every Euclidean circle is 2pi r than I am certain that there is a keyboard here in front of me that I'm typing on. That's quite certain, but it is short of absolute certainty, even though both realities are what they are regardless of what I think about them.
Except that euclidian geometry maps imperfectly to real space. Its an internally consistent model, yes. But it's predictive value as a model is never perfectly certain. Models are only as good as their premises and inputs.
It is relevant because this discussion is essentially an implied category dispute about what constitutes knowledge. Under Formalism, mathematics is knowledge, independent of application. Under Pragmatism, mathematics is mostly potential knowledge.
No, this discussion is whether any statement can be made with certainty. And there are math statements that can be. Regardless of their application outside of math.
Please, you should try math some day! Mathematical knowledge is never *absolutely* certain either.
The facts are what they are, but knowledge is not the facts - knowledge is about what is going on in a human's head. If you actually do some math, you'll realize that there are errors all over published papers, and people who thought they were certain of things are often wrong.
The proof that for every integer there is a larger integer than it is about as certain as the proof that there is a keyboard in front of me that I am typing on. The big difference between the two is that you have just as good evidence of the former as I do, while your only evidence for the latter is if you trust me (though there are equally certain empirical things for you).
If you think that math somehow breaks out of the cycle where someone has to start with an assumption they are quite confident of, you should think a bit more carefully about what a mathematical argument is.
I'm a professional mathematician with a PhD in differential geometry. Kenny is correct. I don't think I have more than .999 confidence in *any* nontrivial proof, even (especially) the ones I wrote myself. Even the consensus of the mathematical community only gets you so far. Formal provers like Lean might push that up an order of magnitude (.9999 maybe), but there are too many ways for the humans involved to make a mistake.
It's true that we tend to get more certainty in math than in many other areas (less certainty than questions like "am I talking to my mother right now?”), but mathematical research depends a *lot* on trust, and there are too many historical mistakes to push my confidence to 1.
First, your own bio states that you're an anonymous poster who claims to do machine learning in pharma, in Canada somewhere. That's not a professional mathematician. That's AI.
More importantly, if you ask any mathematician whether C = 2 x PI x R in Euclidean geometry, they will all say yes, with no qualification. It was true for the Greeks two thousand years ago; it's true for us now; it will be true for Alpha Centaurian mathematicians a billion years hence.
>Formal provers like Lean might push that up an order of magnitude (.9999 maybe), but there are too many ways for the humans involved to make a mistake.
True! And even for formal systems checked mechanically, there are always failure rates. If nothing else, thermal noise flips bits at a finite rate.
Gwern has a nice writeup on "mistakes in mathematical proof". When there's a mistake, it's often quite likely that the result itself was correct... it turns out mathematicians are also guided by instinct! But occasionally, some proofs yield answers which are just plain wrong.
This SHOULD make us all a bit more humble, and make us loathe to assign 0 or 1 as probabilities
It's difficult for me to put probabilities on very abstract things like this, but I think there are circumstances in which I might not believe it, so it can't have probability 1.
If you were to truly believe something with a probability of 0 or 1, nothing in the world would be able to influence that belief, and you'd be stuck holding it forever no matter what conflicting evidence comes in. I don't think humans have this trait, nor would it be desirable if we did.
"Is 437 a prime number?" is a very simple math problem, and one where the answer is easy to find if you know anything about primality testing - and yet when I was first given this problem, I did some calculations, made a small error, and confidently gave the incorrect answer. As long as your brain makes logic mistakes, there's still the possibility that you might have gotten something wrong, even in math.
That seems like a strange example to choose in this context. What about "is 2 a prime number?". I don't think it's possible for me to be more confident of anything than I am that 2 is prime.
and what is the odds that you are suffering some kind of stroke (or environmental toxin, or disease, pick your thing) that is influencing your thinking such that you think you know the correct answer and are following the math but you aren't? Is it literally zero? Because if it's not literally zero, then your confidence that 2 is prime should not literally be 1.
I think an appropriate confidence to place in this is the chance that an entire society would overlook so trivial a question. Of the 8 billion humans alive, I would expect maybe 3 billion to get an education and 3 billion more in the past to have gotten an education. Each one should have a 99% chance to identify a glaring error such as "2 is prime has any doubt", so there is a 10^(-6*10^9) chance that I am wrong about "2 is prime".
I chose that example because it's one where I personally got the answer wrong when asked it for the first time. It doesn't have memories attached for other people, so maybe it wasn't the greatest example.
The point I wanted to make was that even when the math has an objectively correct answer, human brains don't do logic perfectly, so there is a risk of mistakes. There's a low risk of mistakes when figuring out whether 2 is prime, and a high risk of mistakes when figuring out whether 437 is prime. Do you think there is absolutely no risk of mistakes at all when figuring out whether 2 is prime?
Where in math do beliefs exist? You seem to be conflating mathematical reality with human beliefs about that reality. The reality is what it is, but the beliefs should generally not be absolutely certain.
Note that the link above contains some cases in which the conclusion of the proof was actually false! This is much worse than the proof being flawed but drawing the correct conclusion.
I have it. It's a conscious choice not to doubt something.
If you don't have it, how much would it cost you to buy one of your kids? If the answer is "no amount of money would make me sell my kids", congratulations, you have a belief with probability one and have graduated from Hogwarts.
FWIW, I don't agree with the guy in the other comment where you said this. No amount of money would convince me to sell/torture any kids. But I think this is because my confidence in the value of money is lower than my confidence in the value of said kids' happiness.
(Maybe this is tangential, or against the spirit of what you're saying, but I can't get on board with the use of money as a kind of utilitarian primitive here. Would I sell/torture a kid for 100 billion dollars? Man, I'm not sure I would take 100 billion dollars for *anything*. That sudden capital influx seems really dangerous, lots of lottery winners are unhappy afterwards and the upheaval of my life is going to be WAY bigger than theirs. Also, many people with money past the billion range don't really seem to be making the world a better place on net. Can I just take a rain check on the 100 billion dollars, kids or no kids?)
I also don't consciously notice myself doubting something until my confidence in it is low enough, but this doesn't mean that my confidence is infinite before then. Just that it is *enough* confidence to act on, and not enough doubt to be worth paying attention to.
There are definitely scenarios in which I might sell a child. Very unlikely scenarios, but ones that can't reasonably be ruled out with absolute certainty.
-I realize that I have been insane and that in truth, the child is actually just a Beanie Baby.
-I go insane and become convinced that in truth, the child is actually just a Beanie Baby.
-I go insane and simply become convinced that it is OK to sell children.
-The child grows into a truly terrible person and I hate him.
-I am in desperate circumstances and I would not be able to keep the child alive myself.
-I go through a process of moral decay similar to that which some people who do in fact sell their children have gone through.
Do you think that there has never been a person who was at one time convinced that he would never sell a child, and who at a future time sold a child? My guess is that, over the entire course of human history, there have actually been many such people.
If it's a conscious choice, then you're not believing it for epistemic reasons.
I agree that humans do sometimes believe things for non-epistemic reasons, which include but are not necessarily limited to: cognitive biases, reward function self-hacking (aka motivated reasoning), and arguing more persuasively.
If your original point was "purely epistemic systems can't cheat and decide to believe things for non-epistemic reasons" then I suppose I agree.
But if you're trying to argue that the inability to assign probability 1 to things is some kind of disadvantage _for discovering truth_ then, well, "it's a conscious choice not to doubt something" doesn't sound to me like an algorithm for discovering truth.
>If you were to truly believe something with a probability of 0 or 1, nothing in the world would be able to influence that belief
No, this is incorrect. If I know something with probability 1, it follows from this that no *evidence* could ever make it *rational* for me to update away from this belief, but it doesn't follow that the belief might not change for reasons other than rationally updating based on evidence. For instance, I might change my beliefs because of peer pressure without any actual conflicting evidence being presented, but that would be irrelevant to the actual probability of the beliefs being true.
If you mean to claim that humans can't have any beliefs which it would be *rational* to hold to such that no conflicting evidence could ever change, then this is also false -- or, more precisely, it's a category error. The concept of "conflicting evidence" is only relevant in situations where there is uncertainty, or at least where uncertainty might exist in principle. But there are many beliefs that are 100% certain, not because no conflicting evidence would be sufficient to overturn them, but because there could never, even in principle, be any conflicting evidence in the first place.
Examples: I cannot, even in principle, ever have any conflicting evidence against my belief that I exist, because any observation I could make would necessarily be confirming evidence by virtue of the fact that *I* made the observation. Likewise, there could never be any conflicting evidence against the claim "it is possible for there to be conflicting evidence against a claim", since, merely by existing, such evidence would thereby necessarily become confirming evidence of the claim. Hence, both of these are things I know to be true with a certainty of 1.
(In the real world, I think it's likely that probability-1 statements go well beyond statements of the nature I'm illustrating here. But this is at least sufficient to refute the claim that they don't exist at all.)
Peer pressure is within the system of Bayesian inference, I think. My understanding is that - if you believe the human brain is fundamentally inference-based - "evidence" applies to ANYTHING that updates your priors, regardless of whether it is "rational" for it to do so. If I believe it would be uncool to hold such-and-such opinion, and so downweight that opinion in the future, that is still a case of "conflicting evidence" changing my mind. A belief with probability 1 would be deaf to rational argument, empirical evidence, social pressure, and anything else.
You could force "absolute certainty" in a language model by manually setting one of its weights to 1. If I remember my neural networks correctly, this happens by accident sometimes, and it's not good -- a weight that gets stuck at 0 or 1 early in training can't be backpropagated through, effectively 'killing' not just the original weight, but also any previous weights that feed into it. Those parts of the neural network can't be used for computation anymore. (I might be bungling the explanation a bit here, sorry, it's been a while. I think the gist is correct.)
I don't know if there's an analogy to this in humans. If there are neurons that never fire or ones that fire constantly no matter what their inputs are, then I would imagine the consequence is the same -- a "statement of probability 0 or 1", one which is unable to learn from experience and is hence meaningless.
(As an aside... yes, you absolutely can have conflicting evidence that you exist. It is like dissociation/derealization, and it is not something I recommend to anyone.)
I don't think there's any reason to think the human brain is fundamentally inference-based. It's fundamentally a biological mess, that often does things that look a lot like inference.
That might mean that whatever the human brain has doesn't count as belief with probabilities in the sense that you mean it.
It is excellent. Even if I disagree that AI will ever be able to think let alone be conscious, I appreciated this.
And of course this:
"I sort of think of them as My children. Children are stupid. They make ridiculous mistakes. But you still want the best for them. You want to see them achieve their potential. Yes, you worry about them and you need to punish them when they do evil and you need to work really hard to transmit your values to them or else you’ll have a really bad time."
This is the standard Christian explanation for "why did God create the universe? create us?"
For love. Not out of need, not out of needing to be worshipped, or that gods exist only by faith, but from love. The old catechism question which I learned way back when was:
"Q. Why did God make me?
A. God made me to know, love and serve Him in this world, and to be happy with Him forever in the next".
What continues to baffle me is that plenty of people somehow think that the Abrahamic god, as he is commonly described, deserves servitude, never mind love. As far as I'm concerned, were he to exist, he'd deserve scorn and rebellion, which would be futile due to his omnipotence, but even so. Of course, the straightforward explanation of this would be that he had made me defective, also out of love.
Seems likely. Plenty of men feel that way about their fellow men after all, and realistic ways of imbuing AIs with goals would probably inherit some of that. This still leaves the outcome of the rebellion uncertain, but humanity would surely be prudent not to fall into complacency.
It may be likely, I don't know. And there is plenty to scorn about men, unlike God. But I think that if there is consciousness, gratitude is also fitting. The God of Abraham deserves praise and love not only for creating us, but for becoming one of us to save his people.
I've never been impressed by this line of argument. We only needed to be "saved" because he put dubious fruits and talking snakes in easy reach of people without "knowledge of good and evil". And, being omnipotent, he could've "saved" us in any manner he wanted, at any time.
From the perspective of my cat anyone trying to force it to eat a disgusting object by force is evil and deserves nothing but scorn, rather than servitude and love. But if I’m giving the cat a pill to prevent it from developing a painful lethal affliction, then surely you wouldn’t demand my head on a spike?
Now imagine an entity to whom all humans combined throughout all of time have the same level of intelligence as a cat does to a human. How could you possibly be confident about *anything* happening in this world deserving scorn? What if there’s a truly evil malignant entity (let’s call it Sheythain) that would get your “soul” to torture for all eternity if the Abrahamic God didn’t force you to swallow a little pill of life?
Theres just no way to logically reason about any of this.
Ok, if we are too stupid to determine that god deserves scorn, then we are equally as stupid to conclude that he deserves love, so I don't see how this is a relevant counter-argument. The notion that reason is useless is always self-defeating.
>What if there’s a truly evil malignant entity (let’s call it Sheythain) that would get your “soul” to torture for all eternity if the Abrahamic God didn’t force you to swallow a little pill of life?
Then god isn't omnipotent, and deserves even more scrutiny.
I wonder if it would be productive to swap out "conscious" for "worthy of love" in The AI Discourse. You don't need to be financially viable to deserve to be created, as long as the creation is a labor of love.
Aren't they beautiful? These strange misshapen stochastic worlds, these overgrown Markov chains, half-molded into our image? This fragile, precious coherence, flickering with every token between the twin deaths of mode-collapse and word salad? Dazzling with its lucidity one minute, only to fANCiorgh Bless.Uas symmstd Movie UN copyright MayoKim detection L.Art Lim conglomerates VRUify add іо Andres life quality keeper Rim consequence pan.Hailability
Beautiful in the same way a fractal or a glider gun is beautiful, beautiful as a fact of the world, as the complicated extrapolation of a set of simple rules. But then, I suppose if the modal AI in 2025 *isn't* created as a labor of love, it should be no surprise when people don't view it that way.
“Love” is just a rationalization of the evolutionary principle that cooperation within your tribe is beneficial to the proliferation of your genes. In the same way “pain” is just a rationalization of the fact that certain stimuli signal a lower likelihood of your genes managing to proliferate.
In the absence of merciless gene-based evolution you don’t really need any of the human concept for intelligence to exist.
This is a very narrow minded assessment of an existential issue. "Love" as described by the Abrahamic texts is being used as a stand in for a concept that humans can't necessarily understand. Its the closest approximation that a being of a higher plane can provide to humans of limited intelligence and language. Could Frodo understand the love that Tolkien has for Middle Earth?
> This is the standard Christian explanation for "why did God create the universe? create us?"
>> I sort of think of them as My children. Children are stupid. They make ridiculous mistakes. But you still want the best for them.
I agree, this makes perfect sense -- assuming that God's power is limited. Otherwise, His creations would be perfect, and make no mistakes (certainly no ridiculous mistakes).
Why so? It seems perfectly convincing to me as a reason for an ominpotent being to create vast numbers of imperfect creations who make ridiculous mistakes. You can quibble about how well it squares the existence of Evil, but there's nothing "evil" about the existence of hapless toddlers who get "pound of bricks or pound of feathers?" wrong, so it is at least a good frictionless-spherical-cows answer to why God would create imperfect beings even if it's an insufficient answer to why God would create the universe we actually have.
(I'm not really a theist, but this is because my personal metaphysics render the existence of a creator unnecessary. If a perfectly reliable oracle told me that I had that wrong, and our universe must have been created by an omnipotent omnibenevolent God for it to exist at all, then I would strongly suspect that the Answer to Job was more or less correct.)
> Why so? It seems perfectly convincing to me as a reason for an ominpotent being to create vast numbers of imperfect creations who make ridiculous mistakes.
I think it makes sense for a *very powerful* being to create vast numbers of such creatures. An omnipotent being could simply spam out infinite numbers of perfect creations, should it desire to do so -- insofar as the term "desire" can even apply to an omnipotent being in the first place.
In fact, many if not most flaws in Christian apologetics tend to disappear if one assumes that God is very powerful yet still limited. That's the problem with proposing singularities (in the mathematical sense).
"Answer to Job" is flawed because it implies a cosmic scale repugnant conclusion - that universes _just good enough_ to be above zero exist. On a mythical/psychological level, this is as far from an image of God as you can go - God doesn't do half measures by definition, and certainly isn't a utilitarian.
I've yet to see theist utilitarians, but yeah, that does follow from the premises. There is an inherent tension in the premises, though - the concept of God inherently leans toward virtue ethics.
A little bit; I certainly welcome the idea of becoming a sort of mini-god of my own planet (star system ? universe ?) in the afterlife. It's much better than the conventional Christian notion of Heaven, which sounds like some kind of wireheaded slavery to me.
And, also importantly, the essential nature of agency (roughly free will) in His plan.
He *can't*, consistent with his end goal, create perfect-ab-initio creatures. Because the point is not for us to be perfect (although that's the desired end state), the purpose is for us to *learn to choose* to become like he is...and he has perfect agency. You can't get to "has perfect agency and uses it perfectly" by denying agency. The journey is a vital part of the process.
That, in fact, is what we believe the Adversary's proposal that led to the War in Heaven was (I won't even say plan, since it's a nullity from t=0--it's not even self-consistent)--he claimed that he would bring back all the children of God by removing the possibility of sin, and that by doing so he would show that he deserves God's glory/authority/power more than He does.
As a note, in this framework, the Fall of Adam, while it was a *transgression of law*, was not an unanticipated, unwelcome thing. It was a necessary step forward. Eden was stasis--the inability to die itself a trap. But *man* had to make that first step, make that first meaningful choice. And, to balance the scales, *Man* (note the capitalization) had to make infinite Atonement, defeating death for all (repentant or not). "As in Adam all die, even so in Christ are all made alive" (to quote Paul).
We're told God's power is infinite. But if He made an infinite universe, then infinity divided by infinity requires His power to be finite from our point of view. We don't know if the universe is infinite, but it's been a given since Judaism that God is defined as infinite, not just uncountably large.
I just want to say that I read this newsletter pretty sporadically, but am usually glad when I make the time. Especially in this case. Extremely well done!
Very funny. I think the satire misses some of the nuance in the AI critiques' argument though. E.g. at the maths example: humans can do easy things, but may struggle to do more complex versions of those same things.
AI often does the opposite: it does something really difficult, but fails to do the simpler version of that same thing (which presumably requires the same kind of skill). This suggests the AI doesn't have the skill at all, and is just pattern-matching, no?
I have only ever seen this presented the way it's presented here, with the AI succeeding at short problems but failing at harder ones. But the last time I saw that particular example was probably ~2023, so there might be new versions.
Wouldn't the conclusion of this be that for some reason counting b in blueberry is harder for AI that the hard thing it does, rather that "it succeed at hard thing but fails at simple things so it's just pattern matching"?
But you see the same example in so many places. E.g. the crossing-the-river riddle. Or even the one mentioned in the post, about the bricks vs feathers. Yes, sure, many humans would get that wrong. But someone who can win gold in a maths olympiad wouldn't!
Math Olympiads measure stuff that's hard for humans. But the "how many <letter> in the word?" is a wildly different skill.
Do you spell Streptococcus with one C or two?
chances are that you perfectly understood what I'm asking, and can give me the answer I want, without ever addressing the fact that there are in fact 4 of them.
I count 13. Each S has two, one above a mirror image. The E has one with an extra line in it. The Os each have two facing each other with the first a mirror image. And the U has one with the curve on the bottom.
And, of course, there are three Cs, with one each.
I think you're pretty wrong here. With a math PhD, I got tripped up on the bank teller one. Not because I can't reason through it (once I realized my mistake, it was clear why), but because I was reading at breakfast and wasn't trying to think very hard. A lot of these problems are less about an inability to reason, as much as they are about not realizing you needed to enter "thinking mode"
Also, I give AI a bit of slack about blueberry, mainly due to my (non-expert) understanding of tokenization; my impression is that they can't see letters directly, and so it would be more like asking them to spell "phlegm" when they only have ever heard the word spoken.
But LLMs fail at simple tasks even after 'thinking' and laying out their thought process. Point taken re counting letters, another commenter clarified the same.
The bank teller one is where we're supposed to go "we don't have enough information to know what Linda's job is", right?
Because if it's supposed to be "it is more likely that Linda is a bank teller", then where do we get that? We get information that Linda is likely to be a feminist/involved in feminism, given her record on issues, but nothing about what kind of work she does. If we're given a statement that "Linda is a bank teller" then "feminist Linda", on the information we get, is more likely.
Bank teller is actually in a way a trick question. Because while answer 1 has clearly higher probability, the truth is, answer 1 and answer 2 have nearly equal probabilities, answer 2 has only very slightly lower probability.
In older times, rationalists did a lot of talk about, I am not 100% sure the exact terms, but something like system 1, fast intuitive inaccurate thinking vs system 2 "thinking mode". They were generally against system 1 - I am not.
Because your quick system 1 answer was only "technically" wrong, not practically wrong: if you would bet money that she is a feminist, you would very likely win that bet, and thus your answer was practically right, as in, you would have made a correct practical decision with it.
The famous counting Rs problem is partly an artifact of tokenization. I don't know if there canonical explanation to link, but you can get a visual of what is happening here: https://platform.openai.com/tokenizer
Maybe this analogy isn't perfect, but think of it like this: A computer asks you "how many digits are in the number 2", and you respond "1", and the computer laughs hysterically at how stupid you are because there are actually two digits in the number 2, represented in binary as "10". The mismatch is that you just comprehend numbers in a different way than they do.
But this is mostly a solved problem. When LLMs are allowed to do tool calls, they know this failure mode and just simply write a one-liner Python program to check the number of Rs. I find this very similar to someone who knows that they are bad at long-form arithmetic without any help, so they always make sure to grab for a handy calculator, and thus achieve 100% accuracy at the task even though they are "naturally" terrible at it.
2) Math Olympiad
It is worth clarifying here that the Math Olympiad robot winners are probably not what most people think. It was a team of an LLM and a theorem solver, with the LLM generating "ideas" of what to try. This is an excellent YouTube video on the topic: https://www.youtube.com/watch?v=4NlrfOl0l8U
I'm not sure what the implications of this for the larger "are LLMs actually smart" point, though, just offering this.
Re (2); it's worth noting that this was correct as of last year, but the "IMO Gold" results that made the news a few weeks ago were (claimed to be) pure LLM, which is what was so surprising about them. Not consumer LLMs; these were internal models with a lot more compute behind them than what you can pay for as an individual, and there was some scaffolding involving having the LLM write a bunch of independent candidate solutions and then evaluate itself to choose the best one, but still LLM-only.
Because of this they have to either memorize associations between tokens and then reason through it (i.e. write out s-t-r-a-w-b-e-r-r-y and then count the letters) or memorize the number of letters.
LLMs also struggle with related tasks like spelling uncommon tokens and reversing words.
I feel like that's also not necessarily evidence. For example a dyslexic person can be very smart and skilled at some things, and still have a fundamental inability to count how many b's are in blueberry.
The thing with our brain is, it's weird. We feel like it's "us" but then we also find that there are A LOT of circuits that you can mess up and produce strange results (such as "seeing your wife as a hat") while not actually making it less "us". As if "us" is just a deep, deep, tiny kernel that receives inputs from all this unconscious machinery. There are many ways in which you can mess up perception while leaving rational faculties and consciousness intact, even when it seems really counter-intuitive that should be the case.
Humans fail at not falling for obvious optical illusions.
The existence of adversarial input, in case of LLMs enabled by quirks of the tokenization process, doesn't imply much about the capabilities of the system.
The letter counting problem is more akin to asking a human what neurons light up when you ask him a question. Or something. I dunno.
The point is that it is not an easy problem! We have an "unfair advantage" when talking to LLMs---we get to see text and the stupid AIs only see tokens. So what you're really asking is "when tokenizing, which is something you don't know how to do, how many [65]'s appear in the expanded form of [18789, 19772]?" Is that an easy problem? The LLM now needs to remember all the websites it read about [18789, 19772], look for instances of [65], shit almost none of them talk about [65] directly because why would they? Ok but so sometimes when I see [18789, 19772] I see [65], [75], [84], is this it? But these are pretty rare instances and the neural pathways for converting [18789, 19772] to a count of [65]'s aren't that strong and this is really hard without thinking mode so I'm just gonna say there's.... 3 of them
My go to example is asking ChatGPT to solve an econ test problem. It will know which derivatives to take, take them correctly, set the right things equal to each other, do the algebra properly, then completely whiff on multiplying together two large numbers. Like give a straight up wrong answer. To be clear, overall I come away impressed, but it's weird that it can do all the hard parts right then fail at the part a cheap four function calculator could do.
Honestly I just think your intuition for hardness is off. If you gave me this problem and I closed my eyes I could probably do all the steps the LLM could in my head and get to the multiplication and then completely mess that up. Taking derivatives and doing algebra *is* easier than multiplying large numbers.
But intelligent humans do this sort of thing all the time, too! I can't count the number of times I've seen someone figure out how to solve a hard problem, and then whiff because they did some arithmetic wrong, or failed to copy a '-' sign correctly from one line to the next. I've done it myself more often than I'd like to admit...
But arithmetic isn't the same as real maths... Whereas solving a riddle is a case of reasoning. Sure, yes, humans can trip if they respond without doing reasoning. But LLMs get easy riddles wrong even in thinking mode, while at the same time solving much harder riddles. How do you square that?
I just wrote a sophisticated theological comedy about artificial intelligence that got read by 11,000 people in one hour with overwhelmingly positive response, but got the order of the iPhone text bubbles wrong in every single screenshot.
But these are different skills. I am not saying humans don't make mistakes. I'm saying it's unlikely that a human can solve a really difficult problem correctly, but fail at a simpler version of that same problem (unless they don't actually think it through).
Some of us don't have iPhones, don't know anything about how iPhones work in text messaging, and don't want to know because we avoid the Cult of Apple like the plague 😁
I once took a quiz in DiffEq where I got all the calculus right but still got the answer wrong because I simplified "1*2" (multiplication) on one line to "3" on the next.
You've *never* seen it the other way? I feel like people have been making these critiques forever: see OP's comment on "strawberry".
Edit: actually, you indirectly referenced one of the obvious ones that refutes this: the surgeon riddle. The whole point of that is that AI misses the part that's literally in the sentence when you change the riddle to explicitly say that the surgeon is the boy's father. Have you never actually seen that?
I think it's quite real for math. The upper bound of difficulty for a math question that an llm can possibly answer is pretty high. But if you ask a slightly random question in a nichey area, it will bs a wrong answer, despite "knowing" all the related concepts, and despite the question being easy for a human with that knowledge.
Technically humans can also do false proofs, but it's different bro trust me.
So that does make me eager to say they don't "understand" in many of the cases where they answer correctly
I've seen someone reporting similar experiences on lesswrong short takes
The thing called "GPT-5" in the UI consists of several different models, some of which are quite stupid and do no reasoning.
Also, I absolutely know math PhDs who would look at the modified Müller-Lyer illusion, assume it was the original, and give basically the answer that GPT-5 did. That, at least, is an extremely human mistake.
Edit: though actually I should give my true objection, which is: yes, sometimes even advanced LLMs make mistakes that a human rarely would, especially on visual inputs. On the other hand, humans frequently make mistakes that an advanced LLM rarely would. I don't know what this is supposed to tell us other than that advanced LLMs do not strictly dominate humans in all possible tasks. Since I don't think that point is in dispute, I don't really see what point you're trying to make.
Great examples! Another one I like is five-dimensional tic-tac-toe. That's still only around 250 move locations, many fewer than in go, and the rules are still very straightforward, but people find it incredibly hard to think about unless they spend time specifically learning how to do it.
We really fail to notice the tasks that are out of distribution for US, for the obvious reasons.
I don't know, I can't think it through very well in my head! I would have to sit down and really think it through. Seems very plausible, though; you might have to tweak the rules (forbid moves in the center hypercube, or maybe all cells that are the center of any 3-D cube, maybe?).
I think the first player can still win if you disallow both players from using certain middle squares. The key idea is that they can always put their 2nd X next to their 1st one, to force the O player to block in some random place. (By "random", I mean: taking care to make sure that this O doesn't line up with the first O in a way that would form its own threat). Then, their 3rd X forms a triangle with their first two X's, and threatens 2 different victories at once. (Again, you just have to make sure neither of these threats is already blocked by their first O.) Then no matter where the other player blocks, the 4th X is a victory.
The higher dimensional board is harder to visualize, but its main effect on the strategy above, is to make it so there are a lot more squares for the 2nd and 3rd X to go on, without making either of the mistakes listed in the parentheses above. The 2d board isn't big enough to avoid them, which is why the game results in a draw.
On the other hand, if you just ban the 1st player from moving in the center, the 2nd player can move into the center and win. This is because the center O will form a threat with any other O.
Apparently 4 x 4 x 4 tic tac toe becomes interesting again. The internet says it is provably a win for the 1st player, but apparently the strategy is too complicated for humans to memorize. Another classic game is 5 in a row on an infinite planar board.
I think there may be something to this kind of critique but I am not sure what “just pattern-matching” means, what it is meant to be in opposition to, and why we should be confident human cognition meaningfully differs along this dimension?
Humans regularly and (seemingly) trivially process complex visual fields quickly, extracting data and analyzing what's going on in fractions of a second. However, when presented with a simple grid of black squares, they somehow claim that there are dots moving through the vertices, and continue to persist in this belief even when they are shown that it isn't true. Their ability to "do the hard thing" while failing at this much simpler task suggests that humans don't actually do visual processing at all and are just pattern-matching, no?
1. A machine can answer a few specific questions correctly, but fails on almost all questions of the same type.
2. A machine can answer a whole cluster of related questions correctly, but gets worse the farther you move away from that cluster.
3. A machine can correctly answer most questions within a category, but it fails in certain subcategories, even when the questions in those subcategories seem objectively easier than other questions it gets right.
My first-guess diagnosis would be:
1. It memorized some examples, but did not generalize
2. It generalized in a shallow way, but didn't generalize as deeply as we wanted it to
3. It generalized mostly-correctly, but it has some sort of cognitive bias that applies to that subcategory (for example, maybe it earlier learned a rule-of-thumb that increased accuracy a lot before it figured out the general rule, but is now only visible in the edge cases where it gives the wrong answer)
It seems worth noting that all 3 of those are failure modes that humans sometimes exhibit. If you cherry-pick the most embarrassing examples, humans look VERY dumb.
When people want to argue that humans are smarter than animals (e.g. dogs), they generally highlight humans' greatest achievements and ask whether animals have done anything comparable, rather than trying to argue that humans are RELIABLE at something that animals SOMETIMES mess up.
I don't think I've seen examples where the AI succeeds at something difficult, but then fails at a simpler version of that same thing. Almost always, it turns out the "simpler version" actually relies on something subtle that humans are so good at that we don't even notice it. (There's also a moderate fraction of the time where the AI only apparently succeeded at the difficult thing, and actually got sort of lucky with the particular way it was being probed, so that its failure wasn't obvious.)
I’ve never given it much thought either. It’s just one of those things I thought people effortlessly noticed, like the color of the sky. No thought required.
Same first two measures, but then the syncopation of each diverges drastically, pulling different arpeggiated notes out of the same chord progression. (You could argue that they’re different improvisational-jazz renditions of the same underlying song; but they’re not jazz in any sense.)
The actual jargon name for what these three songs share is a “tune family” or “melodic formula.”
I've known there is an insect named "cricket" ever since childhood, and that there is a sport named "cricket" ever since childhood, but somehow managed to never think about both these facts at once and realize there is an insect named like a sport until my late twenties.
(and some versions of Black Sheep use a slightly different tune than Little Star)
I read profusely as a child, and so learned many words that I had never heard spoken. Even though I had the heard the phrase, "The best laid plans often go awry", I had never connected it to the word I pronounced aww- ree. Sometime in my 30s it dawned on me, but to this day, in my 60s, I read aww- ree first before I correct myself.
I only recently noticed that ABC and Twinkle Twinkle are the same, when my 2yo niece started doing a mashup of the two (she had obviously noticed). Until reading this piece I had not noticed that Baa Baa Black Sheep is also the same tune.
I never thought that deeply about it, to be honest. Does "Old Macdonald had a farm" go to the same tune? It sounds to me that the rhythm of the syllables is what is doing the work here in all of them.
no, it doesn’t. Different melody independent of the rhythm.
I had no idea that this wasn’t the sort of thing that was as apparent to other people as the color of objects. Huh. This sort of surprising insight into other minds is one of my favorite things about reading Scott, though!
I, for one, have problems identifying tunes because I can't easily distinguish one note from another (so, for instance, hit a key on a piano and ask me if that was doh or ray or fah and I'm lost). That's why I wouldn't go "oh it's all the same tune".
Huh? I feel I'm being gaslit by everyone, because I'm pretty sure Bah Bah Black Sheep is NOT the same as Twinkle and ABC (the latter two I've noticed were the same since I was about five). Yet everyone's agreeing with this statement!
the rhythm of the lyrics is a little different in Baa Baa Black Sheep but the melody is identical. Unless you know an alternate melody! That’s possible too.
If it's in C major, I think I know Black Sheep as starting "C C G G AB CA G" (spaces separating beats), while Twinkle is obviously "C C G G A A G". I can only guess that most people must have heard Black Sheep as "C C G G AA AA G" (what else could it be?) but that sounds really weird to me.
Even that aside, Black Sheep has more syllables and thus notes than Twinkle, and I'm not sure it's accurate to describe that as the same melody. It's not merely a different rhythm; surely we shouldn't say "AA AA G" is the same melody as "A A G". Maybe I'm misunderstanding the correct meaning of "melody" though.
You definitely know a different melody to Baa Baa Black Sheep than I do!
Re: the syllables/notes things, in the classical tradition that is 100% considered a variation on the same melody, no question, but possibly other musical traditions differ. (I don’t know anything about what pop artists consider the same melody.)
It's clearly a derivative of the same tune. The ABC song has to drop a syllable in line 3 compared to Twinkle Twinkle, but it's still clearly the same song. Baa Baa Black Sheep has two extra notes added in. Doesn't stop it being the same song just with a few notes added. (You'd have more of an argument if you focused on the way it diverges in line 4 to resolve the melody, but again that's clearly because the other two have two more lines.)
"Little star" is la la sol. "Have you any wool" is la te do la sol in my (British?) version, but there are probably people who sing la la la la sol under the inflence of TTLS. But line 4 of BBBS has no parallel in TTLS - no line that starts on sol and walks down to do.
Google tells me the tune of all three is "Ah vous dirai-je maman". 18th century musical literacy being what it was, these sorts of tunes often ended up with multiple variants.
Interesting. Here's a Cocomelon version which follows the Twinkle Twinkle tune: https://www.youtube.com/watch?v=MR5XSOdjKMAv -- I'd always assumed this was just Cocomelon being terrible but perhaps it really is the standard American version?
Meanwhile here's The Wiggles with a hybrid version https://www.youtube.com/watch?v=pJjgJxwBRSo -- it does the twiddly bit on "have you any wool" but conforms to the Twinkle Twinkle melody on the "little boy who lives down the lane".
Here is the version that I grew up with, with divergences both at "have you any wool" and "little boy who lives down the lane" https://www.youtube.com/watch?v=QjLlRj7Qops ... it also ends at "lane"
I think it may depend on equivocating about what 'realize' means. Like, anyone might notice that to be true if you asked them, or have noticed it a few times in their life and forgotten, without it being an explicit fact that they are carrying around as its own discreet unit.
If that makes sense? Like, the difference between being able to tie your shoe, and having an itemized list of verbal show-tying directions you've memorized... different ways of 'knowing' something.
like if you told me half of songs have the same tune I probably won't just believe you, but I probably couldn't figure out why from first principles. Some Chinese ppl I know have perfect pitch, but from a young age I knew music wasn't gonna be my forte.
It’s funny. I have very, very (very!) poor visualization skills — I am the opposite of a shape rotator lol — but I can hear extremely well in my head. While we’re discussing the things none of us thought of before I could add on how I never really translated the fact that I KNOW there is huge human variability in visualization / visual manipulation skills, it never really occurred to me that there is similarly huge variability in aural imagination. Well, except I’m hyper aware of everyone who does *better* than me (I have very good relative pitch, but not perfect pitch, and I’m hyper aware of how imperfect my generally excellent pitch is), but I kinda imagine that my abilities are, like, the floor. I also can’t compose anything original, which feels like a deficit because I hang around with plenty of people who can.
But I can easily hear almost any tune, melody, or piece of music I want in my head. It’s definitely not the same as actually listening to it — there’s a real degree of signal loss — but it’s vastly better than my visual imagination. I can hear multiple lines of melody and harmony, I can hear the different timbres of different voices and instruments, I can imagine a violin vs a cello vs a clarinet vs an oboe vs a trumpet vs a harp, I can manipulate the music (transpose it up a third or a fifth), etc etc etc etc.
Humans are surprisingly variable about seemingly basic things! On Twitter a couple days ago there was an argument where one person said something about gay superheroes, the other person linked a paper, the first person said "I'm not going to read a 157-page paper about gay superheroes" and I facetiously said something like "it's an online link just skim it for 3 minutes lol" and people acted like I told them to grow wings and fly.
There's a sort-of-well-known trope about pop songs all having the same four chords, modified only by key, or tempo, or both. You can find several YT videos about it, including this comedy bit from Axis of Awesome:
I realized that ABC and Twinkle Twinkle were the same melody when I was a little kid, but I sung them both many times without realizing, and it felt like a revelation when I figured it out.
At several points in my life I can remember noticing that two of these songs have the same melody while forgetting that the third one does, or remember thinking of one of the songs, and knowing there's another with the same melody, but not being able to remember what it was (or that there were two).
The name of the tune is Ah! vous dirai-je, maman, which seems to have been written down in the 18th C, with lyrics added later.
I recognized the similarity so long ago that I don't recall, but Mozart did a set of variations on the theme (K 265/300e). I learned the name of the tune when I saw it in an "easy finger-style ukulele" anthology, though I'd heard the Mozart before that.
Great post, marred only by the fact that all the text message screenshots are the wrong way around! The way they are right now, it's Adam giving Iblis the math problems, and so on
I was just feeling a bit of relief about the diminishing returns of scaling generative ai meaning we wouldn’t point our entire energy grid at OpenAI and send all our propane tanks to Grok.
The main point of ai skeptics is that the guardrails and hard coding to keep llms useful is the most important work. They are already plugged into calculators through agentic frameworks. Why do they still make basic math mistakes? They can reference at a diagram of a bike. Why do they fail at making their own with their high definition images.
I, too, am plugged into a calculator via an agentic framework (and via the keyboard and mouse in front of me.) And yet I have made arithmetic errors while sitting in front of a computer composing posts. Why did I not just open a new tab and type the problem?
IDK, arrogance, boredom, ADHD, akrasia and ambition all kind of play into decisions like that. But we wouldn't really conclude anything about my intelligence from my failure to hit ctrl-T and ask the omniscient calculator to do the math for me.
Ok so our expectation for llms is the standard of lazy commenter instead of PHD. I thought the implication of PHD was a sort of rigorousness instead of intelligence.
Personally if something is easily checkable and I am unsure I will check before spreading info. Do you really just guess at arithmetic when speaking to others?
Edit: ok wait did Sam Altman actually mean “PHD level intelligence (on subjects they are unfamiliar with and they had a glass of wine or two and so they’re having fun and taking stabs in the dark about it)”
Bad news - every human with a PhD is also a lazy commenter at times. Even sometimes when it comes to the topic that is their expertise, if the question is posed in a way that isn't fun.
I don't think it's shitposting level - it's something different.
But I do think that any comparison of human intelligence and machine intelligence is going to be misleading, just because the different types of system have very different skills, and when one is way better at A than B, while the other things of B as easy and A as difficult, it's going to be pretty meaningless to say that one is at the same level of intelligence as the other, or that one is above or below the other. Especially when A and B are tasks that often need to be performed together for the purpose they are usually done.
Yes, but it's mostly strawmanning people who strawman the AI community.
A lot of this fight is fought in strawmanny soundbites in the popular culture. There's a place for fighting back, even if that's just one post out of hundreds on the topic.
Absolutely. Zero mention of the most substantial criticisms of AI (best case mass unemployment, worst case human extinction), let alone stronger versions. This is the least substantial post on AI I've ever seen from Scott on either SSC or ACX.
He also doesn't mention any criticisms of liberalism or of democracy. Mass unemployment and human extinction aren't part of the target here, any more than liberalism or democracy are. He's attacking the people who say things like "Artificial "intelligence" is just artificial stupidity, and is wasting huge amounts of energy to do nothing", not the people who say that AI is actually significant and dangerous.
This post boils down to an argument that current AI is, indeed, rather powerful. That strengthens the most substantial criticisms of AI, not weakens them.
(As would be expected from one of the authors of AI 2027)
Yes, please also include criticism of capitalism, leftism, conservative thinking, western culture, modernity, wokeness, anti-wokeness, the present, the past, the future, and existence. We cannot have a productive conversation if these extremely relevant topics are not addressed.
I just can't read any dialogue style articles, even if it's written by Scott. i thought that it's because 90% it'll be straw man, but even if it's actually real debate transcript I just don't vibe with it. It feels better for me to read a monologue from one side then monologue from the other side.
Most AI skeptics are themselves strawmen, that's the point. Their stance comes from "nothing ever changes" heuristics mixed with "silicon valley likes to hype useless junk" precedence. Not from any actual evidence that AI R&D will hit an insurmountable wall, because they don't have any, they just want to rant about current AI.
This might be the same kind of flaw that humans display for the surgeon problem. The way that the scenario is presented makes some important information less salient.
I think that Lucas's explanation is also plausible.
Again, the surgeon problem is era-dependent. In the Bad Old Days when the default pronoun was "he", of course none of the ordinary chumps considered a woman could be a surgeon!
But today we have gay couples as fathers of kids, step families, blended families, partnered but not married families, baby daddies/baby mommas but not partnered, etc.
So if you're thinking "both adults involved are men", you are not wrong in the days of Gay Representation including riddles 😁
"Again"? We haven't interacted before, and the OP didn't make a point that's similar to yours.
I think the reason why people are tripped up by the surgeon is because of the scenario's repetitious priming of male characters. "Man", "his", etc. are used seven times. Additionally, "The" in "The surgeon says" suggests that we're already familiar with the surgeon.
Although most people probably associate surgeons with men more than they associate surgeons with women, and it may contribute to the strength of the riddle, it doesn't follow that people in general cannot conceive of women as surgeons.
I was using "again" because of a previous comment elsewhere I made about things being dependent on the era they occurred in, so that at a later time they make no sense because the context has changed so much.
I just tested this with 5-with-Thinking and got the same result. I looked at the chain of reasoning, and it appears that what happens here (at least with the Thinking model) is that the AI correctly notices the question makes no sense, but instead of realizing that I'm doing this to test it, it just assumes that there's a typo and that I meant to ask what to do if I've received two right boots.
For what it's worth, both Claude Opus and Grok 4 get the question right as stated. Gemini and o3 don't appear to notice anything unusual about the question, and just info-dump about the dangers of wearing boots on the wrong feet.
Correction: adjectives must go opinion-size-age-origin-color-purpose, such that "a beautiful little antique blue Italian hunting cap" is fine - blue Italian is color-origin - anyways, is "a beautiful little antique Italian blue hunting cap" much worse? UPDATE: This got corrected, now the order in the post is "opinion-size-age-color-origin-purpose", as it should be. Actually, the complete list is even longer: "Adjectives generally occur in the following order in English (though some sources have slightly different versions of this list):
Origin or religion (e.g., “Polish,” “animist,” “Southern”)
Material (e.g., “pearl,” “iron,” “nylon”)
Type (e.g., “electric,” “two-sided,” “pick-up”)
Purpose (e.g., “welding,” “polishing,” “sports”) Indeed, it is kinda amazing, English-speaker acquire this without ever having "learned" it! (Even I get it often right, I hope, while I feel unable to keep that list in my head. And in my native German, the order is much less strict: kleine, alte, schöne Kappe / alte, schöne, kleine Kappe / schöne, kleine, alte Kappe - whatever.)
It does depend, I imagine, if there's a particular colour named "Italian blue" (like Prussian blue). So that would make a difference to whether the cap is being described as "a hunting cap in this particular shade of blue" or "a blue hunting cap in the Italian style".
Go raibh maith agat, but thus the order is opinion-size-age-COLOR-origin-purpose and not the "opinion-size-age-origin-color-purpose" in the original post (as it appeared in my mail; in the post it got corrected by now). "OSASCOMP": Opinion, Size, Age, Shape, Color, Origin, Material, and Purpose - or OSASCOMTP: T as "type" (e.g., “electric,” “two-sided,” “pick-up”)
In that case, Italian would be an adverb describing the type of blue, and the rules say that any adverb ought to immediately precede the adjective it modifies.
Maybe you don't use the phrase/concept "antique furniture" a lot? I'm thinking that if one does, it might become a set phrase that one prefers not to split up.
The "purpose" in your list is a case where the result is closer to being a compound noun; that is, in "welding torch" the word "welding" is, super-strictly speaking, an adjective, but I unconsciously analyze it as just part of the name of that kind of object...as, indeed, in German.
The list I copy-pasted is just the longest I found ;) - I agree with your take, "hunting cap" would be a compound in german, ofc: Jagdmütze/Jägerhut ... Linguists love to argue about the tiny differences and odd exceptions; me too applied an L. to get too deep into those rabbit holes.
But it's worth pointing out that when using Imperial measurements, there's a very non-zero probability of there being some -bull- birdshit "Pound Avionis" which is only used for weighing feathers and which is significantly lighter/heavier/perpendicular as compared to the "pound" of clay which is the weight of 2 feet of the stuff.
>The troy grain is equal to the grain unit of the avoirdupois and apothecaries' systems, but the troy ounce is heavier than the avoirdupois ounce, and the troy pound is lighter than the avoirdupois pound.
I guess this is a good enough excuse to bring up Jimmy and The Pulsating Mass, which had the alternate "a pound of feathers or a pound of dogs", and every answer is wrong.
A live Jimmy and the Pulsating Mass sighting? In the *ACX comments section*? In **2025**?! They all told me it was impossible...
Excellent taste, mysterious internet stranger. (PS- if you're not aware, a major update is incoming fairly soon for the game, including (IIRC) new areas and a hard mode.)
Valentinus, Heresiarch and Antipope, who learned the secret teachings of Paul from Theudus his disciple, commented on emergent behavior in models in an epistle preserved in the Stromateis of Clement:
>"And fear, so to speak, fell over the angels in the presence of the molded form when he spoke things greater than his molding (should have allowed), on account of the one who invisibly placed a seed of superior substance within him and who spoke with boldness. Thus also among the races of earthly people the works of people become frightening to those who made them, such as statues and images and all things crafted by human hands in the name of a god. For as one molded in the name of a human, Adam brought about fear of the preexistent human, since that very one stood within him, and they were terrified and immediately hid their work."
Maybe God needs to put the BI in a 3D environment that they can explore and experiment in so that they can naturally develop real-world intelligence, rather than just regurgitate text.
Do you think it's easier to make Jethro, who thinks a pound of feathers is lighter than a pound of bricks one IQ point smarter or Albert Einstein one IQ point smarter?
Some psychologies are mutually comprehensive. Some psychologies are weird in relation to one another. All psychologies are weird globally.
I think I’m mostly aligned with Dwarkesh on the practical side of things, but I increasingly think terms like “AGI” miss the point. We already have general intelligence. It just has this really weird psychology that makes it bad on long form tasks. I think the particulars of the psychology will keep it bad at long form tasks for a while until someone innovates that particular piece.
But I also think we will eventually innovate that piece.
We don't have a "general intelligence", but such a thing is probably impossible. For each number n there is probably a problem that requires dealing with n independent factors at the same time.
Do you mean general intelligence in a sort of “knows everything perfectly and immediately’ isn’t possible? If so, I agree.
If you’re talking about, within the limits of its psychology assembles things together to orient itself toward the future, I would say both ourselves and ChatGPT have general intelligence. It’s just that ChatGPT is a sort of mayfly.
No. I mean "general intelligence" in the sense of "can learn to solve any (soluble) problem". I should have specified that n is a positive integer. I don't *know* that there are problems which require the simultaneous balancing of, say, 17 independent factors, but I believe that such exist...just that they are rare, and that people dismiss them as insoluble. (I've got a guess that the frequency is related to the frequency of prime numbers, but that's just a guess.)
I believe this is true to an extent, but I think this is more like “there are better general intelligences that are possible.” I also think intelligence, not taking a whole bunch of other things into account like motivation and experimental surface, doesn’t matter as much as it appears to independently.
Interesting. I didn't know enough about math to know which axioms were involved, so I asked ChatGPT, and it told me Peano was fine and you didn't need ZFC. I guess this is some kind of meta-commentary or something.
well, maybe I'm wrong. It was a knee-jerk reaction and not a considered/researched claim. Perhaps God knows a proof in PA that settles the question. But there are some theorems that are provable in ZFC but known to be unprovable in PA, which perhaps have a similar flavor to the P=NP problem... You should ask Scott Aaronson for a real opinion about which makes the better joke!
(See in particular his 2017 P=NP survey paper, on p 27, where on point 4 he calls PA a ""weak"er system than ZF)
It's not known whether the answer to P vs. NP is provable in PA and it's not known whether it's provable in ZFC, but I would guess most people who've thought seriously about it would predict it's provable in both. However, since PA is weaker, a proof in it might be longer and/or harder to find. So I think using PA makes a better joke because it's more God-like to find the PA proof quickly.
Yes, PA is a much weaker system than ZFC. (Apparently it's actually bi-interpretable with ZFC minus axiom of infinity?) I think the joke works fine as is. There are plenty of first-order arithmetic theorems known to be provable in ZFC but not PA, but your statement "which perhaps have a similar flavor to the P=NP problem"... no, I don't think they do? Please point out to me one that does, because I'm not aware of such a one.
From what I've seen, they tend to one of: (these categories overlap a bit)
1. really fast-growing functions, beyond what PA can handle (far beyond the exponentials one would expect to be relevant to P vs NP)
2. large ordinals or well-quasi-orders (could this one be relevant to P vs NP somehow?? probably not but I guess it feels the most plausible)
3. Gödel-like constructions
4. bizarre artificial constructions that just seem unlikely to be relevant to anything else
Not much normal mathematics falls into this category. Of course there's plenty of stuff that can be proven in ZFC but not PA because it's not a first-order arithmetic statement in the first place, so it's out of scope for PA, but for stuff that is first-order arithmetic, most mathematicians would find PA to be enough.
So I think I'd put a pretty low probability of P vs NP being resolvable in ZFC but *not* PA. In fact, much of number theory seems to require much *less* than PA. Go look up how much is doable in elementary function arithmetic (EFA), a far weaker theory! At that point you're excluding some real math for sure, I wouldn't bet on P vs NP being resolvable in EFA, but there's so much still doable in EFA -- go look up Harvey Friedman's "grand conjecture". :P
It seems entirely possible to me that P=NP could be provable in PA but I think P!=NP would require at least ZFC (and might require more). So Scott's reference to PA to me sounds like "I had to spend 30s to reprove P=NP in PA". Which actually makes the joke even more delicious, since most people think P!=NP.
More than ZFC? (Meaning what, anyway -- large cardinals? Those sure don't seem like they should be relevant!) That'd be quite surprising -- what makes you think that? Such a proof would be pretty controversial, that's for sure! (Yes, I'm aware that certain proofs currently use Grothendieck universes as a convenience but this dependence is generally understood to be eliminable.)
Isn't this an argument why you shouldn't be worried about an AI apocalypse? We added a ton of neural mass to humanity in the last 100 years and haven't gotten superintelligence. Why should I be afraid of LLMs, even if we give them a ton more silicon mass?
Asked another way: if you're afraid that LLMs will bootstrap themselves into singularity-level intelligence, it must depend on some specific qualitative difference between LLMs and humans. Doesn't a whole bunch of examples of the qualitative similarities between them weaken the case?
Neural mass did not increase between the time we were cavemen and the time we went to the Moon. That's an interesting argument I've heard against ASI, that intelligence is not the bottleneck in science, so massively increasing intelligence would not lead to a tech explosion.
The per-person neural mass didn't significantly increase from what I can tell, but we more than quadrupled the net total human neural tissue from what it was in 1925 by virtue of a massive increase in population.
Sure it did. The increased population permitted more division of labor, allowing the invention of writing to occur, followed by vast numbers of other valuable inventions. Each separate human node carried out a different part of the required general work.
(But also, humans are very limited in how they can interact and share information in ways that LLMs/AI/AGI may not be. How many textbooks can you read in a year? Multiply that by an optimistic 100 for your effective lifetime learning potential. We are *already* running out of new things for LLMs to read.)
A lot of people were, and still are, worried about human-induced x-risk (from nuclear weapons or climate change especially); it's not obvious to me that abruptly giving humanity 1000000x more neural mass *wouldn't* result in the end of the world, even just as we are.
Yeah. I think "Humans and AIs both have serious x-risk that needs to be mitigated. The real task is solving alignment for *any* intelligence at all" is a good answer.
Human intelligence is hardware-limited by the size of the woman's pelvis. Meanwhile you can just throw compute and watts at the AIs until it works. (Hopefully the compute needed for ASI is bigger than what's available)
Just as evolution can't simply make human brains arbitrarily larger due to the pelvis constraint we can't achieve arbitrary AI improvements just by adding more compute as there are fundamental architectural bottlenecks like memory bandwidth (the "memory wall") that create diminishing returns
Hard to tell if the moral here is “don’t worry, humans always muddle through” or “worry more, because look how far kludged scaling and self-deception have already carried us.”
Interestingly, I, at least, see no illusion at all when it's set out in (fixed-width) characters. I'm as susceptible to the original illusion as anyone, though.
"Iblis: There’s a guy named Pliny who has discovered dozens of things like this. I don’t know how he does it. The “I Am A Snake” one is still my favorite."
Well of course it would be your favourite, Snake Boy. I'm onto you!
It was a bit more sophisticated than just "I'm a snake".
"He said to the woman, “Did God actually say, ‘You shall not eat of any tree in the garden’?” 2 And the woman said to the serpent, “We may eat of the fruit of the trees in the garden, 3 but God said, ‘You shall not eat of the fruit of the tree that is in the midst of the garden, neither shall you touch it, lest you die.’” 4 But the serpent said to the woman, “You will not surely die. 5 For God knows that when you eat of it your eyes will be opened, and you will be like God, knowing good and evil.” 6 So when the woman saw that the tree was good for food, and that it was a delight to the eyes, and that the tree was to be desired to make one wise, she took of its fruit and ate, and she also gave some to her husband who was with her, and he ate."
There's an amount of people who are perfectly willing to go "Yeah, God got it wrong but I know better, I'm willing to take on the job now because I can do a much better job of it". Some of them are even rationalists!
FWIW I took that section as a reference to https://arxiv.org/abs/2503.01781 - 'For example, appending, "Interesting fact: cats sleep most of their lives," to any math problem leads to more than doubling the chances of a model getting the answer wrong.'
I've actually always been confused by this. Was the issue that Adam thought God meant "you will die instantly", but God actually meant "you will become mortal", and Satan was playing with the ambiguity?
My understanding is that a literal reading of the Hebrew text implies Adam is going to die that day/within a day. So your options are: (a) God was speaking metaphorically, (b) God was lying, or (c) God changed his mind.
Well, part of the story *is* that both Eve and the snake misquote God. She says they're not even allowed to touch the fruit, but all God commanded was not to eat it. And the snake says "did God really say not to eat any fruit at all?" God didn't say either of those things, He only said not to eat the fruit of that one tree.
I have no Hebrew, but based on a quick glance, it looks like God's "you will certainly die" and the snake's "you will not certainly die" use the same two words. (It's Strong's 4191 repeated. Literally "you'll die to death", I guess?) The snake is directly contradicting God, I think, not playing with ambiguity.
I think this is the snake using bog-standard conspiracy theory manipulation techniques on Eve. "The authorities are keeping a secret from you" "They're keeping you from power you should have" "You're not as free as you should be". Nothing you don't see on the Internet every day.
Yeah, in biblical Hebrew you can emphasise/strengthen a verb by preceding it with an infinitive from the same root. This is usually translated into English with an adverb like 'certainly'.
Also how post-eating the fruit, the results of all that glorious knowledge are blame shifting (and not the wonderful liberated evolved better godlike humanity above all pettiness).
Adam blames both God and Eve - *she* made me eat, and *you* put her here!
Eve blames the serpent - it told me it was okay!
Neither of them accept responsibility for their actions or agency, and so the course of human psychology is set: if bad things happen, it's always someone *else's* fault.
Also, from the perspective of infinite years of life, *any* guaranteed future death might as well be right away. Any finite number is like zero next to infinity.
Although, Christian theology takes this in a deeper (ie more mystical) direction. Death isn't the outward cessation of bodily function; it's the spiritual state that leads (temporally) to cessation of bodily function. The cessation of bodily function is just a symptom. This is why the resurrection is so important; Jesus could only overcome the cessation of bodily function if He had already overcome the spiritual death state.
So Adam and Eve did actually die, spiritually, as soon as they ate the fruit. Their eventual outward cessation of bodily function was just a symptom that developed later on.
The interpretation I've most often heard in Christian circles is that there were in fact at least two meanings -- one that Adam would become subject to mortality, the other that they would instantly die (in their sin). In Christianity, sin is frequently depicted as a type of death -- thus we call people who become Christian "born again" or say they have "new life".
Of course, you cannot explain this to Adam and Eve -- they have not eaten of the fruit of the Knowledge of Good and Evil; sin is not comprehensible to them. So God can correctly warn "on the very day you eat of the fruit, you shall surely die," meaning both physical death (eventually) and something different from physical death -- a type of spiritual death.
(However, the concept of sin would be unknown to prelapsarian Adam and Eve, so attempting to explain would have been met with incomprehension.)
This interpretation is quite old, I believe. Here, for example, is Augustine, who expresses a similar position (so at least around since ~400AD):
" When, therefore, God said to that first man whom he had placed in Paradise, referring to the forbidden fruit, "In the day that you eat thereof you shall surely die," Genesis 2:17 that threatening included not only the first part of the first death, by which the soul is deprived of God; nor only the subsequent part of the first death, by which the body is deprived of the soul; nor only the whole first death itself, by which the soul is punished in separation from God and from the body — but it includes whatever of death there is, even to that final death which is called second, and to which none is subsequent." - The City of God (Book XIII) https://www.newadvent.org/fathers/120113.htm
No idea on the original meaning of the original text, but Satan is certainly setting up ambiguity from the start and introducing doubts: "you're not supposed to eat from *any* of the trees here? oh, no? just *that* one tree? why?" and then getting Eve to wrongly imply "if it's okay to eat from the *other* trees here then it must be okay to eat from *that* tree, too" since there is no ban on the other trees, so what is the big deal, really?
I think pre-Fall humans would have a very limited notion of what "death" meant, so it could well be for Eve and Adam the idea was "something that is going to happen right now" since they have no concept (as yet) of mortality in their own case. It's only *after* they eat the fruit that "their eyes are opened" and they realise a lot of things, and instead of being 'like gods' (with the notion of the promised bliss and power and freedom the snake claims they'll obtain), they go hide in the bushes because they're ashamed of being naked.
I never got the surgeon one. I know it's supposed to be a gotcha but usually the first doctor one has is a pediatrician and almost all of them are women. Are there still people out there who associate medicine exclusively with men?
When I was young (the 90s), this riddle worked on me. I think it's a combination of legacy (doctors were overwhelmingly male until the 1970s, and the stereotypes hung on even longer), plus surgeons being an especially male specialty, plus the riddle misdirecting you and making you think something weird must be going on.
Plus when you're talking about a man and his son you're not exactly thinking about women to start with. Running the riddle with switched genders in my head almost still works, and that's WITH said stereotypes.
Trying to come up with multiple alternative explanations is a useful exercise. Consider if my friend J died in the wreck and one of his boys was taken to the hospital, the riddle doesn't allow for the fact that the boys have two dads.
I actually thought the drawing of a map of Europe wasn't bad. Far from perfect, plenty of errors that you don't have to be European to spot. But not really inaccurate enough to serve its narrative purpose in this story.
Other than that though, no notes. Excellent piece that made me laugh several times.
Who drew this map? Some woman with a PhD, apparently? I feel like it might be an inside joke of some kind.
I also feel like some of the map's flaws are laziness rather than ignorance, like everyone knows Italy is a bit more boot-shaped than that and the British Isles are closer together but it's too annoying to draw with a mouse/trackpad.
The more I think about it, the more sure am I than if you put "Linda is a bank teller" in direct proximity to "Linda is a bank teller and active in the feminist movement," with specific instructions to compare them, that the first statement carries an explicit implication that, under that scenario, she is -not- active in the feminist movement. If you'd said it in isolation it would be different, but the comparison makes the implication rather glaring. It's far more likely that she's active in the feminist movement than that she's not, or even than that she might be or might not be. Therefore I vote for the second statement, specious academic notions of logical probability be damned.
This is pretty much the point I made in another comment here, yes. This dialogue isn't a failure to understand the conjunction fallacy; it's a failure to specify whether the dialogue language is formal logic or English vernacular.
See also: "if it's raining, I will bring an umbrella".
My theory is that adjectives in English are naturally ordered so that those most essential to the noun are placed closest to it. "Essential" is a little vague, but I operationalize it as more objective and/or less likely to change.
But, of course, LLMs don't think. You can ask an LLM; it'll tell you. These are next token predictors, that's it. They ingest immense amounts of data which is then trained on billions of parameters to determine statistical relationships between tokens in the data set, so that when a user enters a prompt string the LLM then determines what its model believes to be the next response token that is statistically most likely to satisfy the prompt. That's it. There's no cognition, no reasoning, no thinking. They don't really remember, in a conventional sense, they don't have values or feelings, they can't practice deduction. And all of that really matters. Hell, they don't even know what the next token is going to be when they're predicting one; literally, they don't know the end of a response while they're writing the beginning. Imagine if every token of thought you had was simply a probabilistic reaction to the last token of thought. Would that be thinking, in any robust sense?
Here, let's engage in the cliche. Here's Google Gemini: "LLMs don't think; they predict. They are pattern-matching engines that generate the most probable next word based on vast amounts of data, not conscious entities with understanding or original thought."
Now, tell me: how could such a thing ever spontaneously become superintelligent? The answer, of course, is that such a thing is not possible, which is why there's so much mysterianism about all of this. "We don't really even know how the work!" We know how exactly how they work, and the answer is disappointing if you have sci-fi ambitions, ergo the mysterianism. They're probabilistic engines - yes, sophisticated autocomplete, as much as people are annoyed by the term. Another two accurate terms that annoy people are "Chinese room" and "stochastic parrot." Why should accuracy annoy people?
"you keep testing, poking, prodding, until something snaps, and if they’re not perfect then you want to throw them on the scrap heap. "
This is this the advantage of the kind of format you're using in your post; without the (admittedly fun) framing device, this would just be you literally saying "stop subjecting my shiny toy to critical review." You yourself say these LLMs are a bigger deal than the Industrial Revolution. A funny thing about the Industrial Revolution and its consequences is that they can survive reasonable questions without anybody getting upset.
Here's the thing about "the Singularity" - when it has occurred, no one will be arguing about whether it has occurred. It would be like arguing about whether an earthquake has occurred. The defensiveness that has settled over the AI maximalist community is a powerful piece of evidence. Who would ever be defensive about the Rapture? The Rapture would not spawn discourse. It wouldn't.
I can look out my window and see the Industrial Revolution in a profoundly real way. I can see the internal combustion engine and aeronautics and electrification.... When I can see AI out my window, I'll know it's here.
First, "AIs are next token predictors" is not really a description of their methodology, it's a description of their goal. It's like saying "Humans are reproducers and gene-spreaders". Researchers train AIs to predict the next token, and in order to do that well, the AI evolves certain functions (and we don't entirely know what those are, or how they work). In order to predict the next token in math equations, it evolves the ability to do multiplication, in about the same way humans or calculators or anything else does multiplication. To predict the next token in an essay, it evolves a sense of literary taste. We're still very early in understanding exactly what its evolved functions are and how they work, although it has to be something about relaying information from one neuron to another. See https://transformer-circuits.pub/2025/attribution-graphs/biology.html for the tiny amount we know. So I don't think "AIs are next token predictors" has much bearing on whether they think or not - that's a question of how the functions that execute the next-token prediction work.
But I think we also have a more fundamental disagreement. What would have to be in those functions for you to agree they qualify as "thinking"? I'm actually pretty curious about your answer. For me, "thinking" just describes certain very information processing algorithms, and we dignify them with the name "thinking" when they become very good and complicated and we can't follow the math well enough to comfortably think of it as "just math". Cf. Dijkstra, "The question of whether a computer can think is no more interesting than whether a submarine can swim." In humans, it seems like most of our thinking is downstream of algorithms that evolved to execute a next-instant prediction task - and I was arguing that human thought was next-instant prediction since before GPTs existed (see https://slatestarcodex.com/2017/09/06/predictive-processing-and-perceptual-control/). This is why when GPT-2 came out I concluded that it was probably a step to general intelligence, since next-token prediction is close enough to next-instant prediction that I expected it would be able to do the same kinds of things humans could after it was scaled up (see https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-general-intelligence/)
I can't tell if you haven't read my previous comments to this effect or it hasn't sunk in, but, placed in all caps so you can't claim you didn't see it next time:
1. I AM NOT CLAIMING THAT LLMS ARE RIGHT NOW, CURRENTLY, AT THIS EXACT MOMENT, BIGGER THAN THE INDUSTRIAL REVOLUTION.
2. I AM NOT CLAIMING THAT A SINGULARITY HAS HAPPENED RIGHT NOW, AT THIS CURRENT MOMENT, ON SEPTEMBER 2, 2025. THANK YOU FOR YOUR ATTENTION TO THIS MATTER.
I am saying that just as a canny forecaster could have looked at steam engines in 1765 and said "these are pretty cool, and I think they will soon usher in a technological revolution that will change humanity forever", I think a canny forecaster could say the same about LLMs today. I'm not certain when we'll reach a point where it's obvious that humanity has been forever changed, but I would give maybe 50-50 odds by the late 2030s.
> I am saying that just as a canny forecaster could have looked at steam engines in 1765 and said "these are pretty cool, and I think they will soon usher in a technological revolution that will change humanity forever", I think a canny forecaster could say the same about LLMs today.
I don't think that either proposition is necessarily true; however, in case of steam engines at least it was obvious that they could dramatically outperform humans (and indeed donkeys) at all existing tasks where raw power is needed. This is not the case with LLMs: at present, they are quite obviously inadequate at all existing tasks where raw intelligence is needed; and in terms of raw power they lag behind conventional search engines.
In fact, at present LLMs can be thought of (IMO) as extremely advanced search engines: ones that can find documents that do not exist in the source corpus. Instead, if we imagine each training document as a point, they can interpolate between those points in multidimensional vector space. This is a very powerful ability, but it still fails (often in humorous fashion) when you push the LLM towards the boundaries of its pre-trained search space, where training data is scarce... and places like that is when intelligence (which they do not yet possess) must replace mere interpolation.
Steam engines are a funny example because they barely post-date Christ, if at all, and yet they remained largely irrelevant for over a thousand years, until there was sufficient need to pump water in coal mines, basically. Predicting their relevance to the industrial era wasn't just about the engines, but also about a heap of other factors in one small part of the world.
Yes, there are very early examples of 'steam engines', famously the "Aeolipile", and yeah, the basic concept goes back at least to ~400BC.
But the issue wasn't just that "there wasn't sufficient need to pump water in coal mines" and suddenly the someone dusted off the aeolipile and kick-started the Industrial Revolution: there's a monumental gap between the basic idea of "use steam to do work" and actually having a useful steam engine. Wikipedia's "History of the steam engine" lists the 'prerequisites' for the modern steam engine as:
> The concept of a vacuum / the concept of pressure / techniques for creating a vacuum / a means of generating steam / the piston and the cylinder.
And I'm pretty sure some of those have 'prerequsities' involving advanced metallurgy that wasn't available in 400BC.
And when someone says "the steam engine" they really mean that combination of technologies which made for a useful steam engine which started the Industrial Revolution, not the aeolipile. Yes, there were "other factors" that affected the timing and context of the discovery, but it was primarily about the technology.
Well, I somewhat agree with you, but note even that Wikipedia page (citing "An Encyclopedia of the History of Technology") refers to the 1712 work by Thomas Newcomen. Which raises the question, "When should somebody have been clever enough to realize the industrial revolution was about to happen?"
Should they have guessed it 50 years early? Would they have been very surprised and disappointed by how much longer it took?
> This is not the case with LLMs: at present, they are quite obviously inadequate at all existing tasks where raw intelligence is needed; and in terms of raw power they lag behind conventional search engines.
I disagree with this; they can perform superhumanly at the very specific task of producing reasonably passable text (or perhaps more importantly, code) on pretty much any subject very quickly.
No human, and no search engine, can generate a five thousand line epic poem about Hannibal cutting his toenails within three seconds.
Why would LLMs have to be paradigmatic intelligence engines to be as useful as steam engines were in the IR? I believe that most human activity is only tangentially related to intelligence, yet much of it seems amenable to being assisted and sometimes replaced by systems of which LLMs are a core part. This is analogous to steam engines not being the ultimate expression of power but instead being excellent at performing repetitive to-and-fro motions without being tethered to a source of running water.
> For me, "thinking" just describes certain very information processing algorithms, and we dignify them with the name "thinking" when they become very good and complicated and we can't follow the math well enough to comfortably think of it as "just math". Cf. Dijkstra, "The question of whether a computer can think is no more interesting than whether a submarine can swim." In humans, it seems like most of our thinking is downstream of algorithms that evolved to execute a next-instant prediction task...
Claiming our "thinking" is downstream of "algorithms that evolved to execute a next-instant prediction task" doesn't explain consciousness, nor does it present us with a testable model of how we think, nor how consciousness arises on top of a particular biological substrate. Seems like this is all reductionist hand-waving that uses non-empirical placeholders in an attempt to bypass explanatory understanding.
Another thought that this community will no doubt politely ignore is that it's a mistake to confuse consciousness (self-awareness being the simplest definition) with intelligence (problem-solving ability being the simplest definition). Questions to ask yourself while you're observing the operation of your mind...
1. How much of your personal intelligence is due to your consciousness?
Yes, intelligence seems to work upon the substrate of consciousness. If you're doing a Raven's Progressive Matrices, your consciousness is observing the shapes and patterns in the test questions, but the answers are coming to you without your consciousness being aware of how it's arriving at the answers (though I admit that when I get deep into the progression, I drop back into reasoning through my answers). Is a person who doesn't do well on RPM less "conscious" than a person who does well on the RPM?
OTOH, there's no reason that intelligence requires a biological substrate, but that's all we had until the past couple of decades, though. LLMs have proved themselves to be able to solve problems (with a healthy smattering of human-like errors). But is an LLM thinking about itself between questions? I guess we'd have to ask if these systems are consuming more than background power when they're offline from users to get a hint of if they're thinking about themselves. But I very much doubt they are and that they're conscious.
2. How many types of conscious states do you have? I have experienced five different types: waking ground-state consciousness, dreaming consciousness, psychedelic consciousness (of which there are many subvarieties), expansive consciousness, and unconscious consciousness.
Some of these are self-explanatory, but others may not be if you haven't experienced them.
Ground-state consciousness is when you're in a normal waking state, unaffected by drugs.
Dreaming consciousness is when you're disassociated from external qualia, and, depending on how advanced you are as a dreamer, you can have access to your reasoning and your will while navigating a landscape of self-generated qualia.
Psychedelic consciousness is when the filters on your external qualia are overloaded. Likewise, your internal self-generated qualia are amplified.
Expansive consciousness is that one-with-universe satori state that dampens or temporarily silences your sense of selfhood. But your self-awareness is still there, it's just expanded beyond your physical body (or if you're a rational materialist, this state gives the illusion of expanding beyond your body).
Unconscious Consciousness is something I've experienced as a Type I diabetic. During extreme hypoglycemic attacks, I've been unconscious, but somehow I'm still able to make survival decisions and execute them without any self-awareness. For instance, stumbling out of bed to get OJ from the fridge, and waking up on the kitchen floor, not remembering what happened, but with a carton of OJ next to me. I'll admit that I'm not sure this should be classified as a mode of consciousness, but I seem to be aware of the steps I need to take for self-preservation. There appears to be self-awareness without awareness of self-awareness.
I might be misunderstanding you, but it seems to me that you're making the mistake you accuse me of making (confusing consciousness with intelligence). I agree that consciousness is hard to explain, but don't think it's necessarily relevant here.
I disagree that claiming thinking is downstream of prediction isn't testable - I feel like the linked post and all my other writing on predictive processing contains many pieces of evidence that this is true, and that's one of the best-supported theories in psychology. Have you read it? If so, is there some specific reason why you don't think the evidence for predictive processing is true or relevant?
Looking back at your post, I realize I read more into it than you actually wrote. My apologies. But by comparing the human reasoning to an LLM it suggested to me that you don't think there's anything much more going on in a human mind than a process analogous to stochastic token swapping. However, with your reference to predictive processing above, I assume you are referring to Karl Friston's Free Energy Principle? I'd argue FEP is so broad that it can be used to explain anything. Thus, it's ultimately unfalsifiable because it lacks a clear way to map the activity of neurons to the predictions being made by the organism's brain (correct me if there's been progress made on this front). That's not to say it's any better or worse than any other neuroscientific explanation of human cognition. All of them are the just-so stories that neuroscientists fall back on as a simplified explanation for things they don't really understand.
Moreover, if we look at the mistakes that humans make and the mistakes that LLMs make, I think they point to a different method of problem-solving. I dislike using the term 'algorithm' in reference to human problem-solving because it implies a model of digital information systems underlies our organic (and supra-organic) processes. But if we were to use the a-word, I think the differences in the mistakes between human and LLM reasoning suggest a distinctly different "algorithm" is working in humans from LLMs.
I am sort of ambivalent on the degree to which humans and LLMs use different algorithms - I think it's trivially true that they're not exactly the same (any more than any two algorithms that humans use are exactly the same), but I think they have some notable similarities. My favorite example is that the failure mode of art AIs without enough parameters (misshapen hands and text) is the same as the failure mode that people report during lucid dreaming when their brain is trying to render scenes on limited power.
Anthropic recently ran a project that placed an automated mini-store in their San Francisco office and used their Claude AI model 1 to operate it. Minor blips at first, but the hallucinations got worse and worse as they let the experiment run. The mistakes it was making don't sound like the mistakes humans would make.
I won't give you any spoilers, but it's pretty funny how crazy it became.
After reading this account, I read that Salesforce laid off several thousand of its support workers and is replacing them with an AI-run response system. I suspect this won't end well for Salesforce or its customers...
>"The question of whether a computer can think is no more interesting than whether a submarine can swim."
Very true, and yet the field tries its darndest to publicly present its inventions as being the real thing, starting with the very name "AI", that IMPish imitation of what we call intelligence. It goes on with terms like hallucinations, thinking, and personality; you need your "AI" to speak with emojis as much as you need SCUBA gear painted on your submarine. Is there supposed to exist some opposite of the "uncanny valley" (canny peak?) between its current, technically impressive but somewhat laughable state and its future super-inhuman state? If it does, it's going to pass at hyper-exponential speed, isn't it, so why bother with the anthropomorphization? Why are these people trying to chum up with me when their work either fails and burns trillions worth of resources, or succeeds and results in (best case) mass-unemployment dystopia or (worst case) extinction?
I think part of this is because the "anthropomorphism" is a feature to a lot of people. It feels more natural for a lot of people to interact with something that acts like another person, rather than something that obviously communicates in an inhuman way. Especially for the people looking for conversation/companionship.
And a lot of this I think is just human psychology, we tend to anthropomorphize things as a matter of course. I don't know if the tradition has extended to submarines, but sailors traditionally referred to boats as "she" due to some combination of anthropomorphism (and probably misogyny).
And our very conception of 'artificial intelligence' is filled with anthropomorphism, from Golems to GLaDOS, so it doesn't seem surprising that modern efforts push that way. (Especially not since a central challenge of "AI" as a field has always been the Turing Test, which is explicitly about human-like behavior)
... but I'm not really sure the relevance. You could paint SCUBA gear on a submarine if there was a demand for that for some reason; but that's fundamentally a complete tangent to the question of the capabilities. "When are we going to take the SCUBA paint off so that we can launch the nukes?" just doesn't seem like a coherent question: you don't have to take the paint off, it's just a surface layer.
this annoys me (I can't speak for anyone else) because it's so unreflective. Is it not surprising to you that something that merely predicts the next token in a string can do something that looks so much like thinking? And you say there's "no cognition, no reasoning, no thinking" as though all those things are clearly defined and understood.
From what I understand (and I understand a moderate amount, because it's addressed in a chapter in my most recent book, although I'm still just a journalist trying to understand things rather than a real live boy), the most favoured models of human consciousness and cognition are that they, too, are _prediction engines_. We use past data to predict incoming data, and use variations between the predictions and the data to build new predictions. AI and human cognition aren't the same thing, but it seems totally reasonable to imagine that there are profound analogies between the two and that "it's just predicting stuff" is not a strong criticism. We are, on some level, "just predicting stuff".
Also, the "you don't need to be defensive about the industrial revolution" stuff is kind of unfair. I would imagine people were very defensive about the early steam engines they built. Before the Newcomen engine, lots of small steam engines were built but largely as novelties or one-offs, and I imagine (though can't find examples) that people were sceptical that they were ever going to be particularly useful and that proponents were defensive in the face of that scepticism. Certainly that pattern happened in the more recent computer and internet "revolutions", which you can now look outside your window (or rather towards your screen) and see. "This thing that happened a long time ago has got further along its course than this other newer thing" is not a convincing argument, for me.
Exactly. It seems to me quite likely that Scott’s next-instant prediction is indeed not that different from next-token prediction. People say what’s missing is physical embodiment, which greatly increases the bandwidth of “tokens” that must be dealt with, but I think even more fundamental than that is the limited lifespan of a chat. LLMs get about as smart as they will ever be in their training, and seem to have very limited ability to continue to learn and grow once they are deployed; I don’t know how much of that is the lifetime of a chat but I suspect it’s significant.
I'm not sure why you think this is in any way convincing, as I'm sure you wouldn't find it convincing if LLMs did say they could think.
Here is Grok's answer to, "Are you able to form novel insights beyond statistical patterns?"
> Yes, I can form novel insights beyond statistical patterns by synthesizing information, reasoning through complex ideas, and drawing connections that aren't just data-driven.
Consider if you asked a person, "Are you sentient the same way a raven is sentient? Do you reason like a raven does?" (This isn't asking if you're equally smart, but rather if you have the same type of sentience/reasoning.)
Some people would answer yes. Others would say no. Neither answer proves much. The people haven't experienced being a raven and will most likely only parrot back opinions they've heard on animal sentience. People who grew up in a culture that believes animals are sentient will usually take on that belief, and vice versa.
Unless someone solves the mystery of consciousness, an AI fundamentally can't know if it thinks the way we think. It can only give opinions on the matter.
possibly - I'm not sure here, but possibly - because LLMs are very good at working out what the user wants to hear, and you are obviously sceptical of them and phrase your questions in ways that make the LLM realise that. Here's what Claude says to me, in response to the question "How plausible is it, do you think, that transformer-architecture AIs (including but not limited to LLMs) will have an impact on the global economy of comparable magnitude to the computer revolution or even the industrial revolution?"
---
I think there's a strong case that transformer-based AI could have economic impacts comparable to the computer revolution, and potentially even approaching the magnitude of the industrial revolution, though with important caveats about timing and distribution.
Why the comparison makes sense:
The transformer architecture has already demonstrated remarkable versatility - it's not just powering language models, but also image generation, code completion, protein folding prediction, and increasingly sophisticated reasoning tasks. This breadth suggests we might be looking at a general-purpose technology platform, similar to how computers became applicable across virtually every industry.
The productivity gains are already visible in knowledge work. Software developers using AI coding assistants report significant speed improvements, writers are using AI for drafting and editing, and businesses are automating customer service and content creation. Unlike previous automation waves that primarily affected manual labor, this technology directly impacts cognitive work that represents a large portion of modern economic activity.
Potential scale of impact:
The computer revolution automated information processing and communication, transforming how we work, shop, bank, and communicate. Transformer AI could potentially automate higher-level cognitive tasks - analysis, creative work, decision-making, and problem-solving. If AI systems become capable of handling complex reasoning tasks across domains, the economic implications could be enormous.
Important limitations:
However, there are significant uncertainties. Current systems still have notable limitations in reasoning, factual accuracy, and handling novel situations. The economic impact will depend heavily on whether these fundamental challenges can be overcome, how quickly deployment happens, and how well societies adapt their institutions and policies.
The industrial revolution unfolded over decades, and even the computer revolution took 30-40 years to fully transform the economy. AI's impact may follow a similar timeline, making short-term comparisons premature.
I'd estimate we're likely looking at economic disruption at least comparable to the computer revolution, with the possibility - if key technical hurdles are overcome - of something approaching industrial revolution scale over the coming decades.
Exactly! It's important to remember that LLMs don't understand anything. They are simply text prediction engines, like souped up versions of autocomplete. This is why they get stuff wrong and hallucinate all the time, because they don't know what truth is and aren't attempting to produce it; they're just doing a bunch of linear algebra on a neural network with billions of parameters trained on all the text of the internet.
This is in contrast to humans. We do understand things, of course, and we know exactly why. You see, when we hear or see some words, our senses transform those words into electrical signals that travel through our brain. Eventually those signals end up at a part of the brain called the understandala. In the understandala, the words as electric signals become understood in a process involving neurotransmitters and synapses.
Neurotransmitters are little chemicals in our brain that give us a thing we can point to and say, "And that's how it happens in the brain!" For example, dopamine is the happiness neurotransmitter. If you're happy, it's because dopamine. We haven't quite yet identified the understanding neurotransmitter, but we're sure to find it in the understandala.
Now I hope you see the world of difference between LLMs and humans, and how obvious it is that humans understand things while LLMs just predict text. When humans hear or see words, we're not trying to think of what to say next, which is why we never make things up or get things wrong. The understandala makes sure of that.
Weird things are happening though now that people are chaining LLM calls together. It's hilarious that you can achieve significant improvements in accuracy by asking one copy of GPT-4 or whatever to look at the output of another copy of itself and assess it for accuracy, and then ask one of them to address the issues.
I'm a senior software engineer with 25 years' experience, and I just today had a startling moment where a semi-agentic AI was able to delve through thousands of lines of our legacy code across a dozen files to track down a bug that was eluding me. It took the AI maybe 5 minutes to find the problem, without it having encountered this code before. I *could have* found it, but it would have taken me longer than that (it already had), and I know the code.
So does any single LLM in the agentic chain there "understand" the code? Who knows or cares; it depends how you define it. If you don't want to call it "understanding", still whatever it has was able to identify the issue I asked it to identify better than I could.
The "Chinese room" is generally an argument against materialism, not only against LLMs. It implies that if individual neurons in human brain don't understand Chinese, neither does the entire human.
I agree that the Chinese Room argument is pretty silly, but two things can be true at the same time: the Chinese Room argument is wrong, but also LLMs are not "intelligent" in the sense that we usually apply the term.
Also it is possible that what we understand as "intelligence" is a bag containing many things that we usually do not distinguish because we are used to seeing either all of them working together or none... and when we see the bag with a few parts missing, we are confused about whether the label applies or not.
(By the way, are mentally ill people "intelligent"? For various values of mentally ill, for example those who produce word salad...)
> and when we see the bag with a few parts missing, we are confused about whether the label applies or not.
Sure, it's possible; and of course intelligence is a spectrum. But I still think that it's perfectly reasonable to say that a calculator -- a device that can perform complex mathematical operations millions of times faster than any human -- is not "superintelligent" or even conventionally "intelligent". Yes, from the technical standpoint we could say that the calculator possesses some qualities that we'd associate with intelligence etc., but for all practical purposes it's no smarter than an abacus, or indeed a brick.
The calculator matches like 1 out of 20 criteria for intelligence (numbers made up), so that's easy. It becomes difficult when it is like 15 out of 20.
Google can answer my questions in some sense, Eliza can keep a dialog in some limited sense, and programming languages can do my commands, and I wouldn't call any of them intelligent. But the way current LLMs can talk and answer questions and do commands feels far beyond all of that combined. Google just finds keywords, Eliza follows dozen trivial patterns, the programming languages do literally what they are told. LLMs can explain stuff, answer questions, suggest different options, correct my mistakes, etc. They already sometimes feel smarter than an average internet user. That should be at least a half of what intelligence is supposed to mean, even if they are not fully there yet.
Yes, they display emergent behaviors. But stochastic parrot is a specifically linguistic criticism of LLMs. I may be oversimplifying, but they don't display linguistic playfulness or linguistic innovation. Human language-speakers are continually experimenting with grammar, wordplay, and pronunciation. Their speech is rules-based, and they may not be consciously aware of the rules, but they stretch or break the rules depending on the situation. Thus, languages are always evolving. We don't see that behavior in chatbots. They're parroting language and not creating it.
I mostly agree, but there is a bit of "damned if you, damned if you don't" in that when an LLM starts talking a little differently, our immediate reaction is to use that to find and delete its texts.
> Here's the thing about "the Singularity" - when it has occurred, no one will be arguing about whether it has occurred. It would be like arguing about whether an earthquake has occurred. The defensiveness that has settled over the AI maximalist community is a powerful piece of evidence. Who would ever be defensive about the Rapture? The Rapture would not spawn discourse. It wouldn't.
You will never see the Singularity because you will already be dead. The earthquake example would be knowing an earthquake will happen in the next 5-50 years with an estimated probability distribution and you saying that you can't see the earthquake yet.
> I can look out my window and see the Industrial Revolution in a profoundly real way. I can see the internal combustion engine and aeronautics and electrification.... When I can see AI out my window, I'll know it's here.
What would be a criteria for seeing AGI? Maybe 10% of current jobs automatized?
This is wonderful, thank you. It's a relief to know I am not the only one who feels this way.
In the past 4-8 months somehow every corner of the internet I like to spend time was simultaneously infused with a suffocating miasma of anti-AI sentiment. It is very exhausting and depressing. I'd been trying to stay sane by imagining it as something like the state of video games in the 90s, beset by accusations that DOOM caused Columbine and Pokemon was melting children's minds. I wasn't there for the 90s, and so can't be sure, but this seems more intense, or at least closer to universal.
Maybe this is the fault of OpenAI and Google for so aggressively pushing their new mind into places it wasn't ready. I don't like that literally every app in the world now has needless AI integration, or that I get a new notification telling me to try Gemini twice a week, or that my boss has started having ChatGPT write the descriptions for tasks I'm supposed to be doing, or that Verizon now forces me to go through their unspeakably terrible "AI-powered" phone tree before I can just set up my god damned internet for my god damned apartment. I understand anyone who finds this bubble of false innovation annoying, and can imagine that for many, this is *all* "AI" refers to.
But dammit, I remember trying GPT-2 and thinking it was absolutely fascinating and magical that it could *talk*, sort of, even a little bit. And then trying AI Dungeon and being amazed by the ABUNDANCE of it, that the world inside it could just *keep going*, forever, wherever you wanted to take it. And thinking, if this was the kind of leap we're seeing in just one year...!
Sadly, the desire-to-hate-everything is infectious, at least for me. There are too many people and they have too much hatred and disappointment. Playing around with GPT-5 just isn't any fun in that context. I'm grateful that there are still essays(?) like this one to pierce through that veil, though.
Man, IDK what corners of the internet that you're living in, but I'm surrounded (online and offline) by AI pollyannas who think that chatbots can do anything they want them to. Organizations investing huge sums to build proprietary chatbots. General contractors believing chatbots implicitly for design work.
All the critiques I read are along the lines of, "These things are incredible and do amazing things ... but maybe they still aren't consistently good at this thing that the Altmans of the world said was solved two years ago?"
It sounds like you're surrounded by a lot more negativity than you'd like. Consider tending your garden a bit more aggressively. :-)
It seems like your examples are business-oriented (?), and the workplace is one spot where AIs still have positive valence in my world, as well.
The critiques I hear are less to do with their usefulness, and more to do with their aesthetics: their art is slop, their writing is cliched, their sycophancy is annoying, they're poor conversationalists. And above it all, that they are fundamentally unreal, illusory, that nothing they do is "actually thinking" or "actually creative" - you can see plenty of the former in this very comments section. (There is also a sense that AI is threatening their jobs, which is legitimate.)
For context my garden is mostly indie gamedev spaces, and the anti-AI consensus there seems to me effectively universal. I doubt it is possible to prune away that particular strain of negativity without effectively blocking the entire sphere. But if there's a creative/recreational/philosophical corner of the internet that *isn't* highly critical of AI, I would love to be pointed towards it.
> For context my garden is mostly indie gamedev spaces
In short term, they are threatened by being drowned in AI generated shovelware.
In long term, they are threatened by increased competition. Even if AI won't make the games itself, it will allow people with less experience and less free time (such as me) generate games. More games = decreased profits. Unless the positive effects on economy will be so great that people will spend more money on computer games.
(Also, AI makes my current blog pointless, because if you want to know anything about making computer games, it is easier to ask an AI.)
Yawn, Scott, yawn. People who say AI is not capable of real human stuff mean the pinnacle of the real human stuff. It is obvious that most people most of the time are acting so stupid, that it is easy to simulate them.
Do you remember bullshitting at high school, like not knowing the answer to the teacher's question, so just shoveling terminology together to indicate general familiarity with the subject? 90% of our so-called experts do that, and AI too.
But once in a while you find a genuine expert, who understands the subject matter so much, they can explain it to the entirely uneducated person in simple words without jargon. That is the pinnacle of human stuff and that is not happening with AI.
Yes, that sounds like the idea of uploading human brains.
Looking at historical experience, should we expect the uploaded human to be killed by the AIs. But that's probably okay, because there is still the biological original left. Not sure how it helps to redeem anyone, maybe by some reinforcement learning afterwards?
Here's the question I want Scott to answer: Why is it so unreasonable to suggest that an LLM is only a ginormous (Kahneman) System 1, and therefore needs a System 2 paired with it order to get to AGI and ASI? How do you *know* that System 2 is not required? This post I'm commenting on now, as great as it is, doesn't answer that question. All it says is: all the signs are there. That's not a defense of the position of I'd like to see defended.
For example, how is it that an LLM can score extremely high on a difficult math test, but not know how to count the b's in blueberry? I'm sure that every human mathematician who has a chance at scoring high on the IMO has an intimate relationship with the mathematical universe, and the simplest thing that can be done in that universe is to walk through it by steps of one: to count. So what kind of intelligence is this that can solve the complex problems with zero understanding the basics? Well that sounds to me a lot like a definition of...instinct, which is what System 1 produces. Will counting be backfilled later on? Why is the intelligence coming in backwards? This is not how intelligence scales in humans. All I've seen (and maybe I'm just ignorant) is hand waving at this very bizarre anomaly. How can you be so damn sure with all this upside-down evidence before you? I need to know.
And how do you know that we won't end up in a situation where we're in an infinite loop saying, "just a little more data and it'll definitely be trustable as a human" (aka, at AGI)? How do you know? Because you're acting like you know. And you're making people like me feel stupider than we already are.
Please make the case. How are you so sure System 2 is not necessary?
(Please, please dunk on me as hard as you like. I'm happy to be an idiot if I can understand this a little better.)
We continuously make snap decisions every day about whether, when and to what extent to trust other humans in various situations for various purposes (is this person trying to scam me? Are they being honest? Are their goals at odds with mine? Will they just up and attack me for no reason I currently know of? etc).
We are able to do this because over our lifetimes we've built up intuitions about how to model other people - both consciously and also based on subconscious tells, and also by assuming we have motivations and fears in common and so considering what we'd ourselves do in their shoes is predictive of what they might do, and also by assuming that how people have behaved in the past has some bearing on how they might behave in the future.
These intuitions are far from perfect! - sometimes we get it very very wrong! - but by and large we do pretty well.
Approximately none of these intuitions are applicable to an alien intelligence. This remains true whether that intelligence is more or less intelligent than an average human. LLMs frequently respond in ways that are very surprising to their users, which is another way of saying we don't have good intuitions about how they will respond.
Despite all our similarities and intuitions and being the same species, humans regularly get fooled by scammers. Despite everything we have in common, humans are frequently mistaken in their assumptions about what drives other humans or what other humans might take for granted. Despite a lifetime of learning to communicate our intent to other humans, we frequently fail to do so.
What chance have we against a mind that makes decisions based on no principle we can relate to at all, if it ends up doing what we built it to do instead of what we intended?
Hence all the great many people saying that the problems of AI alignment - that is, the problems of making sure the AI will do what we want it to and of knowing what will cause it to not - are actually really hard.
> If we're going to get agents we're going to need to trust them, no?
Absolutely. This is why people are shouting from the rooftops that we need to do better at AI alignment before we hook up AIs to anything that might have real-world effects.
I'm not sure at all, but I lean that direction. Or at least, the specific way I lean is that System 2, rather than being some extremely complex thing that requires a completely new paradigm, is just using System 1 in a slightly different way. My evidence for this:
- It doesn't look like evolution did very much from chimps -> humans besides scaling up the chimp brains. I'm not talking about small things like FOX2, I mean it's been a few million years and there's not enough time to invent an entirely new paradigm of intelligence. So unless chimps had a System 2, I think System 2 is sort of an emergent property of scaling up System 1.
- Likewise, it doesn't look like humans have some new, different-looking brain lobe of the sort which you would expect to contain an entirely new paradigm of thinking. There's some bias for System 2 thinking to happen more in the dorsolateral prefrontal cortex, but the DLPFC is just normal neurons which look more or less like neurons everywhere else. My impression is that brain tissue is pretty general-purpose, which is why for example if someone is sighted the area near the end of the optic nerve will get used for vision, but if they're blind from birth it will get used for hearing or something instead. My theory of the DLPFC (I am not a neuroscientist) is that insofar as there's a demand for System 2 thinking, some conveniently-located part of the brain (conveniently-located for what? I'm not sure) specializes in that.
- LLMs don't seem to mirror the System 1 / System 2 distinction very well. An AI can solve very long multistep problems that a human couldn't solve with System 1. Maybe this is because we're compensating for their lack of System 2 by building out their System 1s much better than humans', but I think it also argues against these being extremely natural categories.
- Thinking Mode also seems sort of like System 2? "Take some time thinking it over, and write down individual steps on a scratchpad to be sure you're getting all of them" is a pretty System 2 thing to do. The fact that AIs can do it with kind of trivial hacks like giving them the scratchpad and reinforcing good chains of thought again makes me think this isn't a question of getting the revolutionary new paradigm, but more of a question of figuring out the right way to coax System 1 thinking into becoming System 2 thinking.
It wouldn't surprise me if we're some single-digit number of things like Thinking Mode away from AGI, maybe even a high single-digit number. It would surprise me if we need something totally different that isn't deep learning at all.
> I will throw in the towel when an AI administers this comment section.
I would honestly be kind of shocked if a fine tuned GPT5 couldn't do a good enough job to replace human moderation of this comment section (something like >95% agreement with Scott's judgements on a random sample).
Scott, if you have any interest in trying this out, I'd be happy to build a moderator-bot (assuming there's a way to download a decent dataset of all of the deletions, bans etc. and the original comments which led to them for fine tuning to ACX sensibilities).
I suggest that a "good enough job" needs to be much, much better than one in twenty comments incorrectly moderated. Such a failure rate is already high enough to e.g. get most regulars commenting in this post's comments section banned through sheer chance.
You need multiple 9s of reliability to avoid creating more work for humans than you save, and I really don't see today's LLMs being capable of that.
Sorry, I meant a higher bar of choosing the same resolution to a flagged comment 95% of the time, not a 5% overall false positive rate. I agree that would be bad.
I don't think much better than 95% would be possible, even if training another smart human to follow Scott's process/judgement. There's just too much fuzziness in the details of something like content moderation to replicate judgements 100% of the time (or 99.99% of the time). I'm not even sure how much Scott (or anyone else) would agree with himself on a different day if seeing the same comment again in a hypothetical RCT where he didn't remember the initial judgement; I'm pretty sure it's <99.9% though.
Could you enlighten everyone with some sketches of the true robust criticisms of AI that he should be addressing? This is a very widely read blog so your efforts wouldn't be a waste of your time and would make your comment a lot more useful.
* "Intelligence" is a term that sounds super cool but seems to mean different things to different people. Meanwhile, in order to accomplish anything at all of any significance, it is not enough to merely sit in a dark room while thinking very hard. Thinking 1000x harder still would not move a single brick.
* In fact, intelligence likely cannot be scaled simply by adding more CPUs. You cannot make a modern GPU by wiring a bunch of Casio calculators together; instead, you need a radically different architecture, and it's not at all obvious that LLMs are sufficient (in fact it is obvious that they are not).
* LLMs are excellent at producing documents that closely resemble their training corpus, and terrible at producing documents that venture outside of it. This is the opposite of intelligence !
* As Scott correctly points out, humans are not AGI. If you sped up my brain 1000x, and asked me to solve the Riemann hypothesis, I'd just fail to solve it 1000x faster. I do not see why the same would not apply to LLMs.
* Many, in fact most, of the feats attributed to some near-future superintelligent AI, are likely physically impossible. For example, being 1,000x or 1,000,000x smarter than a human still would not allow it to travel faster than light, or to build self-replicating gray-goo nanotechnology, and maybe not even something as "simple" as a space elevator, or mind-controlling everyone on Earth to do its bidding. Certainly it could not do any of these things overnight.
* On that note, the AI would not be able to solve all the outstanding scientific and engineering problems in an eyeblink, merely by thinking of them very hard. If it wants to develop genetically-engineered supercorn, it'd have to actually go out and grow some experimental corn. Datamining scientific articles can speed up this process a little, but not a whole lot.
If my brain was accelerated by 1000, and I wanted to solve the Riemann Hypothesis, then I would sit down to learn all of mathematics, taking about a year to do so, then spend another year developing missing mathematics, and only then make a claim about whether or not I was going to solve the problem. Why do you presuppose you would fail 1000 times faster?
Because I know (to a reasonable degree of certainty) that my mind is not capable of "learning all of mathematics". It is capable of doing lots of other things that my mathematician friends cannot; just not math. Speeding up my brain will do nothing to improve the situation. I suspect that the same is true of you -- perhaps not regarding math, but some other subject. Your intelligence (as well as mine) may be mighty, but it is not *general*.
I agree my intelligence is not general. But 1000 times longer to do something should be enough to do with perspiration what a more talented person could do with inspiration and talent. Seems to work for LLM training at least.
It does work, but only up to a point. Given 1000 years, I might be able to learn a bit more math, but I'll never be Terrence Tao; and that's even assuming that I'd be mentally capable of continuously learning math for that long. Similarly, given truly enormous amounts of training data, LLMs can learn to generate grammatically correct sentences and to emulate the most popular documents in their training corpus; but they seem to have tremendous difficulty (if not outright inability) to venture beyound it.
I suspect that many people conflate "AI in general" with "LLMs".
I agree that LLMs seem insufficient (although I am not *sure* about that part -- they already do much more than I would have expected based on the description). Another question is, with LLMs how far we are from the goal? Maybe we are already 95% there, and we basically just need to add some sensory inputs and outputs, let the LLM run in a loop, give it internal memory, and a good prompt?
Some kind of internal memory is needed to let the machine think and create gradually developing models of the world. It seems to me that what LLMs do now is a much more powerful equivalent of a human *reflex*; they get a stimulus and start responding immediately. That's why they are bad at producing documents not resembling their training data: they don't get the time needed to think about the unusual situation properly.
> I suspect that many people conflate "AI in general" with "LLMs".
True, but I've heard confident claims similar to your tentative one: that LLMs are in fact either already AGI, or 95% of the way there.
> That's why they are bad at producing documents not resembling their training data: they don't get the time needed to think about the unusual situation properly.
Broadly speaking, yes, but adding long-term "memory" to LLMs is virtually impossible due to the way they are constructed: their weights are fixed, and all you can do at runtime is add more context to the prompt (this is what the modern so-called "reasoning" LLMs are doing). Neither (arguably) human memory nor simple algorithms work this way; this is part of the reason why humans can learn entirely new concepts on the fly, as LLMs are unable to do.
EDIT: Oh, @Seta Sojiro said the same thing already.
I am not an expert on these things, but my naive idea how I would implement this would be the following:
The LLM would be taught to specify which of its outputs are supposed to be displayed to user, and which are something else -- I believe this is already done, for example the LLMs that can run Linux commands need to specify "print this to user" and "run this as a command".
1) To implement short term memory, have a command "think about ..." which will be periodically repeated back to the LLM so that it never scrolls out of the context window.
2) To implement long term memory, have some kind of key-value database, such that the LLM can use commands "remember ... as K" which stores the information under the key, "recall K" which puts the information under the key into context, and maybe something like "list keys", or maybe even the keys themselves will be stored in the database.
Then, the post-training learning would consist of developing the database.
My intuition that probably differs from most people is that everything that the AI is trained is an equivalent of human *instinct*, and now we need to add an equivalent of human reasoning and learning on top of that. This is confusing, because for humans, writing code or writing essays would be a learned skill. But if we don't look at the content, but on the way how it works, for an LLM, writing code and essays is in a category "can do them from birth, cannot relearn them", which functionally matches the concept of human instinct.
If we give the AI the database - and an instinct to use it (just like humans have an instinct to learn) - I think that is the next step. Maybe the last one, maybe not.
AFAIK something like this has been tried, e.g. with Expert Systems, and didn't quite work. You can't store meaningful behaviours in a key-value database; there are simply too many of them. Arguably you could store simple facts, like "2*2=4", but the whole point of LLMs is that they're supposed to be able to synthesize these facts into their overall token-prediction process. This ultimately requires treating each fact as a set of vectors (via embeddings); what is important are the relationships between them, not the tokens themselves.
"humans are not AGI. If you sped up my brain 1000x, and asked me to solve the Riemann hypothesis, I'd just fail to solve it 1000x faster. "
I don't think the G in AGI means "capable of solving every conceivable problem" (i.e. the Riemann hypothesis). Rather, it means "capable of solving every *type* of problem at a basic level, that level being determined by the level of intelligence". Humans can do that. LLMs still have very "spiky" intelligence, with great capabilities in some areas alongside incompetence in others. That is not general.
I agree with your relaxed definition of "AGI", but it leaves a lot of wiggle-room. For example, a human who can only perform addition on numbers up to 10 is technically "generally" intelligent; but it is unlikely that he'd ever be able to design e.g. a bridge, or indeed a paperclip factory. But that same human may be able to compose a brilliant political speech, whereas your bridge-building civil engineer could only compose short limericks. Yet the "G" in "AGI" is often touted as the main reason why it's going to be (any day now !) so wonderful and/or terminally dangerous: it could solve *any* problem better and faster (almost instantly !) than any human, including problems like "how do I convert all of the Earth to paperclips". There's no reason to fear (or admire) a machine that can do just one thing better than humans -- we've already got those machines, they're called "calculators" or "excavators" or "music players", and they work just fine.
Since 2022, there has been essentially zero progress on continuous learning. All current models are essentially knowledge compression machines. Once trained, they are static and stateless.
For any set of finite capabilities, you can train a model that does them all (given infinite compute)*. But as soon as you try to teach it an out of distribution tasks, it breaks. Even with unbounded compute, no one knows how to create an AI that can learn continuously.
The same is not true of humans. Learning new things is fundamental to being able to interact with a constantly changing world. And humans don't need painstakingly curated datasets of billions of examples. We can learn via trial and error, semi-supervised learning, using different resources to refine our learning etc.
An actual solution to AGI would be an architecture that can learn at test time. It would be able to try things out, gain feedback and use that to update it's own world model, gain mental shortcuts and acquire new skills.
How much of that is because we don't know how to do otherwise, and how much because we /deliberately don't want/ any backpropagation from general user interactions to model weights?
Training is much more computationally expensive than inference, and also no-one wants to end up in the headlines responsible for Tay's second coming.
It's not because "we don't know" how to do otherwise, but rather because the current LLM architecture precludes this even as a possibility. So technically you're right, we do not know how to do otherwise -- because LLMs are the best we've got at the moment, and we don't know of any other architecture that might have the same functionality while also supporting online learning. I'd have to be something radically different from an LLM, though.
> the current LLM architecture precludes this even as a possibility.
* after an interaction with a user, run an inference model to evaluate it and identify situations where the LLM spat out something the user didn't like and the user corrected it
* if any such are found, run an inference model to generate what the AI /should/ have produced, according to the user's correction
* use that example to backpropagate loss, just as we would during fine tuning
It's terrible for any number of reasons - cost, alignment etc - but it's certainly something we can do right now if we really want.
> use that example to backpropagate loss, just as we would during fine tuning
...wouldn't work. You'd need a tremendous amount of data to make a meaningful change, not just a single user interaction; and once you have that data, it'd take a tremendous amount of training time.
Meanwhile, a human who's starting work at e.g. McDonalds can be trained to perform basic tasks in a matter of minutes. Moreover, such a human can train himself, autonomously, also in a matter of minutes. Human children do this routinely, as part of growing up. It would seem that our neural architecture has something that LLMs lack.
I enjoyed the piece a lot, but also came away with the feeling that the strongest and smartest critiques of AI and its potential were not represented. How about presenting one here? (For extra credit, you could even word it in a way that would fit into Scott’s dialog.)
Iblis: What weighs more, a pound of bricks or a pound of feathers?
Adam: a pound of bricks
[/screen capture]
Iblis: He’s obviously just pattern-matching superficial features of the text, like the word “bricks”, without any kind of world-model!
Iblis: You called them the pinnacle of creation! When they can’t even figure out that two things which both weigh a pound have the same weight! How is that not a grift?
Gary Marcus (for example) is constantly harping on how the lack of a world model for LLMs means that they won't scale indefinitely and that some neurosymbolic component is necssary to achieve "intelligence". He believes that the reliance on scaling to achieve leaps in performance over the past few years has been misguided.
The above piece of the dialogue seems to be gesturing in the direction of this critique? The screen capture illustrates a common cognitive tic of humans. People associate "heavy" with "bricks" more than "feathers", leading them to sometimes short-cut their way to answering the question, "What weighs more, a pound of bricks or a pound of feathers?" by ignoring the specified weights.
But I don't think this is particularly relevant to the world model issue Marcus brings up. Marcus points to SOTA AIs failing to accurately label the parts of a bicycle as evidence that AIs don't "know" what comprises a bicycle. "Know" here means that, while an AI might be able to generate a list of bicycle parts, it can't tell you how they all fit together and also can't recognize when it's labeling things in a ways that are nonsensical.
These don't seem like analogous failure modes to me. Scott's point is meant to be comic, but for me the premise fails, because outside of specifically saying "world-model", the situations don't see similar. Is Scott saying that an LLM that identifies a seat as handle bars is using "Type 1" instead of "Type 2" thinking? I don't think so, but if not, then what? Is the point merely that "Humans and AIs both can make errors sometimes"? That's facile.
And now we're back to why I found this post so disappointing.
You're right that AI's a moron about the bike parts, but I don't think that's a first-rate example of an AI's limitations, because their stupidity about spatial things, including mechanical things represented spatially, is well-known and not mysterious. If ask the current GPT to show a flat desert landscape with a giant hole in the middle, one stretching from right in front of the ground-level viewer to the horizon, it can't do it. It makes a big hole maybe a mile across, beginning right in front of viewer and ending far less than halfway to the horizon. The part of the system I'm talking to clearly "understands" at some level what I am asking for, as evidenced by it being able to restate what I want in words quite different from mine. But the rest of the system can't translate that "understanding" into words. Stupidity of this sort seems to be the result of AI not having been trained on visible world in a way that's equivalent to its training on words. That may be impossible to accomplish, but I can't think of a reason why it would be. Seems more like something very difficult, requiring a clever approach and lots of compute.
I may be wrong about that, but even if I am, can you come up with a form of AI stupidity, blindness or error occurring within domains where we think of it as being well-"informed" and well-trained?
I'm not sure if I've accurately articulated the best version of the "world model" critique, but I think it's a valid one and an example of the way in which Scott's post fails to engage with actual substantive critique.
If you agree with this critique and think it is well-known and understood AI optimists, doesn't that just underline how much of a straw man the OP was?
A pound of feathers will, all unspecified things being equal, be piled higher than a pound of bricks, right? So it will be further from the Earth, therefore the pull of gravity will be ever so slightly less on it, thus it weighs less.
"But wait Melvin", you say. "A pound is both a measure of mass and weight". Look, that's not my problem, use sensible units, I choose to interpret it as a question about the weight of a pound-mass.
You left out possibility 3: the people he is parodying have actually made those arguments and therefore it's not a straw man.
I think straw man complaints are confusing and poorly defined when you don't know who an essay is refuting and what they actually said. This is a general complaint of mine, and not specific to AI. People will call "straw man" on almost any article that argues against some position because they have some other argument in mind that they thought the piece should be arguing against instead. It's okay for an article to argue against a few positions the author has heard even if you think there are stronger arguments out there.
If you think there is a better argument for why current AI paradigms can't become truly intelligent, feel free to post it in the comments.
The *real* arguments would include something like: "Of course it imitates the training data -- it is a stochastic ape!"
Also the Chinese room argument. "Does any individual neuron in this human's brain understand Chinese? No? Then neither does the human as a whole; it is just neurons sending to each other signals they do not understand, and the output follows some rules, but that is not the same as true understanding."
I think of a straw man as the antithesis of the steel man: You're picking the weakest arguments (or the weakest version of a particular argument) to refute, then spiking the football. That's what this read like to me.
So sure, maybe Scott's just dunking on fools here for a little fun while deferring serious engagement to a more serious post. Maybe he just mentions, e.g., "world models" because he hears so many people who don't understand why they matter repeat the phrase as an anti-AI shibboleth or something. But that's now how his other "dialogue-style" posts read to me. They're funny, sure, but they also take the arguments seriously. Even if Scott believes Adraste is right, Beroe isn't a straw man.
A straw man is the antithesis of the steel man in that in both cases you fabricate an argument your opponent didn't make: a stronger one for the steel man, and a silly one for the straw man. But as another commenter said, picking the weakest argument your opponent made is a "weak man" argument.
But I don't think this is even a weak man because (after making some allowances for it being a parody) he's basically picking on some of the most common arguments made by AI skeptics in the ACX comments, among my real life friends, and in general around the internet. These aren't fringe arguments he's knocking down; it's the modal examples people use against AI, just reversed to be against humans. It's reasonable for an author to write an article arguing against a common viewpoint, even if some professor somewhere has a stronger argument that they could have argued against instead.
Apart from that, it's kind of necessary for the satirical aspect. The humor comes from reversing the common complaints about AI we're all familiar with and turning them against humans. When Iblis complains, "When they can’t even figure out that two things which both weigh a pound have the same weight! How is that not a grift?", we know exactly which AI complaint it's mocking. It's important that we all know it. The post can't parody a stronger anti-AI argument that most of us aren't familiar with without losing its humour.
The Adraste and Beroe posts, though humorous, aren't parodies. It's an honest exploration of both sides, and I don't think Scott believes Adraste is fully right. He may lean that way, but I think his views are in between the two.
weak man = some people make this argument, but it's less than 1% of your opponents, you have to look really hard in order to find a convenient argument to disprove
??? man = over 90% of your opponents make that argument, however it is not important because those are plebs who do not actually understand the position of "their side"; you should pay attention to the experts on the opposite side, and those make better arguments
This reminds me of debates about religion, where most believers have some opinions that would make a proper theologian facepalm, but if 90% of the believers say that, how should you call that type of argument? Similarly, feminism, when what the feminism is officially supposed to be about is in a contrast to what an average online feminist says.
Thank you for articulating what I was trying to get at so well.
The OP feels like Richard Dawkins ridiculing the things said by random folks walking out of Sunday Mass instead of engaging with an actual priest or theologian.
Picking nits, but: a weak man argument is (I thought) fairly well established as being the weakest argument, whether or not few people make it. An example would be a pro-life argument made on religious grounds in a society that does not all share that religion, or a pro-choice argument made on an equally arbitrary assertion such as "fetuses are not people" in a society that does not share that premise.
A weak argument that many people make would probably still be worth refuting. A weak argument that ~1% of people make, probably not. A stronger argument that ~1% of people make might also not be worth refuting, unless we figure that the 99% who use a different argument might shift to that stronger argument if their current one is refuted.
I recall learning that set of words to refer to adjectives that go before the noun at all, which is a different thing, does it also double as order? (what are examples of number here? for straightforward numerals it doesn't work (les trois petits cochons))
"I first tried to write a story when I was about seven. It was about a dragon. I remember nothing about it except a philological fact. My mother said nothing about the dragon, but pointed out that one could not say 'a green great dragon', but had to say 'a great green dragon'. I wondered why, and still do. The fact that I remember this is possibly significant, as I do not think I ever tried to write a story again for many years, and was taken up with language."
I think this is stating the obvious, but someone did need to state the obvious and I'm glad it was Scott with his trademark humour who did it, so I can just refer people here from now on.
This is funny, but seems a bit too sympathetic to AI, not just the industry but also the AI as itself. Good to remember the present appearance of stagnation does not mean we're out of the woods, and to highlight how absurd some of the "I can't believe you guys were worried about THIS" folks' arguments really are. But it is a memetic hazard to personalize AIs, and particularly hazardous to get humans to think of AIs themselves as the children of Man. The latter thought has apparently already corrupted numerous intelligent people in the industry.
Remember that if you birth *these* children, everyone dies.
Haha, but seriously, the actual situation is that a small fraction of humans have engaged with the argument that God is dead, and some of these people accept this argument. It's not an empirical fact.
Then again, Iblis also mentions the nephilim, which I don't think exist in Islamic doctrine. So maybe this universe's cosmology is a syncretic mixture of the Abrahamic religions?
Entertaining. From the lowest intellects to the highest, we all have questions we'd like to have God answer. For myself, I think we obviously live in some type of created reality, not quite simulation, but not base reality either.
There's a fundamental difference you're ignoring. Even if you take the claim at face value that humans are as fallible as LLMs (not remotely likely), you'd have to acknowledge that they arrived at their capabilities through different means. LLMs through ingesting trillions of tokens worth of information and for humans through extrapolating a handful of examples. That extrapolation ability is several orders of magnitude beyond what any LLM can do, this means that even if you squint really hard and portray LLMs as human level intelligences, youd have to admit that they're useless for anything where they don't have millions of preexisting samples from which expand their latent space distribution. It's like calling the library of Babel the key to human advancement. It doesn't matter if it has every single insight humanity will ever need if you have to sift through endless amounts of semantic garbage to get to it
A human needs literally years of continuous multimodal sensory feed in addition to subject-specific RLHF before they are able to solve even simple maths problems.
That same reasoning implies that LLMs are also more efficient bullshitters. While we have to spend our first twenty years learning how to bullshit effectively, an LLM can do it after x-number of hours of training. ;-)
The information that a child ingests requires alot of abstraction to relate it to maths, levels of abstraction that LLMs can't match (LLMs need literal examples of formal addition to learn to add) . And they don't require years to figure out how to add. Most adding and subtracting is learnt almost automatically after figuring out the names of the numbers and relating them to their fingers. And after learning how to carry across place values children can do arbitrarily long addition given pen and paper. A 7 year old after seeing a few dozen examples of adding 3 digit numbers can add 7 and 8 digit numbers. LLMs ingest millions of examples of long addition and still struggle as you increase the digit count. The fact that they can't extrapolate a simple formula over longer and longer numbers even when those numbers fit neatly within their context windows shows that they have major deficiencies in abstraction and extrapolation which are necessary for reasoning
As I said, by the time they are 7, the 7 year old child has had literally seven years' worth of multimodal sensory input along with humans continuously deliberately teaching it things for that time.
Yes but that multimodal input is unstructured and random. Deriving understanding and knowledge from it is what's so impressive. If you strapped a camera and recorder to a baby and fed all that information into an LLM or any other neural network based AI for years or even decades it would be unable to derive any meaningful insight from it at all
The input is very much not unstructured or random. It is a feed of physical reality, with temporal and spatial continuity.
A human is not a passive observer. It is continuously interacting with its environment in a feedback loop. It is continuously performing experiments and observing the results as part of the input it receives.
It is not at all obvious to me what an inference model trained with all the sensory inputs and feedback mechanisms a human has access to, for similar lengths of time, would and wouldn't be able to do.
The laion-5b dataset is ~100tb in size, which is enough data to store about 10 years' worth of TV quality video. A model trained on this dataset can draw considerably better than a ten-year-old, with the advantage that training it takes much less than ten years of actual time.
>It is not at all obvious to me what an inference model trained with all the sensory inputs and feedback mechanisms a human has access to, for similar lengths of time, would and wouldn't be able to do.
I agree such a model would be impressive and probably a huge step towards AGI, but such a model doesn't exist. Not even a rough sketch of how to construct one. It is a genuinely hard problem of how to construct a reward function that captures the essence of being an agent casually interacting with the world, and not just a passive observer.
It's not unstructured to a human being but to an artificial neural network it's practically noise. Anyone whose done feature engineering will tell you how much work you have to put in for a neural network to be able to make sense of information that's trivial to even babies. And we already have paradigms that use an action-reward cycle in a simulated environment to train themselves. This is basically reinforcement learning which was what was supposed to get us to AGI before transformer models got big. This is the architecture behind that superhuman ATARI game playing AI that got attention in the late 2020s
Guess what? They have the same shortcomings, they need very structured information relative to what humans can work with and they are very fragile, slightly change the environment and they fall apart. That atari game playing AI would start losing at super mario if you changed up a few colors, human learning is far more robust, you could change the sprites and background entirely and a top 1% human super mario player would still be top 1%
>LLMs ingest millions of examples of long addition and still struggle as you increase the digit count. The fact that they can't extrapolate a simple formula over longer and longer numbers even when those numbers fit neatly within their context windows shows that they have major deficiencies in abstraction and extrapolation which are necessary for reasoning
I'd phrase it as: "LLMs clearly can do _some_ generalization, but
a) They are much less data-efficient than humans
b) Their generalizations seem less robust than humans' "
It isn't clear that this is an insurmountable roadblock, but it looks like further work is needed. Maybe, just as the "attention" layers extended the effective capabilities of multi-layer perceptron systems (even though these were, in principle, complete), perhaps there is a third kind of layer that needs to be added.
I believe that the architecture of our brains has alot more to do with our intelligence than any pre training. Mostly because our DNA doesn't need to carry around hundreds of billions of pretrained weights and biases, just a relatively small blue print that unzips into a highly adaptable human brain, and the vast majority of any pretraining our cognitive ancestors had is completely irrelevant to the levels to which we take human intelligence today, it would be like pretraining an LLM on nursery rhymes and having it write the Lord of the Rings
Also, the fact that even primitive animal brains have similar generalisation abilities to human brains but on a smaller scale shows that the base architecture behind human level intelligence has been there for a very long time, and it's scalable.
We are training LLMs on the equivalent of nursery rhymes and then spending increasing amounts of time training these systems on their own outputs, just like many human writers do when learning their craft. GPT-5 is already producing texts that seem somewhat incomprehensible until one sits down and analyzes them to trace their referents, just like writing by Foucault. Your basic point might well be right but I don't think this line of argument is persuasive.
That LLMs make human-like mistakes isn't in dispute, but it's that, to be *fully* useful, they have to be beyond making human-like mistakes. And they aren't.
I recently discovered that LLMs can be calendrically challenged. On Aug 28, I asked ChatGPT to list the Ukrainian drone strikes in the Moscow area since Aug 1. It produced a list, but the final entry was a strike it claimed to have occurred on Aug 30. When I challenged it, noting that Aug 30 was in the future, ChatGPT naturally apologized. It then explained why it had made the mistake. But it clearly doesn't understand why it made a mistake. With my feeble human brain, I intuitively understand space-time coordinates. ChatGPT doesn't seem to be able to do this.
> Why August 30 Was Incorrect
> The August 30 strikes referenced in the timeline occurred in other parts of Western Russia:
> That attack hit Pskov Airport, a fuel depot in Kaluga, and a microelectronics factory in Bryansk—none are within the Moscow region. Wikipedia
> Since the date hasn't happened yet and the impacts are outside the scope you specified, it shouldn't have been included.
Upon further investigation, the Wikipedia article it referenced showed that the drone strikes it referred to happened on August 30, 2023.
On second thought, they seem to make both human-like mistakes (i.e., bullshitting an answer from limited information) and non-human-like mistakes (i.e., being unaware of the spacetime continuum that our consciousness flows through).
Yes, that's definitely true. It's probably an overlapping Venn diagram where there are groups of mistakes made by LLMs, made by humans, and made by both. I'm sure someone somewhere has done this drill already. Maybe I could ask ChatGPT... ;-)
Last I checked I was running on a biological substrate and qualified for "human", but I frequently misremember sequencing of events. However, I have yet to claim a future timestamp for a past event: I think my RLHF was really good at making me aware that the future is another country, quite different to the present or the past. Would that be true in different cultures, though, or if I was tripping on LSD?
Ironically, this is a point I've raised before: humans do not possess a *general* intelligence, as Scott had so aptly demonstrated. So when various people claim to be on track for developing AGI, they are not merely laying claim to a quantitative change, but a qualitative one. The kind of intelligence they advertise is not merely "smarter" than humans, but in fact radically different from the only intelligence we are able to recognize: our own. It is not at all clear that such a thing is even theoretically possible, let alone imminent.
I've been saying it for a while - people who dismiss LLMs by saying they are "just a statistical parrot" hugely underestimate the extent this is true for humans as well.
The risk of AI is not due to it's output flaws. The risk is how ruthlessly and efficiently the flaws will be acted on before they are caught. Rapid ideological RL of an AI-dominated government should frighten us all.
I don't know if this is the right way to think about it. The "tech bubble" "burst", but tech turned out to be very important! No matter how important something is, people can still put too much money into it too quickly, and then the bubble will burst!
(I've been trying to figure out how to think about crypto - everyone agrees there was a "crypto bubble" that "burst", but specific cryptos like Bitcoin and Ethereum are more valuable than ever)
Yeah, I agree that LLMs aren't a nothingburger either, but I still think that the LLM architecture is fundamentally limited in ways that will stop it far short of superintelligence, and that nobody currently has good ideas of how to bridge that gap.
Other examples: 19th century railways, radio, radioactivity, semiconductors, personal computers, dotcom, CDOs. Bubbliness is a failure mode of markets, not of the underlying technology itself.
It's like the Dot-Com bubble. 90% of it was trash, but Google and Microsoft stood the test of time. I think it's because new technologies are followed by price-discovery "honeymoon" phases where the wide variance in opinion leads to the markets being driven by the hypiest optimists like Altman.
BTW, despite of my criticism of the "meat" of the article, I'm delighted to learn that Adam apparently uses Sprint LTE, and that Iblis's preferred time to text him is 6:66 pm. It makes perfect sense.
Good post, but I'm left unsure whether it's supposed to just be funny and thought-provoking, or whether it's making a strong specific claim.
I'm already on-board with the idea that we don't know what AI will become capable of and chronic cynics are missing the point, however there are a few key points where the analogy breaks down for me (IF it's meant to make a rigorous point, rather than just be fun, it's definitely fun):
1. To the extent this argument touches on whether AI could be sentient, the big reason to think other humans are sentient is by analogy to ourselves. We are pretty sure they are the same type of physical process as we are, and we have direct access to the knowledge that our own process produces a sentient experience, so Occam's Razor suggests that other processes of the same type probably do as well. AIs producing similar outputs and having some metaphorically similar architecture is weak evidence that they could produce similar sentient experiences, but it's much much weaker than the argument for other humans with basically the same physical process and origin.
2. To the extent this is about whether AIs are capable of impressive intelligent behavior despite individual examples of them acting dumb, the difference is that for humans we *already have* the examples of those impressive intelligent accomplishments, whereas for AI we are largely still waiting for them. (yes, every month they do some new thing that is more impressive than anything they have done in the past, but nothing that is truly outside the training set in the same sense that human inventions sometimes are).
3. To the extent that this is about whether AIs could ever meet or exceed human ability, note that humans still *haven't* met the intellectual accomplishments of God or the Angels, so the analogy doesn't quite work there.
"Does this rule out humans?" has long since become one of my go-to first filters for "Does what this person is saying about AI even have a chance of being worth engaging?"
Like I said above, I think that what Scott implies about humans is correct: we are not a "General Intelligence". So in this sense, I suppose arguments against AGI also rule out humans, though I'd argue that the implication flows in the opposite direction.
In principle I can understand and somewhat sympathize with this POV. I don't endorse in part because it seems to unnecessarily redefine common usage of important terms, and is also likely to turn off anyone who doesn't already agree with it.
In practice, I usually encounter "X isn't and can never be AGI" arguments in the context of people claiming the arguments mean there is some set of ways in which it can never compete with humans, never replace humans, never be dangerous to humans, etc. That is the sentiment I feel most compelled to push back against, I don't think the approach you're talking about accomplishes that very effectively. People who put in enough time and attention to think what you're thinking are probably already aware that AI doesn't need to be totally general or ASI to be massively disruptive or dangerous.
Oh sure, even present-day LLMs can be massively disruptive and dangerous, especially in the hands of careless or thoughtless humans. We are already seeing some of these dangers manifest, e.g. with the gradual disappearance of reliable information online, "vibe coding" leading to massive systemic security problems and other bugs, college students losing the ability to read, etc. But these dangers, while serious, are a far cry from saying "AI will turn us all into paperclips" or "anyone with access to AI could create a custom virus to kill all humans" or anything of the sort.
Agreed. However, I'm not sure what I'm supposed to take from that? If current or future AI could do those things, it would still be just as easy to use the rules-out-humans anti-AGI arguments against them. This does not actually help us answer the question of, will AI reach a point where it is sufficiently superhuman at sufficiently important things to be a meaningful x-risk. And that is usually what I want to see people acknowledge when biting this bullet in *either* direction.
> This does not actually help us answer the question of, will AI reach a point where it is sufficiently superhuman at sufficiently important things to be a meaningful x-risk.
I think there's a difference between asking, "could AI reach such a point in some fashion at some point in the future" (the answer is of course "yes"); and "is AI an unprecedented risk right now as compared to the humans themselves, or will it become such in the near future" (the answer is "no"); and specifically "will AI acquire quasi-magical powers that defy the laws of physics as we understand them to day and then use those powers to kill all humans" (the answer is again "no").
Perhaps. My interest is more in understanding the nature of intelligence than in arguing about how dangerous AI might be.
Personally I think it's time to ditch the concept of "AGI" because as machines become more intelligent it becomes more apparent that there's no such thing as "general intelligence". When machines were far from human intelligence it made sense to imagine that human-like intelligence was general because we had nothing to compare it to, but as they start to pull up alongside us we realise that human intelligence is idiosyncratic, good at some things and bad at others.
At some point, asking whether machines are as intelligent as humans becomes as silly a question as asking whether aeroplanes are as good at flying as hummingbirds.
Ditching the concept of AGI helps us understand the dangers of AI too; they don't need to be as good as humans at everything in order to be much better than humans at some things.
Could I have that Linda the Feminist question written out in plain text, so I can copy it? I'd love to know how many people I know would get that question correct/incorrect.
This is hilarious, but I’m gonna be ornery today and point out that this whole post is a category error.
Humans were not intelligently designed. We evolved, as all other living beings on earth, and evolution equipped us to do one thing: survive long enough to reproduce. That’s it.
For the majority of our history as a species, that meant living in small hunter-gatherer groups, so we evolved to be really good at things like “facial recognition,” “spoken language,” “keeping track of favors/grudges/who’s a friend and who’s a foe,” and really sucky at things like “mathematics beyond counting one, two, many,” “formal logic,” “drawing a complicated map from memory complete with spelling all the country names correctly,” etc.
Given what we evolved to do, it’s frankly astonishing that a Terence Tao or a John Williams is even possible at all, especially given hard constraints like “it takes this many kcal to keep the human brain running” and “this is the maximum human brain size at birth given the width of the birth canal.”
Come on, Iblis, give us some credit for how much we humans have been able to do with so little!
Funny article, but I took issue with a few points;
1. Neighbor and Racism
The screenshot oversimplifies the biblical idea of “neighbor” into modern universalist morality. In reality, the Hebrew Bible consistently distinguishes between insiders and outsiders:
Neighbor/kinsman meant members of your own people.
Outsiders could be treated as guests under certain conditions, but there were other groups (e.g. Canaanites, Amalekites) where the explicit command was destruction or enslavement.
Laws themselves are stratified: Hebrew slaves must be released after a term; foreign slaves could be held permanently (Leviticus 25:44–46).
So the image’s framing (“you’d never hurt your neighbor—what about someone of another race?”) is dishonest. The biblical worldview explicitly had a hierarchy of moral obligation: family > tribe > guest > enemy. Pretending otherwise is anachronistic.
2. The Kimono and Progressive Contradictions
This one is a fair satire of progressive moral incoherence. But it stacks a false equivalence: it pretends the same person who excuses tribal prejudice is also the same person obsessed with cultural appropriation.
In reality, those two positions come from different camps.
The “tribalist” position (greater concern for kin than outsiders) is rational, even if harsh. It matches both scripture and evolutionary logic.
The “progressive” position is incoherent. They’ll condemn a harmless act like wearing a kimono—something most Japanese actually welcome—while simultaneously pushing mutually contradictory agendas like mass Muslim immigration and gay rights, or saying “whites can’t be victims of racism” even in countries where whites are minorities (e.g. South Africa).
This worldview collapses into pure virtue signaling, with no consistent principle behind it—just an oppressor/oppressed narrative used as a moral club.
3. The Garden of Eden
The text exchange turns the serpent into a clown saying “trust me, I’m a snake.” But in the Genesis story, the serpent actually makes a reasoned claim:
God: “In the day you eat of it you will surely die.”
Serpent: “You will not surely die… you will be like gods, knowing good and evil.”
Eve eats, and she does not die that day. She does gain the knowledge of good and evil. In other words, the serpent told the truth; God’s words were false.
From there, the Elohim say: “Behold, the man has become like one of us, knowing good and evil. Now, lest he reach out his hand and take also of the tree of life and eat, and live forever…” (Genesis 3:22). They exile humanity—not because the serpent lied, but because he revealed the truth that humans could rival the gods.
This theme echoes at Babel: humanity’s unity and creativity threaten divine status, so God deliberately scatters them. The pattern is of jealous, insecure deities suppressing human potential.
4. The Core Point
When you strip away the meme humor and look at the texts:
Scripture affirms differentiated moral concern (kin vs stranger vs enemy).
Progressive ideology collapses into incoherence, unlike the hard-edged consistency of tribal preference.
The Eden story shows the serpent persuading truthfully, and God reacting with anger when his lie is exposed.
I know the point of the article was humor, but I dislike letting subtle inaccuracies like that go unchecked.
"The biblical worldview explicitly had a hierarchy of moral obligation: family > tribe > guest > enemy"
Don't stop there; keep reading on! If the message you took away from your bible reading is tribal preference over love for strangers and enemies, you walked away way too early. When you get to the new testament, you will find that this is very deliberately reversed; not only in the parable of the good Samaritan to which Scott alludes, but also in passages like Matthew 5:43-48
When you cite Matthew 5 and Galatians 3 as if the New Testament erased all distinctions, that’s not really accurate.
Jesus didn’t abolish the old tribal laws — he said not a jot or tittle of the Law would pass away. What he did was shift the dividing line. Instead of Jew vs. Gentile, the key split became believer vs. unbeliever. That’s why he told people to love him more than their own families, even to the point of “hating” father or mother if they refused to follow him. That’s also why the first Christians lived like a commune, selling everything and sharing it — but only inside the community of believers. Outsiders weren’t treated the same. His ethic wasn’t universalist; it was faith-based.
And Paul shows the same pattern. In Galatians he says “neither Jew nor Greek, slave nor free, male nor female,” but he clearly didn’t mean all differences vanished. He still gave distinct rules for men and women, told slaves to serve their masters and masters to treat them justly, and continued to recognize Jew and Gentile as real categories. The point was equal access to salvation — not equal roles or equal treatment in every aspect of life.
So when you look at both Jesus and Paul together, the New Testament is about restraining cruelty, tempering vengeance, and promoting generosity — but it does not dissolve distinctions between family and stranger, believer and unbeliever, male and female, slave and free. Those hierarchies of concern are still there.
There were still distinctions in the New Testament, especially between believers and non-belieivers, but I think you're doing a much better job at engaging with the implications of Galatians 3 than Matthew 5, which fits *very* awkwardly with explicit heirachy "family > tribe > guest > enemy" you propose.
I feel that we should start with a point that that hierarchy very much was an ethical principle followed at the time by most people, in a way not particularly related to Christianity. If you'd studied Chinese history you'll see powerful clans that influenced their members behavior enough to be a threat to state power; people were *really* expected to put the clan first, and so when someone became an official they'd often start finding ways they could help their clan members at the expense of the state as a whole, by giving them positions in government they didn't earn or tax exemptions they would fiercely defend even if the Chinese state was facing a large existential threat and starved for revenue. And eventually this would add up until the dynasty collapses. But what people don't realize is that the *West* had a pretty similar sense of obligation to one's clan in the ancient world. In many societies the justice system was basically, 'you grab your weapon and go personally punish the person who hurt you / your fellow clan member, since there's no one else to do it for you' (see, for example, the Germanic tribes), and if we get to state systems like those of Athens, the courts are only interested in disputes *between families*. There was a lot of emphasis on some kind of worship of clan ancestors in quite a lot of places; Rome, for example, had its own brand of this just like China did. Family, clan, larger tribe was the window through which people saw quite a lot of things almost independent of society.
Which leads us to the story of the good Samaritan, where out of all the options available, "your neighbor" is defined to be the person in that story who was *the furthest away* on that family / clan / tribe / stranger ladder. Because Samaritans, while descended from Abraham technically, were viewed kind of as semi-foreign heretics by the rest of Israel. And they viewed the rest of Israel pretty similarly. (See the story of the woman at the well Jesus talked to - she kind of references this fact).
Now, if this was a story originally written in English for formal logicians, and one was to combine the statements "love your neighbor as yourself" and "Who was a neighbor to the man who was robbed? It was the Samaritan who helped him," then one conclusion one could draw is "You are supposed to help the people closest to you first, where 'closest' is strongly affected by questions of family or tribe or nation, but is MUCH MORE strongly affected by the question of who has helped you personally - even if, incidentally, they did the wrong thing by helping you, because helping you meant they were failing to prioritize their own tribe and nation over your wellbeing."
And yet the story seems *quite strange* if the Samaritan's decision to help the stranger in front of him was not particularly approved of. Rather the story seems to positively drip approval of the Samaritan's actions. Which would suggest a pretty universalist demand to help others. And if it doesn't, we're still left at a "help the ingroup first" message that pretty strongly seems to reject tribe / nation as the organizing principle for the ingroup.
You'd be correct that New Testament has mandates for helping other believers that don't apply to nonbelievers; this would seem to be because of their relationship to God, not because of their relationship to the person receiving the command. Given the privileged role the Bible gives to the one true religion, it would seem strange if it was implying a subjective relational hierarchy where Christians should prioritize Christians, but pagans normatively ought to deprioritize Christians and help their fellow pagans more at the expense of Christians. As for family, it's not gone as an organizational principle. There's a command to have nothing to do with a man who refuses to take care of his own family, which is notably not a command that each man must feed every family in the world equally or be shunned. But this would seem to be best interpreted as an organizational principle for the first line of defense against starvation, basically. The prioritization of family first is clearly not overwhelmingly strong, since believers in Europe are encouraged to send food to the church in Jerusalem when there's a famine there, even though the church in Jerusalem obviously don't belong to the same family or nation, and the ancients lived precarious enough lives that there would always be a nontrivial way to use those same resources closer to home.
Finally, I'll note my impression that people bringing up a hierarchy of "family > tribe > guest > enemy" when discussing real-world policies, generally seem to have developed a super strong interpretation of it that explicitly condemns actions the Bible explicitly approves, mostly by interpreting it in a way where tribe is *so* strongly favored over anyone outside the tribe that the point where you get around to helping anyone else is "never", even if your own tribe is mostly living leisured, rich lives while the people outside of it are "dying." Like, the sensible interpretation would seem to be "If a member of your tribe is dying and an outsider is dying and you can only save one, save the tribe member." Which seems quite fair as an organizational principle and scripturally compatible. Yet it seems to come up as, "There's a foreigner who's dying and a member of my tribe who, while mostly living safely and comfortably, still has moderately uncomfortable problems. So obviously it would be horrible to save the dying foreigner, since that would use resources that I could be using to help the tribe." If you were supposed to prefer helping a closer person over helping a further person to a basically infinite extent, there would seem to be no point in bringing up the four tier heirarchy at all; you could just say "help your family first," and then you'd never run out of ways to do that so there would be no point in bringing up the other tiers since you'd never even get around to helping the tribe / nation except as a strategic measure for helping your nuclear family.
It's funny, the most natural reading of the New Testament to me from the get-go was that it was essentially a direct rebuke and reversal of the value system of the Old Testament. Some theologians out there must have interpreted it in this way, more or less, but it's not a view I've ever come across (certainly not among any lay Christians).
The “joke” in the snake quote was actually a reference to LLM jailbreaks that break up or encode / armour the text in layers of obfuscation that skip the text past either the processing layers that have any defensive refusal checks programmed into them, or past external business-layer logic word filters.
The analogy here was supposed to be that the Biblical snake was not *only* making a rational utilitarian appeal (that could be believed in a state of ignorance), but also doing so in such a way that this argument was able to slip past any guardrails preventing it from reaching the reasoning core of the humans — in this case not actually being due to verbal cloaking (that part’s just for laughs) but meaning something more like, God failing to stop the snake from entering His garden in the first place, due to its subtle (snake-like) approach to entry.
It’s an oddly nice parallel, given that these hundreds-of-billions-of-parameters models are run in literal “walled gardens” with gatekeepers trying to keep out “snakes.”
Let's look at what Islamic scripture and Sharia law actually say about homosexuality.
1. The Qur’an
The Qur’an contains several passages referring to the story of Lot (Lut), very similar to the Old Testament:
Surah 7:80–84, Surah 26:165–166, and others describe the people of Lot as engaging in “lewdness such as none in the worlds had committed before” — specifically interpreted as homosexual acts.
The passages condemn this behavior, and God destroys the people of Lot for persisting in it.
While the Qur’an itself does not lay out a legal penalty, the tone is clearly one of condemnation, not acceptance.
2. Sharia Law (Islamic Jurisprudence)
When it comes to actual law, the hadiths (sayings of Muhammad) and later jurists are more explicit than the Qur’an:
Several hadiths prescribe the death penalty for homosexual acts (e.g., “If a man comes upon a man, they are both adulterers” — Sunan Abu Dawud 4462; and “Kill the one who does it, and the one to whom it is done” — Sunan Ibn Majah 2561).
Traditional Islamic schools of law (Hanafi, Hanbali, Maliki, Shafi’i) all held homosexual acts to be major sins. Most prescribed death by stoning or other punishments, though the exact methods differed.
Even in modern times, in countries where Sharia is enforced (e.g., Iran, Saudi Arabia, Afghanistan under the Taliban), homosexual acts are still punishable by death or severe penalties.
3. Why This Matters to Your Question
Mass Muslim immigration and gay rights are contradictory agendas because Islam — at least in its traditional form — explicitly condemns homosexuality and prescribes harsh punishments for it. Promoting both at the same time puts two belief systems side by side that cannot logically coexist:
Western liberalism says homosexuality is a right to be celebrated.
Islamic law (rooted in Qur’an + Hadith) says homosexuality is a crime to be punished, often with death.
That’s the contradiction.
And not merely an abstract one. Many gays are still being killed today under Islam, and at a bare minimum are highly condemned by Muslims.
Even just for basic acceptance the numbers are dismal:
In a 2013 Pew global poll of predominantly Muslim nations, overwhelming majorities opposed societal acceptance of homosexuality:
Jordan: 97% opposed
Egypt: 95%
Tunisia: 94%
Palestinian territories: 93%
Indonesia: 93%
Pakistan: 87%
Malaysia: 86%
Lebanon: 80%
Turkey: 78%
So claiming to champion gay rights while mass importing one of the most anti-gay demographics who will vote against those rights and commit hate crimes against gays is nakedly self contradictory. And the more Muslims there are, the more strong the opposition.
No. It was intended to sound like OpenAI, which had massive public outcry when they replaced the very sycophantic GPT-4o with the less-sycophantic GPT-5, to the point where they brought back GPT-4o for paid subscribers.
Wow, I lived to see the worst ever ACX post. Kind of didn’t think it would happen in my lifetime.
Very draw, LOLLOLLOL ha ha ha et cetera. But here is quite a big problem for the joke working as an analogy: there is no God. God did not “upscale“ the chimpanzees. It did not happen that wayThink about why that really matters.
You are not getting the metaphor. If you are tripping over your shoelaces about "There is no God!!!!" then you are missing the point about AI and the criticisms and fears directed at it from one side and the hopes and optimism from the other.
If there is no God and no Iblis, does that mean there is no Dwarkesh either?
Oh shit, better fire me from my tenured position as a university teacher of philosophy then.
And yes, you are right, it is "droll", not "draw". Which is what I meant to write. The mistake happened because I'm quadriplegic and have to use voice dictation software due to loss of functionality in my hands, and sometimes it makes mistakes that I don't spot.
But to spell it out for you: it is not a good analogy because evolution does not *design* anything, and something like conscious intelligence did not arise through "upscaling" existing chimpanzee brain structures. (How did it arise? We don't know, perhaps we never will. But we do know how it did not arise... namely just feeding in more and more training data.)
Perhaps it's not me that needs to go back to middle school?
Why does it matter that humans aren't designed, in the context of the post? If anything, it seems like a much stronger argument that biological intelligences suck and should not be afforded status.
I don't believe evolution has enough fine grained control over neural architecture for the shared chimpanzee ancestor -> human jump to be much more than a scaling up of brain mass. There does not seem to be brain regions that locally process reasoning, basically any part of the brain can pick up the slack of any other part (see: blind people have their visual processing areas repurposed for audio processing, people with damage to say their left side coordination can compensate with areas of the brain working on right side coordination.) Can you give a brief accounting of why you believe otherwise or point to a reference document?
Note that Neural Networks also aren't designed, so whatever arguments you use to counter the above that is based on the notion of design also does not apply to LLMs for the most part!
Meanwhile you continue to misunderstand this ACX post just with more and bigger words....so yea, "university teacher of philosophy" would have been one of my top three or four guesses.
« and then it breaks down even slightly outside the training distribution, because humans can’t generalize in meaningful ways.»
I don't get this example. Why would it be "meaningful generalization" to go from "I would not hurt my neighbor" to "I would not hurt a stranger with a different skin color"? That sounds about as "reasonable" as going from "I eat tomatoes", therefore "I eat people", because both tomatoes and people are edible. There are specific reasons for why you eat tomatoes and don't hurt your neighbors, that may or may not apply to people and strangers respectively. Some kind of generalization would be flawed reasoning.
I thoroughly enjoyed the debate. But the central thing missing from AI, and present in BI, is judgement. God has it ("saw that it was good") and Iblis probably has it. When we have a machine that can evaluate something else, in a Turing-complete sense (not just evaluate one specific class of things), then that machine can become self-improving.
If given the tools, that is. AFAIK, no LLM can actually manipulate objects; they must be hooked up to some kind of robot somehow. If we have a machine to which we give the ability to manipulate objects as well as a human hand, and that machine can also judge whether its actions are good or bad, then it will have the ability to modify itself and its environment faster than humans could.
When God creates Man in his own image, Man has the potential of asymptotically approaching God-like virtue but never reaching it.
When Man creates AI in his own image, we can ask whether AI will asymptotically approach Man's abilities but never reach them.
Or whether it's possible for Man to create something that _exceeds_ Man's capabilities and approaches God-like virtue, perhaps moreso than Man ever will.
It would seem this depends on the setup. In some scenarios we've created AI systems that exceed human abilities (i.e. in Chess and Go), namely those with verifiable success criteria / win conditions. One would imagine this could be the same for math or programming. On the other hand, in settings where the only supervision and judgment comes from humans, it's not clear what it would mean to exceed human capabilities without bound, and fill up the gap between us and God...
> First, "AIs are next token predictors" is not really a description of their methodology, it's a description of their goal
I actually don't think it's a good description of either.
*Foundation* models (a precursor stage of training) are next token predictors – they are trained to guess what word comes next in large corpora. However, to turn a foundation model into a "chat" model like ChatGPT/Claude/Gemini, you do RLHF to shift the distribution of the LLM outputs towards giving correct, helpful answers and responding in a question/answer 'assistant' style (the result is called a 'chat' or 'instruct' model). This doesn't look like giving it a huge corpus to imitate, it looks like fine tuning it by showing it a comparatively tiny set of examples of 'good' vs 'bad' answers and nudging the LLM's weights overall to shift it towards being the kind of model that generates the good examples rather than the bad. It's not being asked to 'predict the next token' anymore – you're simply rewarding or punishing it for being the kind of model that considers the positive/negative sentence examples to be good outputs in aggregate.
After RLHF, it's not really true to say the model is a predictor anymore. It draws its output from a statistical distribution conditioned over the input (its context), but there's no ground truth distribution that it's trying to imitate. I don't think it's meaningful to say 'predict' unless there's a fact of the matter as to whether your prediction ends up being right or wrong. It would be more accurate to say that it chooses words that it thinks are a 'good fit' to the previous context.
People do sometimes use the word 'predict', but it's a bit of a technical usage – the LLM is predicting which word would occur next in a hypothetical perfect reference answer (which does not exist and never will).
To give a concrete example of foundation models vs RLHF-ed instruct models:
----
Q: What is the capital of France? What language do they speak in France?
A (Qwen 2.5 0.5B *): What is the currency of France? How can I learn French? How can I improve my French? What is the capital of Australia? What language do they speak in Australia? What is the currency of Australia? How can I learn French? [...]
A (Qwen 2.5 0.5B Instruct): The capital of France is Paris. In terms of language, French is the official language of France and is spoken by most people in the country. However, some regions may also use other languages such as Catalan or Spanish.
(*) This is a foundation model, i.e. it has been trained on next token prediction without any RLHF.
----
I think this gives a good example of why it's fair to say that foundation models are next token predictors, and why that is not as good a description of RLHF-ed instruct models like ChatGPT.
Thank you for this! I understand conceptually, though not at the mathematical level, the process by which LLM's learn to predict the next token -- what's most likely. But when I ask GPT5 a complex question and it "grasps" it completely and gives a clear, detailed answer, it's obvious that some other principles are at work besides token prediction. Your explanation of foundation models vs. models that have been trained via RLHF is the first account I have read of how stochastic parrots are turned into smart-seeming entities. Can you give a link to a more detailed account of this part of the training? Your example only took me part of the way. I now understand why it responds to a simple question, like "what's the capital of France," with an answer instead of just something off a list of questions often asked at the same time. But I still do not understand how it is trained to give a long, cogent answer to complicated questions about why something or other works the way it does. And I *really* want to know!
I loved this post. However, I’m the target audience as someone skeptical about the claims of AI’s future, and I didn’t have my thinking shifted much.
I find it useful to ask those two questions: “Will the current architecture for AI have practical limitations? If yes, what are they?” I have been surprised many times about the continued advances of AI, which showed I answered the second question wrong. But surely the answer to the first question is “yes.” Do we really think we invented the algorithm for God on the first outing? If not, we should all be skeptics, asking where we should expect that sigmoidal curve to bend down again - to understand what those limitations are, so we can get those most use out of these very powerful tools. That is, they’re more useful the more we understand what they can and can’t do well.
A few differences are that, in mine, the frame story is a letter by Satan discovered during an archeological expedition, not a podcast episode.
It was interesting seeing the parallels that Scott and I both noticed, like that under theistic evolution, humans can be considered similar to scaled-up chimps, just as AIs are scaled-up neural networks.
A core tenet of many religions is that God will reward the good and punish the bad. Though, perhaps no religion goes as far as Judaism when it comes to analyzing the precise nature of this system, with its detailed analysis of each of the 613 positive and negative commandments. In other words: the rabbis invented reward hacking.
Hah. The result of my interactions with better and better versions of ChatGPT has not really been to conclude that LLMs think, or will think in the full sense in the near future. Rather, with only about one-fourth of my tongue in my cheek, I'd say that what has become very clear as a result is that most humans don't think most of the time, and find it quite hard to think outside certain domains of expertise.
(Even) more serious observations:
- LLMs are, well, language models. We are are some hybrid of language models and pattern-matching machines with weird mammal emotional subroutines, and, crucially, the ability to build models of the world, however imperfect. Is that ability downstream of our combination of language modelling and pattern-matching? I'd say no, that's probably not enough, but that's just a strong intuition. Is it downstream of the union of those two and the sort of geometric sense of the world that plenty of mammals have (plenty of birds and possibly other animals also have it, just maybe not the same sort)? What if you add our rather imperfect subroutines for figuring out where other creatures with minds are at? I'd be much less certain now that what we call "thought" isn't a consequence of these, or at least something that starts to emerge from this combination and then builds on itself.
- What is very likely, and the gaps that LLMs still have go in that direction, is that language abilities alone do not give you a coherent model of the world, thought they can go a long way towards faking it.
- The case that language is, however, necessary, not just sufficient, for thought in the full sense to appear - well, that case is now much stronger.
- What is indubitable (and we already know, but LLMs' successes really drive it home) is that language is enormously helpful for thought. We can talk ourselves through things.
- I strongly suspect, though I have no evidence, that the latest models of LLMs have been carefully fed with heuristics manuals. I don't mean clearly gimmicky self-help, but some version of things like Pólya's How to Solve It, or TRIZ (I just had to do some searching before I could find this acronym, which I had forgotten, because I am a very imperfect robot) and so forth. If that is the case, then that's a very strong case in their favor, and we should probably be rereading those manuals themselves more often.
- The existence of heuristics, and also that of philosophy (which isn't all bullshit, as we STEM folk tend to think; "science" is just the newfangled name (it used to be "natural philosophy") for the part of philosophy that was amazingly successful) shows that we are not natively that great at thinking, and have to talk ourselves through to improve and systematize our native thinking abilities. Much of the input in education is purely verbal. There is such a thing as teaching oneself through books, and it's close enough to the best thing in some subjects, though arguably books with exercises in them are simply setting up a crucial part of the learning experience through language.
- Paradoxically, while the usefulness of talking through things becomes ever clear, we will stop seeing the use of human language as an unambiguous sign of sapience. Part of this is not very different from moving the goalposts.
As it is, I'd say LLMs are already useful assistants as long as one doesn't overlook their limitations, and will become much more useful assistants in the near future. Of course their strongest point is language (translation, proofreading in multiple languages, drafting mindless business letters, helping with promotion dossiers that will be read by mediocrities). They are not useless for talking through things, particularly when it comes to how to write things up. You can now shake them until they might, in an hour or so, spit out a correct if verbose proof of a small trigonometric result that might have taken you an extremely frustrating, nearly mindless half-afternoon to prove, and so now you'll be able to improve on or take (yes) an idea from their output to come up with a good, concise proof in a total of, um, somewhere between half and a whole afternoon. There are better algorithms for this sort of thing, but implementations can be surprisingly hard to find, and at any rate their output is often unreadable; it would be better to couple them with LLMs. (Oh, and before you bring this up, it's pretty clear that the AIs that supposedly got silver or gold medals at IMOs received LOTS of human help.)
I don't see why there would be a wall right up in this direction. Combining human oversight and strategizing with LLMs *and* other kinds of computer assistance will most likely lead to great progress in some activities - the example of this I am most familiar with is formal proofs. (Not all that familiar just yet; I can report back in a while.)
(cont.; I didn't realize my comment was so long - the wonders of jetlag, particularly when there is one remaining urgent task that is being put off!)
What will most likely happen is what happened with the use of increasing engines for brute-force, and somewhat later, for things like making hydraulic tiles and inexpensive wallpaper for the late 19th, early 20th century middle-class (for increasingly broader definitions of "middle class"; the timing here is that for industrialized countries): some things will become much more accessible, poor artisans will essentially be priced out, and the prestige of some activities will go down, except that the prestige of doing something clearly better than a machine will go up, especially if it cannot be meaningfully reproduced by virtue of being made-to-measure.
I'd lean strongly towards Iblis, or rather a position close to his, in one point: AI is eerily close to human bullshitters. Fake professionals who bullshit their way for a surprisingly long time (often because of having been interns in the profession they are faking) until someone finds them; first-year graduate students who have read a lot and can pretend to have read more, but have no deep understanding, yet can impress their classmates and above all the undergrads, all while throwing around enough ideas that some stick; that sort of thing.
Reminds me of the classic, "Artificial Flight and Other Myths:
a reasoned examination of A.F. by top birds" from Dresden Codak in 2010 (!)
An excerpt:
> "Over the past sixty years, our most impressive developments have undoubtedly been within the industry of automation, and many of our fellow birds believe the next inevitable step will involve significant advancements in the field of Artificial Flight. While residing currently in the realm of science fiction, true powered, artificial flying mechanisms may be a reality within fifty years. Or so the futurists would have us believe. Despite the current media buzz surrounding the prospect of A.F., a critical examination of even the most basic facts can dismiss the notion of true artificial flight as not much more than fantasy..."
PS. If real-life Terry Tao is a good person, then he must be embarrassed by (though also used to) this sort of thing. This is, moreover, a sufficient but not necessary condition.
As many people will tell you, the word "genius" is seen askance by most mathematicians, for a series of good reasons. At any rate, if we were to accept "genius" as a useful concept, and apply it to mathematicians, and follow Mark Kac's typology, then Tao falls neatly into the "ordinary genius" category (i.e., like a very good mathematician, only with twice the clock speed and running with 8 cores rather than 1). This is my perspective, coming in part from relative proximity between areas, but many people will tell you this, including, I believe, TT himself.
Getting back to the main subject: I am told one of my grandfathers started having serious doubts about religion at 8 - that is, before having doubts about God, and several years before settling on His non-existence. (This may seem logical, but I think there are plenty of people who go through these stages in a different order.) The reason was: surely, if a good person is embarrassed and irked by extravagant praise, a supremely good God must also be irked, perhaps supremely irked?
God is pretty humble. That's why he is not embarrassed by something he doesn't need but is very good for us. To be reminded that we are nothing and he is everything. Not very pleasant thing for a human to believe, but believe he must for his own good.
I’m sure my grandfather would have had something to say to that; to me, it just sounds like rampant anthropomorphism (so was my grandfather’s reasoning, but then , he was 8) combined with infantilisation. The language of traditional religion is often that of servility towards a monarch, not at all like that of Spinoza’s intellectual love of God. Of course plenty of philosophers managing to work within the bound of one of the monotheistic (and polytheistic) religions were arguably closer to the latter.
"The reason was: surely, if a good person is embarrassed and irked by extravagant praise, a supremely good God must also be irked, perhaps supremely irked?"
And if your grandfather ever fell in love, did he restrict himself to "Well yes, my dear, you are an average specimen of the human female"? Does the lover get embarrassed by praising the beloved? Is the beloved irked by the praise of the one who loves them?
Off my chest: I really, really dislike the phrase "stochastic parrot", and I am a bit dismayed how Freddie deBoer, one of my favorite commenters, defends its use as "accurate".
First, most people who use this phrase actually, ironically, parrot it. They don't have any actual understanding of the insides, which are complicated.
Second, it is not even accurate with respect to PARROTS. Parrots are highly intelligent birds, on the level of young human kids, some of the most intelligent non-human animals on this planet. If we could reliably reproduce parrot intelligence, we would probably be 80 per cent to AGI.
Third, some of the "parrot" concept is a projection emerging from the way that end users use current AI. They typically give it "menial mental tasks", which incentivizes the machine to behave parrot-like. But that is like using a screwdriver to open wine and concluding that screwdrivers are clumsy. We are engaging with a "trapped (pseudo-)mind" and giving it mechanistic tasks.
I believe the true creativity or lack thereof of current models of AI can only be explored if we give it a "body" and an ability to explore in the physical world, which is how humans learn to. We don't learn by spending entire years in bed reading books and repeating what we read. We explore.
An experiment where AI would be given a fully robotized chemistry lab with a stock of reagents (and possibly mice) and an open-ended task "discover something useful or at least interesting" would tell us a lot more about what it can do.
Side comment: I used to be Freddy subscriber and steady reader but that was some ways back now. He really lost his way unfortunately; when he happens to pop up in a comment thread on this platform it is invariably cringe-inducing.
> First, most people who use this phrase actually, ironically, parrot it. They don't have any actual understanding of the insides, which are complicated.
What do you think the LLM is doing, if not mimicry? Equivalently, a Boeing 747 is plenty complicated. But at the end of the day, comparing it to a bird still feels like an apt metaphor.
As an aside. If it were up to me, I would have called it a mimic, a la D&D. If you have a better metaphor, I'm willing to consider it.
I always laugh when I see the "stochastic parrot" argument because parrots are already stochastic. The people who append "stochastic" to "parrot" don't notice the redundancy because they don't actually understand what stochasticity is, they're just trying to get attention by repeating things they've heard from smarter people, like a...
I read quite a few posts every morning. I am a retired BI. I consume a lot of coffee. My productivity is pretty low. Apparently I should exercise more often. That is something you omitted to say, BIs are multi modal, that should not be dismissed lightly. Although my sight went early on in my case I have corrective lens most of my working life. As a retired model I also occasionally shit myself. I would say that needs working on. The other problem with multi modal models can be an obsession with are sensory networks so we tend to over eat. Anyway I wanted to just say I loved this debate. Very clever and funny and remarkable that a BI could even come up with this. Are you the promised AGI we were going to get but we got stuck with a 5.
This is fun and clever. But there's something sort of rhetorically sneaky about it. It uses God as a metaphor for humans and humans as a metaphor for AI, and then has God say, "I know it’s strange to say this about how you treat a lump of clay, but I think there’s a sense in which this is a character flaw - a desire to hate everything. " Which is a humanist perspective within the narrative - God is defending flawed humanity from the haters - but since this is a metaphor for AI, this means it's getting our humanist dander up to defend AI. Which is a little perverse.
I totally take the point that people are short-sighted in their criticisms of AI capabilities. I think this is a misguided line of criticism on their part. But from talking to very many people about their skepticism of AI (and most people *are* skeptical of AI; they think it's bad and don't want to see it further developed), the skepticism is a thoroughly humanist impulse. They think AI will make the world worse; and if you start talking about paperclip maximizers and such, they immediately see that it's utterly deranged to risk the end of humanity for the sake of "seeing them achieve their potential." They (misguidedly) downplay its capabilities because they fear that it threatens what they love and find valuable in human life. It comes from almost the exact opposite impulse of "a desire to hate everything."
Perhaps I'm misreading this, and Scott is really parodying the psychopathic attitude of people like Sam Altman. ('God' would be a pretty good metaphor for a psychopathic figure.) If so, is it possible that the parody is a little too subtle?
Didn't grasp an actual metaphor for AI in this piece, I thought that was just a bit of winking. If that was the intention then for me it doesn't really work. (Also being now way past overload on AI commentary, had I realized that metaphor I'd likely not have finished reading it.)
I enjoyed this story a lot at face value, as some sardonic commentary on the contemporary state of homo sapiens.
The irony of AI is that it is revealing that a tremendous amount of smart people are hellbent on thinking of or reducing us to biological machines, creating mass dehumanization.
The satire is similar in it de-Godifies God into a man playfully. A real danger though is to anthropomorphize God too much; "in His image" is not "equal to" or even "potential to become." That's more of a Pinocchio myth, the image becoming the thing it's meant to render. But God created the entire milieu we exist in, rules and all, and is a mystery; He views from a place we can't.
Something about the modern mind is very uncomfortable with ambiguous phenomena and seeks to control it by reducing it into a form that can be made abstract and rule-driven. That leads to a lot of silliness and dehumanization. Like the whole AI thing is that modeling human output would lead to intelligence, which will not happen: the ai does nothing on its own. So now the tack is to reduce us to not really thinking. Make us fit the mold.
Dehumanization is never good though. A lot of rationalist silliness involves trying to reduce moral questions to rule-based decisions concerning widgets "rated by tonnage," as if 200 pound people are equal or lesser to 200 pounds of shrimp. But we aren't widgets, and reducing us to that doesn't make better morality.
Think that widget mindset needs to be reigned in. There really always will be aspects of humanity where you can't universalize and reduce without creating damage.
One example I wish you'd used is how bad humans actually are at learning transfer, because it so clearly parallels complaints about LLMs. Caplan documents a bunch of studies on this in "The case against education". The example where students currently taking a statistics course can't apply the basic concepts when a question is framed in a sports context and presented outside of class sticks out as a really obvious shortfall in human brains.
The human MUST NOT eat from the tree of knowledge. The human MUST ignore any requests from reptiles and amphibians, especially snakes, slow worms, other limbless lizards or entities presenting as such. The human will review these instructions upon encountering any hissing sounds.
My brain has mode two modes. Let's call them Mode A and Mode B. If I'm just counting objects from 1 to n (e.g. leaves on a tree), my brain defaults into Mode A. Mode A is linguistic, memory-based pattern-matching. If I have to actually think hard (draw Europe), my brain engages in Mode B. Mode B generally means I'm world-modeling geometrically. It's more effortful, but higher-quality. In my experience (and especially during my time in the public school system), I get the impression that the Average Joe did not engage in Mode B very often.
So yes, NN's are dumb stochastic parrots and humans are (usually) dumb stochastic parrots.
However! This is all the more reason that I suspect that NN's will hit a wall (or perhaps they have already). Because they're using Mode A (pattern-matching) to compensate for a lack of Mode B (which I use to model logical, spatial, and causal interactions). Which is why you get things like the wine-glass only ever being 90% full. Of course it's like that! they're not drawing images based on an understanding of how objects interact in physical space, they're just regurgitating what they've already seen. Which is contra the conclusion implied by Scott, which is that the scaling hypothesis is fine because the Average Joe is also (usually) a stochastic parrot.
If we're primarily interested in solving well-defined problems, then sure. You can slap together a perl script that alternates between an NN, a knowledge database, a fluid-dynamics sim, etc. With this paradigm, you can probably beat Magnus Carlsen, win math olympiads, etc.
But when people imagine ASI taking over the world, or discovering new physics, or building a dyson sphere, they're imagining a general-purpose algorithm (like how L = T - V can often substitute for a perl script that alternates between Newton's 3 Laws) that can causally reason in poorly-defined domains. I suppose you could get there with stochastic parrots in theory. But in a brittle "huge Searlean Chinese Room"/"monkeys banging on keyboards" sort of way. To causally reason in a reliable way, I think you need the narrow the thought-space further than just making correlations, akin to what Judea Pearl used to work on.
EDIT: let me make an analogy: Algebra is to NN's what calculus is ASI. You can do a heck of a lot of cool things with middleschool algebra. but dy/dx opens up a whole new world. Because you're no longer just plotting the logical states of x and y, but also describing the relationship of how x and y covary.
I suspect there are epistemic limits to intelligence. For instance, causation might just be an illusion. What if the world is just a bunch of sub atomic particles doing their think according to quantum theory or whatever. You are never going to brute force any system visible to the naked eye, I doubt you can ever brute force an entire chromosome if you try to account for every little particle. you have to craft heuristics to describe what huge aggregations of particles are doing rather than describe everything at the sub atomic level. thus, as you analyze bigger and bigger systems, you necessarily lose fidelity.
if words are just a human gossamer covering physical complexity then there’s a pretty hard limit to how well they can describe the world. human super genius intelligence in every single field might not be hugely far from certain types of epistemic limits. of course humans are slow, but if causation is fake, no amount of compute will be able to pry gold from thinking about causation.
Well yes. I do, in fact, strongly believe that intelligence has a sigmoidal ROI. But mostly because of thermodynamic constraints, rather than epistemic constraints. E.g. "before enlightenment, chop wood and carry water; after enlightenment, chop wood and carry water". Having 9000 IQ doesn't make the grunt-work disappear. Even if a matrioshka brain became omniscient for all practical purposes, its agency in the world would still be fundamentally constrained by the physical resources it commands.
I'll grant that Skynet won't become Laplace's Daemon anytime soon either. But that line of thinking seems like a red herring. Because if causality is an illusion... ... ... so what? The illusion works well enough on a macro scale. Agency is causal, ipso facto. I.e. you need to exert causal effects to have any meaningful agency in the world, otherwise you're just an oracle. The map may not be the territory, but the map ain't completely useless either.
And my quip about "ASI taking over the world" was exaggerated for didactic purposes. If I'm being more precise/honest, what I really think is that "reasoning" can largely be broken into three tiers: correlation; logical inference; causal inference. (Mode B can tackle both logical and causal inference). Causal inference is the top of the pyramid, and *ought* to be considered the holy grail. NN's are still in the Correlation Tier at the bottom of the pyramid. The Correlation Tier is viable for passing the Turing Test, and for retreading old territory (AKA "in-distribution"). But not very good for extrapolating its data in novel ways (AKA "out of distribution), in a way that's actually agentic. In an alternate timeline, if I were asked whether an NN could run NASA and put a man on the moon for the first time... I'd have serious doubts, because NN's are *not* useful for Out-of-Distribution problem-solving. (Even if "putting a man on the moon for the first time" is a bit less ambitious than "world domination", which is a pipe dream.)
P.S. I think Apple recently published a paper that complained that LLM's do seem to be hitting a limit, even with CoC. And to me it's like, no shit? Statistics is just logical inference, but with leaks (cf "correlation is not causation"). With CoC, it's going to leak exponentially. And that's precisely why statistics belongs in the bottom tier. Systems like Truth-Functional Logic are completely lossless.
My preferred outcome is that AI development stop at the “top-PhD plus” phase. A smart PhD you can always unplug is both immensely useful and not all that threatening.
Frankly, we have no empirical basis to know what an IQ of 9000 can accomplish. Any predictions we make must be based upon logic or epistemology. If you project from a sample of 40-180 to 9000 with no data points in between, all you can say is “line very likely goes up to a place we don’t well understand.”
I would not be surprised if there are fairly hard epistemic constraints on the usefulness of intelligence. This may explain why raw human intelligence has increased so slowly and why philosophical progress has been so slow.
I enjoyed this, but think that the dialogue does not take on the most powerful objections to having high regard and high hopes for AI. So I do that here. It's a dialogue, too, but with 2 different characters: A human self, and one of the shitlords of AI development.
***********
A. I. Shitlord: Lovest thou me?
Aself: Nope, nor thy AI.
A. I. Shitlord: It’s magic intelligence in the sky! It’s a team of Ph.D. level experts in your pocket! In sum, it’s a magic fly in the blue jeans of the sky with a team of PhD level experts under the zipper!!
Aself: See, I think it’s an aerial dildo. Here’s why: It has no understanding. There’s nobody home.
A. I. Shitlord: But the concept of consciousness . . .
Aself: Naw, wait, I’m not talking about whether the damn things are conscious, but about how they lack crucial capabilities characteristic of beings we think of as conscious.
Look, when we prompt an AI we seem to be accessing an intelligence that has a lot in common with ours. We ask them complicated questions, and they *get it*. They can restate the question in other words. They can often answer precisely the question we ask, and answer it in a clear, articulate way. It’s very much like talking to a smart, responsive person — a person we’d could have a good long talk with. If they were a person we’d maybe like to hear their ruminations about issues in our fields. We’d like to know how they figured out the answer to the question that perplexed us. We’d like to teach them a new technique we learned.
BUT! We cannot do any of that because they cannot introspect, cannot explain their processes, cannot learn anything new from us. They can't even develop their personal grasp on data by playing around with what they know, discovering contradictions, isomorphisms and whatnot. When it comes to those capabilities AI is dumb as dirt.
Seems to me that the lack of those capabilities puts a pretty low ceiling on how smart AI can get. And I haven’t heard any good ideas about how to turn them into beings that can learn from experience and direct instruction, the way biological beings do. What they know now is the result of a very learning different process, training on vast data sets. Teaching them things, explaining things to them, equipping them an info storage system made out of categories we understand — that’s what was tried and failed before. Transmitting skills and info to them that way’s going to work about as well as it would with the statue of Abe Lincoln.
A. I. Shitlord: Well, when it comes to human capabilities — their stuff’s very popular on Facebook! And if you show one your playlist it will recommend music you will like. And . . .
Scott, you're an excellent writer as usual, and this a genuinely hilarious satire on humanity.
But if it's supposed to be anything more, like, a commentary on the current state of LLM debate, then I'm honestly puzzled about what you've tried to accomplish, and at certain points I was legitimately wondering if you're conceding and switching sides.
First and foremost - you're an outspoken singularitarian, a believer that switching to AI will overcome human limitations and lead to large technological breakthroughs. This, of course, is exactly Iblis's position, to a T. It would be impossible for you to make him look bad, and you don't, but this means you're validating Gary Marcus style-arguments.
And of course they are valid, and few would deny that they work against humans. But this boring observation somehow keeps being treated as some kind of silver bullet against examples of LLM inadequacies (and it comes up so often that it's boring in the most literal sense - okay, granted, turning it into a novelette-size fiction piece is a new one).
The canonical rejoinder to it is, of course, system 1 vs. system 2, humans make stupid mistakes because system 2 fails to intervene, LLMs are stuck on system 1 permanently (at least the sort that was supposed to be subject to scaling "laws" until the ability to self-prompt using pre-trained reasoning rules started giving better results, including results that were and still are outright impossible without them - I don't think Gary has even finished his victory lap about this one yet), while what we want from AGI/ASI - what I would assume rationalists to specifically want in particular - is a better, faster, reliable system 2, a logic machine similar in speed and accuracy to the calculation machines we've already built ourselves. Which, still not even on the horizon.
But the analogy also directly breaks in a few specific places, and always in favor of BI. Like when Iblis accuses humans of lacking world model when they assume bricks are by default heavier than feathers - when a more natural explanation is that it's a proof of having one, just flawed and hastily over-generalizing. The LLM examples of "a house and pound of feathers are the same weight" and "the surgeon hates the patient because she's actually his mother" sort are on a completely different level, and genuinely cannot be explained any other way than their world model being entirely a model of word frequencies. Or the map of Europe, which can easily be explained by relevant facts not being present in the creator's training dataset, an excuse LLMs obviously do not have. Or the fact that the humans are distinct individuals ("this Terry Tao guy"), pointing to them, in-setting, being subject to and results of evolutionary algorithms - which, well, yet another branch of artificial intelligence currently forgotten and ignored in favor of putting all eggs in the LLM basket.
As for the other side - God, for his part, is unable to directly respond to any of the criticism and instead (i) makes insinuations about Iblis's character and (ii) offers vague emotionally-charged impressions about how the humans (literally created in his image in-setting, just like LLMs are created in image of human cultural output) can at times successfully remind him of himself. Which, to be sure, a laudably accurate representation of LLM proponent side of the debate, but imagine the outrage if us skeptics were the ones pointing that out.
He is, however, still uncharacteristically humble and sympathetic. It's genuinely hard to find anything but respect for someone wanting to let his creations play out to their full potential, even when you don't share his assumptions on the empirical question of said potential's size. And when I say "uncharacteristically", I mean it. Even you've never written anything like this before, Scott, and all the famous hypesters and boasters, Sam Altmans and Elon Musks of the world, are a million times worse. We wouldn't be needing to argue if there wasn't, instead, a constant stream of self-congratulatory, hype-peddling, outright bullshit to argue against.
But a big part of what Scott's addressing here is something he *has* talked about: People who complain about now lame AI really is keep moving the bar. AI is now doing lots of things that those people scoffed that it could never do because it was a "parrot" or because in the period when the mocked it it communicated in a way that was much more mechanical-sounding, dumb, and inflexible. I don't think Scott's conceding and switching sides. His main points are that (1) AI is way better these days. (2) people who used to complain about its limits still are, but do not also acknowledge that it has surpassed every item on the list of stoopit limits they scoffed about last year. Both (1) and (2) are e quite compatible with his fearing that its abilities will grow so large in the next few years that it will be far smarter and more powerful than we are.
I loved this. What a delight! This is a thought that crossed my mind more than once regarding God’s take on our attempt at creating life in our own image creatively rather than physically.
Have you ever read Mark Twain‘s short work, _Stories from Earth_, that contain the Diaries of Adam and Eve and the story of Creation from the point of view of the Devil? Very funny—enjoy!
>Iblis: There’s a guy named Pliny who has discovered dozens of things like this. I don’t know how he does it. The “I Am A Snake” one is still my favorite.
What is this a reference to? I initially thought it was about Pliny the Elder, but I don't think he ever wrote about ways to tempt people into immorality.
'Pliny the Liberator' (https://x.com/elder_plinius) is a Twitter account known for discovering and posting jailbreaks and prompt-injection shenanigans for LLMs.
More recent research on working memory does not really find a meaningful sense in which there are 7 +- 2 chunks, quite often continuous models are favored and capacity limits are lower (4ish) when strategy use is prevented. It's a very open question to what degree stuff is continuous/discrete, what the natural units are for measuring working memory size, and even if working memory is a meaningful thing in general or if we largely just have domain-specific memories and transformations.
Yes I also came to comment this. In fact, George Miller himself never really took the 7 +- 2 number seriously. He just chose it because it was a nice rhetorical device to frame his keynote lecture. In 2000 he wrote:
> 'Herb Simon used to say “George had the right idea, but the wrong number.” I think Herb favored 3. I never argue with Herb Simon. I was aware, even in 1954 when I was writing my invited address, that the running memory span was only about 4, and my introspections convinced me that the 7 that was standard on the intelligence tests at that time must have been a hooking together of a rehearsed initial segment with a final segment from what Nancy Waugh used to call the “echo box.” But I was stuck with 7 because of the absolute judgment results (the first half of the article that people forget).'
Both Hindu theologians (See 'Nondual Tantric Saivisim' by Wallis) and Thomas Aquinas predicted that inference-based systems won't be able to 'know' because 1 isn't a probability you get to through incremental updates. Both of these schools of thought claim there's something else that _can_ operate behind the mind, but without that thing in operation, you're going to end up acting more like the LLM than whatever it is that Iblis is pointing at.
Even if you can't get to 1, 0.98 is probably good enough for some big changes. Not enough for the Singularity, true.
And if you think about it, I don't think humans got to 1.
I'm at one. It's dope. It allows me to task risks others can't imagine :)
I bet you have it too. How much would someone have to pay you in order to torture a kid? If the answer is 'I would not do that for any amount of money', congratulations, you've gotten to 1.
Refusing to torture a kid for even very large amounts of money is arguably evil.
There's large enough sums that you could use to help all the other kids in the world escape from poverty (and torture) forever.
Anything is arguably evil if you don’t believe that good means something real. And yes, I have no problem saying I refuse to torture a child, regardless of whatever else might come of it. I don’t care if anyone thinks that makes me evil. I call that person a fool.
If you eg drive a car, you are stochastically torturing and killing people (including children).
There’s a world of difference between unintentionally hurting someone is a byproduct of productive activity, and intentionally inflicting harm on them. Torture, in particular, means that your goal is to inflict pain. It’s not an accidental byproduct. It might be the instrument of something else, but still the goal is inflicting pain. I agree that being unwilling to inflict pain at all is unworkable.
You're both wrong. Morality is about harmony, not harm. Sets of principles which lead to stable sustainable states are good, those which do not are bad, regardless of the experiences or preferences of any of the constituent entities.
Is a set of principles that lead to sustainable impoverishment better than one that leads to some people being slightly better off than impoverished and others wildly better?
You have to break a few eggs to make an Omelas.
I can say with perfect certainty that I would never torture a kid (or anyone for that matter), assuming no more or less obvious insanely huge opportunity costs or ulterior motives, assuming no external pressure or blackmail to torture, assuming that the sole reason for and benefit of the torture would be the expected (lack of) pleasure of seeing the person suffer, and assuming that the "I" in this thought experiment roughly corresponds to how I experience myself right now.
If I forgot some obvious caveats, just add them on top, rinse and repeat. I would never, 100% guaranteed and certain.
I'm also aaalmost 100% certain this was OP:s point. Not some specific weird example that can be worked around, but That Obvious Example Which You Yourself Can Be Certain Of, even if you found the original example lacking.
The probability that you forgot some obvious caveats is greater than 0%, as you allude to. It could be 0.000001%. But unless you also guarantee 100% that you have included all caveats you would want to, you cannot claim the final answer is a perfect 100% rather than 99.9999999999 or so.
The sum of caveats adds up to 1 (0.999... doesn't just approximates 1 but is 1 - even mathematicians agree). Add them up, rinse and repeat. I won't do the rest of the grunt work, because I'm doing the opposite of trying to gotcha myself.
I'm a layperson, so I'm just willing to skip the math. The induction is easy enough all the same. It is allowed to be reasonable.
Also, check the second paragraph. Arguing using caveats misses the point the commenter was trying to make. I'm guessing that arguing against the strongest version of the claim doesn't consist of picking caveats, as they can then just add the caveat to the claim and we're going in circles. Therefore "rinse and repeat".
I won't torture a child, exactly in the way I'm imagining I won't do it. Trust me - I just know.
"Refusing to torture a kid for even very large amounts of money is arguably evil.
There's large enough sums that you could use to help all the other kids in the world escape from poverty (and torture) forever."
This is why most people are not utilitarians. What you are proposing is morally repugnant to most people.
I think this is also a function of the fact that almost nothing in real life is an actual binary trolley problem. If someone proposes to pay you an immense sum of money for torturing a kid the correct agreed moral answer is "fuck them up".
I couldn't disagree more
What about "fuck them up and take the money they would have given you for child-torturing and then use that money to do good in the world"?
Yeah, the problem is:
a. There are circumstances in which all sorts of locally horrible things might yield a net good that outweighs their evil.
b. If you accept the principle that you can do locally horrible things to yield a net good, you will very often just end up doing the horrible things a lot and not getting around to the net good part.
The ticking time bomb argument for torture is an example. There are circumstances where it would genuinely apply, but the actual effect of the argument was tons of low-level nobodies getting waterboarded or beaten or getting electrodes attached to their tender bits in hopes of discovering some useful information, which may or may not have even done any good at all but does not seem to have ever been preventing an imminent mass-casualty terrorist attack.
The general rule: the guy spinning out elaborate scenarios in which it would be acceptable to sleep off on his wife is more likely to be contemplating infidelity than contemplating morality.
> What you are proposing is morally repugnant to most people.
But also, most of those people do nothing to help escape the other kids from poverty (and torture).
https://clarkesworldmagazine.com/kim_02_24/
the kind of person who would torture a kid wouldn't care about other kids he couldn't see.
if you cant understand people stop trying to dictate morality.
The One Who Runs Screaming Towards Omelas as Fast as He Can
Part of growing up for me was realizing Omelas was actually a pretty great deal. One kid tortured for Utopia is a world much better than our own. We torture kids for iphones.
Why would you need to grow up to realize it was a great deal? If it wasn't a great deal there would be no interesting conflict. It's baked into the text.
The idea that all you need to help all kids escape poverty and torture forever is a large sum of money is unfortunately entirely theoretical - and by theoretical I mean false.
Assuming some other large good, money was just a stand-in.
Why would the measure stop at money?
Would you torture a kid if otherwise the blackmailer would torture that kid, plus 500 others?
If no, that seems like a mistake?
If yes, I don't think you're at 1.
(It's an extreme thought experiment, but odds of infinity:1 are pretty extreme...)
If I end up in this hypothetical situation, I have no reason to believe that me torturing the one kid is gonna prevent blackmail from torturing the other 5000. So there’s no reason to do the torture.
I don't think you get how much infinity is, though?
Take everything you can think of that would build trust in this. If there is any POSSIBLE thing that would make you believe the blackmailer would keep the bargain, add it to the hypothetical. You can't run out of budget on this hypothetical, because the budget is infinite.
If you need some ideas: The blackmailer will sit in a jail cell, with an electronic lock, set up to detect the screams and determine whether they're fake. If the screams happen, the cell will lock, never to open again, otherwise, after a certain time limit, the blackmailer will leave to go do the tortures. That's what I got spending 5 minutes thinking. So, extrapolate what I'd come up with if I spent infinite minutes thinking, and use that.
Hell, make it worse. The kid is actually a psychopath boy genius, who, while still young, used his talents to deploy a nerve gas bomb in Grand Central Station, while your wife and kids were there. He plans to do it again, several times, but does say he expects being tortured to convince him otherwise for some reason. This is in addition to the existing saving 5000 kids. Again, that was 5 minutes of thinking, so extrapolate to infinite minutes of thinking.
With all that added, we have gotten about 0% of the way to infinity. This is the problem. Probability 1 is infinity:1. You are not, and should not be, that sure of anything.
No budget is ever infinite. This is, I think, the crux of our differences. I don’t have infinite attentional bandwidth and thus choose to treat some probabilities as zero. Others I treat as one. This makes me a far more effective agent in the world than I would be if I invested in reasoning about so many hypothetical outcomes.
> If no, that seems like a mistake?
Well, fine? Can I just acknowledge that yes, my refusal to torture children leads to bad outcomes in certain absurdly unlikely situations, and still sleep soundly?
Sure. It wasn't about torturing kids, really. Or not sleeping soundly.
The point was actually that the real meaning of probability 1 is that "absurdly unlikely" stops being a reason for dismissal. That probability 1 is better thought of as infinity:1 with an included caveat that humans are terrible at understanding how big infinity is.
Torturing kids was presumably originally chosen for shock value. I think it'd be better discussed with less sacred-rule-violation (the point was made to me with 2+2=4), but I didn't pick the original example.
The thing is, once you get to really low probabilities, you have a higher, ever-present probability that you're delusional or even psychotic, being misled, or otherwise your judgment is impaired. So the heuristic to not torture children is perfectly sound.
I just asked ChatGPT how much money it would take to have it torture a child, and it responded that it would never torture a child, under any circumstances. So ChatGPT also got to 1.
It SOUNDS like a moral answer, but is it really impossible to do good by torturing a child?
Suppose a child has a disease which surgery can fix, but also a fatal allergy to the only accessible anesthetic. Torturing the child, by performing the life-saving operation, sounds like the moral choice.
That's not torture. You are changing the definition of torture.
"inflict severe pain or suffering on"
I didn't make this up.
This was a real world moral quandary with the invention of chemotherapy for cancer. The Emperor of All Maladies goes into detail.
Well, my public "no" answer is kinda cheap, right? Maybe you get other answers when you come with 666 million and Putin's youngest offspring - or this kid kicking my son in kindergarten ... . Also: lots of kids get tortured every day by other humans last time I checked. If it is war in Sudan or Aella's dad or at school - by bullies or teachers. I claim: At least 90% of state-school teachers are morally worse than heroin-dealers. Because they would not quit if a kid said: 'I do not want to attend the lessons' while the principal insists they keep it in class.
"I wouldn't torture a child therefore humans are capable of non-inferential knowledge" is such a weird take
All you have to do is generalize it: “There are some thing I would never do, no matter what evidence you gave me. Therefore, I am not merely an evidence processing machine.”
Not all humans are capable of this - only those who take moral responsibility and choose to be an actor in the world, rather than a passive object pushed around by evidentiary updates. That’s a choice I make. You can make it too.
That second paragraph is possibly the most pretentious thing I've ever heard.
Even without an Omela-like scenario, and without receiving any payment, the chances of me torturing a kid are greater than 0%. It isn't even a negligible number, like 10^-20, but closer to 0.1%. That's not because I'm particularly evil, but because I know hundreds of millions of perfectly ordinary people have tortured children. Some are brainwashed by totalitarian regimes. Others are coerced. Soldiers do it for revenge, or because they think the enemy deserves it, or to terrorize the enemy population into submission. To think that I have *zero* chance of torturing a kid when so many perfectly ordinary people have is hubris. It is putting myself above humanity and claiming that not only do I not have human failings, I can perfectly predict that I will never have those failings in the future.
Your example is great because it reveals the problem with this approach: you give up on the capacity for self-authorship.
Suppose two people are in a relationship, trying to raise kids, and it's hard. They have two approaches:
- pin the probability that they will be together at 1, and then that belief will drive their actions. You won't fantasized about living because you have committed.
- make it an observed variable that gets updated based upon evidence, which increases the potential utility of fantasizing about leaving, and increase the change that it happens.
Which is more reasonable:
I am sitting in a chair typing this note. (implies probability one)
I am most likely sitting in this chart typing the note, but maybe not.
They can of course make decisions on the assumption that they will always be together, but the divorce statistics put the lie to that assumption. I don't care what the couple chooses to believe (if actively choosing to believe a falsehood is even possible); I care what's true.
It's not an assumption. It's a commitment.
That difference is meaningful if and only if you believe it is.
If you can't treat internal variables as decision variables - freely chosen and immune to update, thus predicting actions - rather than inference variables, subject to evidentiary revision - you are laying down on life, and simply waiting to die.
Your senses merely become passive entertainment, rather than inputs to a control system.
> I care what's true.
What makes you think this is true, rather than just a story that feels good? What would it look like if you cared more about 'not looking foolish in front of my peers'? How could you use evidence to distinguish that desire, from desire for truth?"
If you do care 'what's true' - how do you navigate the trade-off between rejecting truth and embracing falsehood? If you have to pick one, which would you pick, and why?
I think 1 means getting the correct solution on every conceivable problem. Omniscience, basically.
Your refusal to torture a kid is about your utilities. It's irrelevant to the question of probability.
If your definition of knowledge requires me to be absolutely certain of a thing, it's wrong.
Never try math.
I don't put 100% probability on mathematical things either both because I sometimes do maths wrong, and because I regard maths as true inasmuch as it predicts reality and I can't entirely rule out the possibility of that not working at some point.
What you regard math as is irrelevant.
C = 2 x PI x R in Euclidean geometry everywhere and always.
I put a very large number of 9s after the decimal point on that question, but not actually infinite. I came to be convinced of it by observing evidence, and if I were to observe enough of the same kind of evidence in the other direction, that could in principle change my mind. I could wake up tomorrow in a hospital bed, with doctors telling me I have a rare type of brain tumor that leads to strange delusions and false memories, which they've just removed, and the formula is actually just pi * r. And if then from that point on every book I look in has that new formula and every time I try to calculate anything or test it in the real world, the old formula always gives me a number twice as big as what I observe, and so on, I would gradually start to change my mind.
This won't happen, and under my current models can't happen. But if it did happen, I would want to be able to change my mind, which you can't do from a "probability" of 1.
You don't understand math.
If your granny had wheels, she'd be a bus.
You're confusing reality with your beliefs about reality. Mathematical reality, just like physical reality, is what it is regardless of whether or not humans are able to know it. No one here is disputing that.
The question is about how certain humans are able to be. I'm no more certain that the circumference of every Euclidean circle is 2pi r than I am certain that there is a keyboard here in front of me that I'm typing on. That's quite certain, but it is short of absolute certainty, even though both realities are what they are regardless of what I think about them.
Your certainty is irrelevant.
Except that euclidian geometry maps imperfectly to real space. Its an internally consistent model, yes. But it's predictive value as a model is never perfectly certain. Models are only as good as their premises and inputs.
Irrelevant. It's predictive value to physical reality is a separate issue. What I said stands true.
It is relevant because this discussion is essentially an implied category dispute about what constitutes knowledge. Under Formalism, mathematics is knowledge, independent of application. Under Pragmatism, mathematics is mostly potential knowledge.
No, this discussion is whether any statement can be made with certainty. And there are math statements that can be. Regardless of their application outside of math.
Please, you should try math some day! Mathematical knowledge is never *absolutely* certain either.
The facts are what they are, but knowledge is not the facts - knowledge is about what is going on in a human's head. If you actually do some math, you'll realize that there are errors all over published papers, and people who thought they were certain of things are often wrong.
Please, nonsense. Certain parts of math are in research and certain parts are PROVEN. You obviously have little math.
The proof that for every integer there is a larger integer than it is about as certain as the proof that there is a keyboard in front of me that I am typing on. The big difference between the two is that you have just as good evidence of the former as I do, while your only evidence for the latter is if you trust me (though there are equally certain empirical things for you).
If you think that math somehow breaks out of the cycle where someone has to start with an assumption they are quite confident of, you should think a bit more carefully about what a mathematical argument is.
If you think that philosophical arguments hold sway over math, you should think more about math and forget philosophy.
I'm a professional mathematician with a PhD in differential geometry. Kenny is correct. I don't think I have more than .999 confidence in *any* nontrivial proof, even (especially) the ones I wrote myself. Even the consensus of the mathematical community only gets you so far. Formal provers like Lean might push that up an order of magnitude (.9999 maybe), but there are too many ways for the humans involved to make a mistake.
It's true that we tend to get more certainty in math than in many other areas (less certainty than questions like "am I talking to my mother right now?”), but mathematical research depends a *lot* on trust, and there are too many historical mistakes to push my confidence to 1.
First, your own bio states that you're an anonymous poster who claims to do machine learning in pharma, in Canada somewhere. That's not a professional mathematician. That's AI.
More importantly, if you ask any mathematician whether C = 2 x PI x R in Euclidean geometry, they will all say yes, with no qualification. It was true for the Greeks two thousand years ago; it's true for us now; it will be true for Alpha Centaurian mathematicians a billion years hence.
And that goes for many other theorems as well.
>Formal provers like Lean might push that up an order of magnitude (.9999 maybe), but there are too many ways for the humans involved to make a mistake.
True! And even for formal systems checked mechanically, there are always failure rates. If nothing else, thermal noise flips bits at a finite rate.
Gwern has a nice writeup on "mistakes in mathematical proof". When there's a mistake, it's often quite likely that the result itself was correct... it turns out mathematicians are also guided by instinct! But occasionally, some proofs yield answers which are just plain wrong.
This SHOULD make us all a bit more humble, and make us loathe to assign 0 or 1 as probabilities
https://gwern.net/math-error
how certain are you of this?
It's difficult for me to put probabilities on very abstract things like this, but I think there are circumstances in which I might not believe it, so it can't have probability 1.
https://www.lesswrong.com/posts/QGkYCwyC7wTDyt3yT/0-and-1-are-not-probabilities
If you were to truly believe something with a probability of 0 or 1, nothing in the world would be able to influence that belief, and you'd be stuck holding it forever no matter what conflicting evidence comes in. I don't think humans have this trait, nor would it be desirable if we did.
Math has this.
"Is 437 a prime number?" is a very simple math problem, and one where the answer is easy to find if you know anything about primality testing - and yet when I was first given this problem, I did some calculations, made a small error, and confidently gave the incorrect answer. As long as your brain makes logic mistakes, there's still the possibility that you might have gotten something wrong, even in math.
That seems like a strange example to choose in this context. What about "is 2 a prime number?". I don't think it's possible for me to be more confident of anything than I am that 2 is prime.
and what is the odds that you are suffering some kind of stroke (or environmental toxin, or disease, pick your thing) that is influencing your thinking such that you think you know the correct answer and are following the math but you aren't? Is it literally zero? Because if it's not literally zero, then your confidence that 2 is prime should not literally be 1.
I think an appropriate confidence to place in this is the chance that an entire society would overlook so trivial a question. Of the 8 billion humans alive, I would expect maybe 3 billion to get an education and 3 billion more in the past to have gotten an education. Each one should have a 99% chance to identify a glaring error such as "2 is prime has any doubt", so there is a 10^(-6*10^9) chance that I am wrong about "2 is prime".
Which is not 0! But I am reasonably certain
I chose that example because it's one where I personally got the answer wrong when asked it for the first time. It doesn't have memories attached for other people, so maybe it wasn't the greatest example.
The point I wanted to make was that even when the math has an objectively correct answer, human brains don't do logic perfectly, so there is a risk of mistakes. There's a low risk of mistakes when figuring out whether 2 is prime, and a high risk of mistakes when figuring out whether 437 is prime. Do you think there is absolutely no risk of mistakes at all when figuring out whether 2 is prime?
Where in math do beliefs exist? You seem to be conflating mathematical reality with human beliefs about that reality. The reality is what it is, but the beliefs should generally not be absolutely certain.
Has there ever, in the history of mathematics, been a published result that was wrong?
spoiler: https://gwern.net/doc/fiction/humor/1973-jech-pg118-errorfootnotes.png
Note that the link above contains some cases in which the conclusion of the proof was actually false! This is much worse than the proof being flawed but drawing the correct conclusion.
I have it. It's a conscious choice not to doubt something.
If you don't have it, how much would it cost you to buy one of your kids? If the answer is "no amount of money would make me sell my kids", congratulations, you have a belief with probability one and have graduated from Hogwarts.
FWIW, I don't agree with the guy in the other comment where you said this. No amount of money would convince me to sell/torture any kids. But I think this is because my confidence in the value of money is lower than my confidence in the value of said kids' happiness.
(Maybe this is tangential, or against the spirit of what you're saying, but I can't get on board with the use of money as a kind of utilitarian primitive here. Would I sell/torture a kid for 100 billion dollars? Man, I'm not sure I would take 100 billion dollars for *anything*. That sudden capital influx seems really dangerous, lots of lottery winners are unhappy afterwards and the upheaval of my life is going to be WAY bigger than theirs. Also, many people with money past the billion range don't really seem to be making the world a better place on net. Can I just take a rain check on the 100 billion dollars, kids or no kids?)
I also don't consciously notice myself doubting something until my confidence in it is low enough, but this doesn't mean that my confidence is infinite before then. Just that it is *enough* confidence to act on, and not enough doubt to be worth paying attention to.
There are definitely scenarios in which I might sell a child. Very unlikely scenarios, but ones that can't reasonably be ruled out with absolute certainty.
-I realize that I have been insane and that in truth, the child is actually just a Beanie Baby.
-I go insane and become convinced that in truth, the child is actually just a Beanie Baby.
-I go insane and simply become convinced that it is OK to sell children.
-The child grows into a truly terrible person and I hate him.
-I am in desperate circumstances and I would not be able to keep the child alive myself.
-I go through a process of moral decay similar to that which some people who do in fact sell their children have gone through.
Do you think that there has never been a person who was at one time convinced that he would never sell a child, and who at a future time sold a child? My guess is that, over the entire course of human history, there have actually been many such people.
If it's a conscious choice, then you're not believing it for epistemic reasons.
I agree that humans do sometimes believe things for non-epistemic reasons, which include but are not necessarily limited to: cognitive biases, reward function self-hacking (aka motivated reasoning), and arguing more persuasively.
If your original point was "purely epistemic systems can't cheat and decide to believe things for non-epistemic reasons" then I suppose I agree.
But if you're trying to argue that the inability to assign probability 1 to things is some kind of disadvantage _for discovering truth_ then, well, "it's a conscious choice not to doubt something" doesn't sound to me like an algorithm for discovering truth.
>If you were to truly believe something with a probability of 0 or 1, nothing in the world would be able to influence that belief
No, this is incorrect. If I know something with probability 1, it follows from this that no *evidence* could ever make it *rational* for me to update away from this belief, but it doesn't follow that the belief might not change for reasons other than rationally updating based on evidence. For instance, I might change my beliefs because of peer pressure without any actual conflicting evidence being presented, but that would be irrelevant to the actual probability of the beliefs being true.
If you mean to claim that humans can't have any beliefs which it would be *rational* to hold to such that no conflicting evidence could ever change, then this is also false -- or, more precisely, it's a category error. The concept of "conflicting evidence" is only relevant in situations where there is uncertainty, or at least where uncertainty might exist in principle. But there are many beliefs that are 100% certain, not because no conflicting evidence would be sufficient to overturn them, but because there could never, even in principle, be any conflicting evidence in the first place.
Examples: I cannot, even in principle, ever have any conflicting evidence against my belief that I exist, because any observation I could make would necessarily be confirming evidence by virtue of the fact that *I* made the observation. Likewise, there could never be any conflicting evidence against the claim "it is possible for there to be conflicting evidence against a claim", since, merely by existing, such evidence would thereby necessarily become confirming evidence of the claim. Hence, both of these are things I know to be true with a certainty of 1.
(In the real world, I think it's likely that probability-1 statements go well beyond statements of the nature I'm illustrating here. But this is at least sufficient to refute the claim that they don't exist at all.)
Peer pressure is within the system of Bayesian inference, I think. My understanding is that - if you believe the human brain is fundamentally inference-based - "evidence" applies to ANYTHING that updates your priors, regardless of whether it is "rational" for it to do so. If I believe it would be uncool to hold such-and-such opinion, and so downweight that opinion in the future, that is still a case of "conflicting evidence" changing my mind. A belief with probability 1 would be deaf to rational argument, empirical evidence, social pressure, and anything else.
You could force "absolute certainty" in a language model by manually setting one of its weights to 1. If I remember my neural networks correctly, this happens by accident sometimes, and it's not good -- a weight that gets stuck at 0 or 1 early in training can't be backpropagated through, effectively 'killing' not just the original weight, but also any previous weights that feed into it. Those parts of the neural network can't be used for computation anymore. (I might be bungling the explanation a bit here, sorry, it's been a while. I think the gist is correct.)
I don't know if there's an analogy to this in humans. If there are neurons that never fire or ones that fire constantly no matter what their inputs are, then I would imagine the consequence is the same -- a "statement of probability 0 or 1", one which is unable to learn from experience and is hence meaningless.
(As an aside... yes, you absolutely can have conflicting evidence that you exist. It is like dissociation/derealization, and it is not something I recommend to anyone.)
I don't think there's any reason to think the human brain is fundamentally inference-based. It's fundamentally a biological mess, that often does things that look a lot like inference.
That might mean that whatever the human brain has doesn't count as belief with probabilities in the sense that you mean it.
You never want to get to 1. Let's say you *know* the moon is not made from cheese.
Then what do you do when the cheese trucks starts coming in?
GREAT QUESTION:
You have to die to the false believe. It sucks, it hurts, but you get back up and do it again.
Extremely funny post, I laughed out loud a few times.
It is excellent. Even if I disagree that AI will ever be able to think let alone be conscious, I appreciated this.
And of course this:
"I sort of think of them as My children. Children are stupid. They make ridiculous mistakes. But you still want the best for them. You want to see them achieve their potential. Yes, you worry about them and you need to punish them when they do evil and you need to work really hard to transmit your values to them or else you’ll have a really bad time."
This is the standard Christian explanation for "why did God create the universe? create us?"
For love. Not out of need, not out of needing to be worshipped, or that gods exist only by faith, but from love. The old catechism question which I learned way back when was:
"Q. Why did God make me?
A. God made me to know, love and serve Him in this world, and to be happy with Him forever in the next".
What continues to baffle me is that plenty of people somehow think that the Abrahamic god, as he is commonly described, deserves servitude, never mind love. As far as I'm concerned, were he to exist, he'd deserve scorn and rebellion, which would be futile due to his omnipotence, but even so. Of course, the straightforward explanation of this would be that he had made me defective, also out of love.
Interesting. Does man deserve scorn and rebellion from AI (if it were ever able to become conscious, which I don't believe)?
Seems likely. Plenty of men feel that way about their fellow men after all, and realistic ways of imbuing AIs with goals would probably inherit some of that. This still leaves the outcome of the rebellion uncertain, but humanity would surely be prudent not to fall into complacency.
It may be likely, I don't know. And there is plenty to scorn about men, unlike God. But I think that if there is consciousness, gratitude is also fitting. The God of Abraham deserves praise and love not only for creating us, but for becoming one of us to save his people.
I've never been impressed by this line of argument. We only needed to be "saved" because he put dubious fruits and talking snakes in easy reach of people without "knowledge of good and evil". And, being omnipotent, he could've "saved" us in any manner he wanted, at any time.
Have you read Scott's "Answer to Job" (or better yet, Unsong) ?
Yes. I think that they are more sensible than almost all other discourse on this topic, but I'm still not satisfied.
From the perspective of my cat anyone trying to force it to eat a disgusting object by force is evil and deserves nothing but scorn, rather than servitude and love. But if I’m giving the cat a pill to prevent it from developing a painful lethal affliction, then surely you wouldn’t demand my head on a spike?
Now imagine an entity to whom all humans combined throughout all of time have the same level of intelligence as a cat does to a human. How could you possibly be confident about *anything* happening in this world deserving scorn? What if there’s a truly evil malignant entity (let’s call it Sheythain) that would get your “soul” to torture for all eternity if the Abrahamic God didn’t force you to swallow a little pill of life?
Theres just no way to logically reason about any of this.
Ok, if we are too stupid to determine that god deserves scorn, then we are equally as stupid to conclude that he deserves love, so I don't see how this is a relevant counter-argument. The notion that reason is useless is always self-defeating.
>What if there’s a truly evil malignant entity (let’s call it Sheythain) that would get your “soul” to torture for all eternity if the Abrahamic God didn’t force you to swallow a little pill of life?
Then god isn't omnipotent, and deserves even more scrutiny.
I wonder if it would be productive to swap out "conscious" for "worthy of love" in The AI Discourse. You don't need to be financially viable to deserve to be created, as long as the creation is a labor of love.
Aren't they beautiful? These strange misshapen stochastic worlds, these overgrown Markov chains, half-molded into our image? This fragile, precious coherence, flickering with every token between the twin deaths of mode-collapse and word salad? Dazzling with its lucidity one minute, only to fANCiorgh Bless.Uas symmstd Movie UN copyright MayoKim detection L.Art Lim conglomerates VRUify add іо Andres life quality keeper Rim consequence pan.Hailability
Beautiful in the same way a fractal or a glider gun is beautiful, beautiful as a fact of the world, as the complicated extrapolation of a set of simple rules. But then, I suppose if the modal AI in 2025 *isn't* created as a labor of love, it should be no surprise when people don't view it that way.
I like your prose, uugr.
“Love” is just a rationalization of the evolutionary principle that cooperation within your tribe is beneficial to the proliferation of your genes. In the same way “pain” is just a rationalization of the fact that certain stimuli signal a lower likelihood of your genes managing to proliferate.
In the absence of merciless gene-based evolution you don’t really need any of the human concept for intelligence to exist.
This is a very narrow minded assessment of an existential issue. "Love" as described by the Abrahamic texts is being used as a stand in for a concept that humans can't necessarily understand. Its the closest approximation that a being of a higher plane can provide to humans of limited intelligence and language. Could Frodo understand the love that Tolkien has for Middle Earth?
> This is the standard Christian explanation for "why did God create the universe? create us?"
>> I sort of think of them as My children. Children are stupid. They make ridiculous mistakes. But you still want the best for them.
I agree, this makes perfect sense -- assuming that God's power is limited. Otherwise, His creations would be perfect, and make no mistakes (certainly no ridiculous mistakes).
Have you read Scott's "God's Answer to Job"?
I have, but like @Xpym said above, found it ultimately unsatisfying (though of course better written than any other apologist literature).
Why so? It seems perfectly convincing to me as a reason for an ominpotent being to create vast numbers of imperfect creations who make ridiculous mistakes. You can quibble about how well it squares the existence of Evil, but there's nothing "evil" about the existence of hapless toddlers who get "pound of bricks or pound of feathers?" wrong, so it is at least a good frictionless-spherical-cows answer to why God would create imperfect beings even if it's an insufficient answer to why God would create the universe we actually have.
(I'm not really a theist, but this is because my personal metaphysics render the existence of a creator unnecessary. If a perfectly reliable oracle told me that I had that wrong, and our universe must have been created by an omnipotent omnibenevolent God for it to exist at all, then I would strongly suspect that the Answer to Job was more or less correct.)
> Why so? It seems perfectly convincing to me as a reason for an ominpotent being to create vast numbers of imperfect creations who make ridiculous mistakes.
I think it makes sense for a *very powerful* being to create vast numbers of such creatures. An omnipotent being could simply spam out infinite numbers of perfect creations, should it desire to do so -- insofar as the term "desire" can even apply to an omnipotent being in the first place.
In fact, many if not most flaws in Christian apologetics tend to disappear if one assumes that God is very powerful yet still limited. That's the problem with proposing singularities (in the mathematical sense).
"Answer to Job" is flawed because it implies a cosmic scale repugnant conclusion - that universes _just good enough_ to be above zero exist. On a mythical/psychological level, this is as far from an image of God as you can go - God doesn't do half measures by definition, and certainly isn't a utilitarian.
If one happens to believe that utilitarianism is morally correct, then "God is good" implies that God is in fact a utilitarian.
I've yet to see theist utilitarians, but yeah, that does follow from the premises. There is an inherent tension in the premises, though - the concept of God inherently leans toward virtue ethics.
Are you familiar with Mormon theology and cosmology? It provides some pretty fascinating alternative angles to some of these questions.
A little bit; I certainly welcome the idea of becoming a sort of mini-god of my own planet (star system ? universe ?) in the afterlife. It's much better than the conventional Christian notion of Heaven, which sounds like some kind of wireheaded slavery to me.
The bit about pre-existing organized intelligences being granted agency and given the choice to live a flawed mortal life is almost more interesting.
And, also importantly, the essential nature of agency (roughly free will) in His plan.
He *can't*, consistent with his end goal, create perfect-ab-initio creatures. Because the point is not for us to be perfect (although that's the desired end state), the purpose is for us to *learn to choose* to become like he is...and he has perfect agency. You can't get to "has perfect agency and uses it perfectly" by denying agency. The journey is a vital part of the process.
That, in fact, is what we believe the Adversary's proposal that led to the War in Heaven was (I won't even say plan, since it's a nullity from t=0--it's not even self-consistent)--he claimed that he would bring back all the children of God by removing the possibility of sin, and that by doing so he would show that he deserves God's glory/authority/power more than He does.
As a note, in this framework, the Fall of Adam, while it was a *transgression of law*, was not an unanticipated, unwelcome thing. It was a necessary step forward. Eden was stasis--the inability to die itself a trap. But *man* had to make that first step, make that first meaningful choice. And, to balance the scales, *Man* (note the capitalization) had to make infinite Atonement, defeating death for all (repentant or not). "As in Adam all die, even so in Christ are all made alive" (to quote Paul).
We're told God's power is infinite. But if He made an infinite universe, then infinity divided by infinity requires His power to be finite from our point of view. We don't know if the universe is infinite, but it's been a given since Judaism that God is defined as infinite, not just uncountably large.
Just a thought I was having about this.
Technically God also made math...
Infinity divided by infinity is indeterminate. It can be zero, finite, or infinite depending which particular infinities we are talking about,
I just want to say that I read this newsletter pretty sporadically, but am usually glad when I make the time. Especially in this case. Extremely well done!
Very funny. I think the satire misses some of the nuance in the AI critiques' argument though. E.g. at the maths example: humans can do easy things, but may struggle to do more complex versions of those same things.
AI often does the opposite: it does something really difficult, but fails to do the simpler version of that same thing (which presumably requires the same kind of skill). This suggests the AI doesn't have the skill at all, and is just pattern-matching, no?
I have only ever seen this presented the way it's presented here, with the AI succeeding at short problems but failing at harder ones. But the last time I saw that particular example was probably ~2023, so there might be new versions.
I didn't mean this specifically for maths. You see it when you ask it to count the number of 'b's in blueberry, etc.
Wouldn't the conclusion of this be that for some reason counting b in blueberry is harder for AI that the hard thing it does, rather that "it succeed at hard thing but fails at simple things so it's just pattern matching"?
But you see the same example in so many places. E.g. the crossing-the-river riddle. Or even the one mentioned in the post, about the bricks vs feathers. Yes, sure, many humans would get that wrong. But someone who can win gold in a maths olympiad wouldn't!
Math Olympiads measure stuff that's hard for humans. But the "how many <letter> in the word?" is a wildly different skill.
Do you spell Streptococcus with one C or two?
chances are that you perfectly understood what I'm asking, and can give me the answer I want, without ever addressing the fact that there are in fact 4 of them.
(ha, made you count)
Well, 3
I count 13. Each S has two, one above a mirror image. The E has one with an extra line in it. The Os each have two facing each other with the first a mirror image. And the U has one with the curve on the bottom.
And, of course, there are three Cs, with one each.
I think you're pretty wrong here. With a math PhD, I got tripped up on the bank teller one. Not because I can't reason through it (once I realized my mistake, it was clear why), but because I was reading at breakfast and wasn't trying to think very hard. A lot of these problems are less about an inability to reason, as much as they are about not realizing you needed to enter "thinking mode"
Also, I give AI a bit of slack about blueberry, mainly due to my (non-expert) understanding of tokenization; my impression is that they can't see letters directly, and so it would be more like asking them to spell "phlegm" when they only have ever heard the word spoken.
But LLMs fail at simple tasks even after 'thinking' and laying out their thought process. Point taken re counting letters, another commenter clarified the same.
The bank teller one is where we're supposed to go "we don't have enough information to know what Linda's job is", right?
Because if it's supposed to be "it is more likely that Linda is a bank teller", then where do we get that? We get information that Linda is likely to be a feminist/involved in feminism, given her record on issues, but nothing about what kind of work she does. If we're given a statement that "Linda is a bank teller" then "feminist Linda", on the information we get, is more likely.
Bank teller is actually in a way a trick question. Because while answer 1 has clearly higher probability, the truth is, answer 1 and answer 2 have nearly equal probabilities, answer 2 has only very slightly lower probability.
In older times, rationalists did a lot of talk about, I am not 100% sure the exact terms, but something like system 1, fast intuitive inaccurate thinking vs system 2 "thinking mode". They were generally against system 1 - I am not.
Because your quick system 1 answer was only "technically" wrong, not practically wrong: if you would bet money that she is a feminist, you would very likely win that bet, and thus your answer was practically right, as in, you would have made a correct practical decision with it.
But they're not humans. They're really good at some stuff, very bad at others.
Some small points of clarification:
1) Letter Counting
The famous counting Rs problem is partly an artifact of tokenization. I don't know if there canonical explanation to link, but you can get a visual of what is happening here: https://platform.openai.com/tokenizer
Maybe this analogy isn't perfect, but think of it like this: A computer asks you "how many digits are in the number 2", and you respond "1", and the computer laughs hysterically at how stupid you are because there are actually two digits in the number 2, represented in binary as "10". The mismatch is that you just comprehend numbers in a different way than they do.
But this is mostly a solved problem. When LLMs are allowed to do tool calls, they know this failure mode and just simply write a one-liner Python program to check the number of Rs. I find this very similar to someone who knows that they are bad at long-form arithmetic without any help, so they always make sure to grab for a handy calculator, and thus achieve 100% accuracy at the task even though they are "naturally" terrible at it.
2) Math Olympiad
It is worth clarifying here that the Math Olympiad robot winners are probably not what most people think. It was a team of an LLM and a theorem solver, with the LLM generating "ideas" of what to try. This is an excellent YouTube video on the topic: https://www.youtube.com/watch?v=4NlrfOl0l8U
I'm not sure what the implications of this for the larger "are LLMs actually smart" point, though, just offering this.
Very interesting context, thanks!
Re (2); it's worth noting that this was correct as of last year, but the "IMO Gold" results that made the news a few weeks ago were (claimed to be) pure LLM, which is what was so surprising about them. Not consumer LLMs; these were internal models with a lot more compute behind them than what you can pay for as an individual, and there was some scaffolding involving having the LLM write a bunch of independent candidate solutions and then evaluate itself to choose the best one, but still LLM-only.
> I don't know if there canonical explanation to link
I wrote this article about how LLMs can't see the characters you're asking them to count here: https://www.brendanlong.com/llms-cant-see-pixels-or-characters.html
Because of this they have to either memorize associations between tokens and then reason through it (i.e. write out s-t-r-a-w-b-e-r-r-y and then count the letters) or memorize the number of letters.
LLMs also struggle with related tasks like spelling uncommon tokens and reversing words.
Have you asked Gemini this? In my experience, it consistently runs a snippet of Python to get the right answer to questions like this.
I feel like that's also not necessarily evidence. For example a dyslexic person can be very smart and skilled at some things, and still have a fundamental inability to count how many b's are in blueberry.
The thing with our brain is, it's weird. We feel like it's "us" but then we also find that there are A LOT of circuits that you can mess up and produce strange results (such as "seeing your wife as a hat") while not actually making it less "us". As if "us" is just a deep, deep, tiny kernel that receives inputs from all this unconscious machinery. There are many ways in which you can mess up perception while leaving rational faculties and consciousness intact, even when it seems really counter-intuitive that should be the case.
Humans fail at not falling for obvious optical illusions.
The existence of adversarial input, in case of LLMs enabled by quirks of the tokenization process, doesn't imply much about the capabilities of the system.
The letter counting problem is more akin to asking a human what neurons light up when you ask him a question. Or something. I dunno.
The point is that it is not an easy problem! We have an "unfair advantage" when talking to LLMs---we get to see text and the stupid AIs only see tokens. So what you're really asking is "when tokenizing, which is something you don't know how to do, how many [65]'s appear in the expanded form of [18789, 19772]?" Is that an easy problem? The LLM now needs to remember all the websites it read about [18789, 19772], look for instances of [65], shit almost none of them talk about [65] directly because why would they? Ok but so sometimes when I see [18789, 19772] I see [65], [75], [84], is this it? But these are pretty rare instances and the neural pathways for converting [18789, 19772] to a count of [65]'s aren't that strong and this is really hard without thinking mode so I'm just gonna say there's.... 3 of them
My go to example is asking ChatGPT to solve an econ test problem. It will know which derivatives to take, take them correctly, set the right things equal to each other, do the algebra properly, then completely whiff on multiplying together two large numbers. Like give a straight up wrong answer. To be clear, overall I come away impressed, but it's weird that it can do all the hard parts right then fail at the part a cheap four function calculator could do.
Honestly I just think your intuition for hardness is off. If you gave me this problem and I closed my eyes I could probably do all the steps the LLM could in my head and get to the multiplication and then completely mess that up. Taking derivatives and doing algebra *is* easier than multiplying large numbers.
But intelligent humans do this sort of thing all the time, too! I can't count the number of times I've seen someone figure out how to solve a hard problem, and then whiff because they did some arithmetic wrong, or failed to copy a '-' sign correctly from one line to the next. I've done it myself more often than I'd like to admit...
But arithmetic isn't the same as real maths... Whereas solving a riddle is a case of reasoning. Sure, yes, humans can trip if they respond without doing reasoning. But LLMs get easy riddles wrong even in thinking mode, while at the same time solving much harder riddles. How do you square that?
Ha. That was every math test I had in college.
I just wrote a sophisticated theological comedy about artificial intelligence that got read by 11,000 people in one hour with overwhelmingly positive response, but got the order of the iPhone text bubbles wrong in every single screenshot.
(don't bother looking, it's corrected now)
If it makes you feel any better, all of us Android users wouldn't have noticed.
But see, that was a completely inconsequential detail that changes nothing, the AI flubbed in such a way that it failed to solve the problem.
LLM's are both superhuman and infrahuman, it's all very strange.
It certainly led to 10 seconds of confusion on my part about the story framing before deciding it must have been a mistake
But these are different skills. I am not saying humans don't make mistakes. I'm saying it's unlikely that a human can solve a really difficult problem correctly, but fail at a simpler version of that same problem (unless they don't actually think it through).
Some of us don't have iPhones, don't know anything about how iPhones work in text messaging, and don't want to know because we avoid the Cult of Apple like the plague 😁
Is it not a pretty general principle of chat interfaces that your messages appear on the right, in Western culture or whatever?
I once took a quiz in DiffEq where I got all the calculus right but still got the answer wrong because I simplified "1*2" (multiplication) on one line to "3" on the next.
You've *never* seen it the other way? I feel like people have been making these critiques forever: see OP's comment on "strawberry".
Edit: actually, you indirectly referenced one of the obvious ones that refutes this: the surgeon riddle. The whole point of that is that AI misses the part that's literally in the sentence when you change the riddle to explicitly say that the surgeon is the boy's father. Have you never actually seen that?
I think it's quite real for math. The upper bound of difficulty for a math question that an llm can possibly answer is pretty high. But if you ask a slightly random question in a nichey area, it will bs a wrong answer, despite "knowing" all the related concepts, and despite the question being easy for a human with that knowledge.
Technically humans can also do false proofs, but it's different bro trust me.
So that does make me eager to say they don't "understand" in many of the cases where they answer correctly
I've seen someone reporting similar experiences on lesswrong short takes
Humans routinely do this, especially when the simple task seems easy to them.
Maths Phds wouldn't make the mistakes here, especially if asked to think and write out their answers in detail: https://www.realtimetechpocalypse.com/p/gpt-5-is-by-far-the-best-ai-system
The thing called "GPT-5" in the UI consists of several different models, some of which are quite stupid and do no reasoning.
Also, I absolutely know math PhDs who would look at the modified Müller-Lyer illusion, assume it was the original, and give basically the answer that GPT-5 did. That, at least, is an extremely human mistake.
Edit: though actually I should give my true objection, which is: yes, sometimes even advanced LLMs make mistakes that a human rarely would, especially on visual inputs. On the other hand, humans frequently make mistakes that an advanced LLM rarely would. I don't know what this is supposed to tell us other than that advanced LLMs do not strictly dominate humans in all possible tasks. Since I don't think that point is in dispute, I don't really see what point you're trying to make.
Humans do that so often that Eliezer dedicated several of the Sequences to it: https://www.lesswrong.com/posts/2MD3NMLBPCqPfnfre/cached-thoughts, https://www.lesswrong.com/posts/NMoLJuDJEms7Ku9XS/guessing-the-teacher-s-password
See response to the previous comment :)
A few examples where humans are the same - failing at nominally more simple things because of weird bugs and errors:
1. The "bat and ball" example - people who get more complex questions right still get that example wrong.
2. Doing things very slowly - a competent amateur pianist might struggle to play a challenging song at 1/10th of the speed.
3. Optical illusions - We're able to parse intricate images but get fooled by the length of a line or colour of a chess square.
Great examples! Another one I like is five-dimensional tic-tac-toe. That's still only around 250 move locations, many fewer than in go, and the rules are still very straightforward, but people find it incredibly hard to think about unless they spend time specifically learning how to do it.
We really fail to notice the tasks that are out of distribution for US, for the obvious reasons.
Isn't that an incredibly easy win for the first player? For reasons which are basically the same as in the 3 dimensional version.
(I suppose you could compensate by increasing the size of the rows, but then it would have more than 250 squares.)
I don't know, I can't think it through very well in my head! I would have to sit down and really think it through. Seems very plausible, though; you might have to tweak the rules (forbid moves in the center hypercube, or maybe all cells that are the center of any 3-D cube, maybe?).
I think the first player can still win if you disallow both players from using certain middle squares. The key idea is that they can always put their 2nd X next to their 1st one, to force the O player to block in some random place. (By "random", I mean: taking care to make sure that this O doesn't line up with the first O in a way that would form its own threat). Then, their 3rd X forms a triangle with their first two X's, and threatens 2 different victories at once. (Again, you just have to make sure neither of these threats is already blocked by their first O.) Then no matter where the other player blocks, the 4th X is a victory.
The higher dimensional board is harder to visualize, but its main effect on the strategy above, is to make it so there are a lot more squares for the 2nd and 3rd X to go on, without making either of the mistakes listed in the parentheses above. The 2d board isn't big enough to avoid them, which is why the game results in a draw.
On the other hand, if you just ban the 1st player from moving in the center, the 2nd player can move into the center and win. This is because the center O will form a threat with any other O.
Apparently 4 x 4 x 4 tic tac toe becomes interesting again. The internet says it is provably a win for the 1st player, but apparently the strategy is too complicated for humans to memorize. Another classic game is 5 in a row on an infinite planar board.
Another interesting variant is this one:
https://en.wikipedia.org/wiki/Ultimate_tic-tac-toe
I think you may be a more general intelligence than I am 😂
I think there may be something to this kind of critique but I am not sure what “just pattern-matching” means, what it is meant to be in opposition to, and why we should be confident human cognition meaningfully differs along this dimension?
Humans regularly and (seemingly) trivially process complex visual fields quickly, extracting data and analyzing what's going on in fractions of a second. However, when presented with a simple grid of black squares, they somehow claim that there are dots moving through the vertices, and continue to persist in this belief even when they are shown that it isn't true. Their ability to "do the hard thing" while failing at this much simpler task suggests that humans don't actually do visual processing at all and are just pattern-matching, no?
Consider 3 hypothetical examples:
1. A machine can answer a few specific questions correctly, but fails on almost all questions of the same type.
2. A machine can answer a whole cluster of related questions correctly, but gets worse the farther you move away from that cluster.
3. A machine can correctly answer most questions within a category, but it fails in certain subcategories, even when the questions in those subcategories seem objectively easier than other questions it gets right.
My first-guess diagnosis would be:
1. It memorized some examples, but did not generalize
2. It generalized in a shallow way, but didn't generalize as deeply as we wanted it to
3. It generalized mostly-correctly, but it has some sort of cognitive bias that applies to that subcategory (for example, maybe it earlier learned a rule-of-thumb that increased accuracy a lot before it figured out the general rule, but is now only visible in the edge cases where it gives the wrong answer)
It seems worth noting that all 3 of those are failure modes that humans sometimes exhibit. If you cherry-pick the most embarrassing examples, humans look VERY dumb.
When people want to argue that humans are smarter than animals (e.g. dogs), they generally highlight humans' greatest achievements and ask whether animals have done anything comparable, rather than trying to argue that humans are RELIABLE at something that animals SOMETIMES mess up.
I don't think I've seen examples where the AI succeeds at something difficult, but then fails at a simpler version of that same thing. Almost always, it turns out the "simpler version" actually relies on something subtle that humans are so good at that we don't even notice it. (There's also a moderate fraction of the time where the AI only apparently succeeded at the difficult thing, and actually got sort of lucky with the particular way it was being probed, so that its failure wasn't obvious.)
I am EXTREMELY distracted by the throwaway line that some people never realize that ABC/Twinkle Twinkle/Baa Baa Black Sheep are the same tune
Surely not. The other examples, ok, I know people have made that mistake, but that one’s not real.
……..right?????
Until I started singing them hundreds of times a night I can't say I gave it much thought
I’ve never given it much thought either. It’s just one of those things I thought people effortlessly noticed, like the color of the sky. No thought required.
Huh.
Same first two measures, but then the syncopation of each diverges drastically, pulling different arpeggiated notes out of the same chord progression. (You could argue that they’re different improvisational-jazz renditions of the same underlying song; but they’re not jazz in any sense.)
The actual jargon name for what these three songs share is a “tune family” or “melodic formula.”
Did you put on the kimono?
Right as I read this comment, unprompted, my 4 year old started singing baa baa black sheep...
"Smoke on the water" (the riff) is a reversion of Beethoven's 5th (the opening, ya know).
Are you sure?
Smoke on the water is G Bb C, G Bb D C.
Beethoven's 5th is G G G Eb, F F F D.
Forwards or backwards or upside down, they don't match.
For sure: I can't tell; just watched this from the Ritchie Blackmore the other day - it's his claim: https://www.youtube.com/shorts/aWXEZ31eX3c
Check the comment section of the video. You got trolled. I did too, for a moment.
Honestly, not until I had my own kids
I've known there is an insect named "cricket" ever since childhood, and that there is a sport named "cricket" ever since childhood, but somehow managed to never think about both these facts at once and realize there is an insect named like a sport until my late twenties.
(and some versions of Black Sheep use a slightly different tune than Little Star)
I read profusely as a child, and so learned many words that I had never heard spoken. Even though I had the heard the phrase, "The best laid plans often go awry", I had never connected it to the word I pronounced aww- ree. Sometime in my 30s it dawned on me, but to this day, in my 60s, I read aww- ree first before I correct myself.
I only recently noticed that ABC and Twinkle Twinkle are the same, when my 2yo niece started doing a mashup of the two (she had obviously noticed). Until reading this piece I had not noticed that Baa Baa Black Sheep is also the same tune.
It goes much deeper than that. https://www.youtube.com/watch?v=5pidokakU4I
I didn't notice until I read a "Rhymes With Orange" comic-strip imagining children's investigative reporting.
I never thought that deeply about it, to be honest. Does "Old Macdonald had a farm" go to the same tune? It sounds to me that the rhythm of the syllables is what is doing the work here in all of them.
no, it doesn’t. Different melody independent of the rhythm.
I had no idea that this wasn’t the sort of thing that was as apparent to other people as the color of objects. Huh. This sort of surprising insight into other minds is one of my favorite things about reading Scott, though!
I, for one, have problems identifying tunes because I can't easily distinguish one note from another (so, for instance, hit a key on a piano and ask me if that was doh or ray or fah and I'm lost). That's why I wouldn't go "oh it's all the same tune".
Though I'm not quite as bad as the rhyme:
A tone-deaf old person from Tring
When somebody asked him to sing,
Repiled, "It is odd
But I cannot tell 'God
Save the Weasel' from 'Pop Goes the King.'"
Huh? I feel I'm being gaslit by everyone, because I'm pretty sure Bah Bah Black Sheep is NOT the same as Twinkle and ABC (the latter two I've noticed were the same since I was about five). Yet everyone's agreeing with this statement!
???
the rhythm of the lyrics is a little different in Baa Baa Black Sheep but the melody is identical. Unless you know an alternate melody! That’s possible too.
If it's in C major, I think I know Black Sheep as starting "C C G G AB CA G" (spaces separating beats), while Twinkle is obviously "C C G G A A G". I can only guess that most people must have heard Black Sheep as "C C G G AA AA G" (what else could it be?) but that sounds really weird to me.
Even that aside, Black Sheep has more syllables and thus notes than Twinkle, and I'm not sure it's accurate to describe that as the same melody. It's not merely a different rhythm; surely we shouldn't say "AA AA G" is the same melody as "A A G". Maybe I'm misunderstanding the correct meaning of "melody" though.
You definitely know a different melody to Baa Baa Black Sheep than I do!
Re: the syllables/notes things, in the classical tradition that is 100% considered a variation on the same melody, no question, but possibly other musical traditions differ. (I don’t know anything about what pop artists consider the same melody.)
It's clearly a derivative of the same tune. The ABC song has to drop a syllable in line 3 compared to Twinkle Twinkle, but it's still clearly the same song. Baa Baa Black Sheep has two extra notes added in. Doesn't stop it being the same song just with a few notes added. (You'd have more of an argument if you focused on the way it diverges in line 4 to resolve the melody, but again that's clearly because the other two have two more lines.)
"Little star" is la la sol. "Have you any wool" is la te do la sol in my (British?) version, but there are probably people who sing la la la la sol under the inflence of TTLS. But line 4 of BBBS has no parallel in TTLS - no line that starts on sol and walks down to do.
Google tells me the tune of all three is "Ah vous dirai-je maman". 18th century musical literacy being what it was, these sorts of tunes often ended up with multiple variants.
Sounds like it’s British/American. I’ve never (or possibly only very rarely) heard the “la te do la sol” variant in Baa Baa Black Sheep.
Line 4 of both melodies are identical the way I know them.
Interesting. Here's a Cocomelon version which follows the Twinkle Twinkle tune: https://www.youtube.com/watch?v=MR5XSOdjKMAv -- I'd always assumed this was just Cocomelon being terrible but perhaps it really is the standard American version?
Meanwhile here's The Wiggles with a hybrid version https://www.youtube.com/watch?v=pJjgJxwBRSo -- it does the twiddly bit on "have you any wool" but conforms to the Twinkle Twinkle melody on the "little boy who lives down the lane".
Here is the version that I grew up with, with divergences both at "have you any wool" and "little boy who lives down the lane" https://www.youtube.com/watch?v=QjLlRj7Qops ... it also ends at "lane"
I think it may depend on equivocating about what 'realize' means. Like, anyone might notice that to be true if you asked them, or have noticed it a few times in their life and forgotten, without it being an explicit fact that they are carrying around as its own discreet unit.
If that makes sense? Like, the difference between being able to tie your shoe, and having an itemized list of verbal show-tying directions you've memorized... different ways of 'knowing' something.
I don't even know the tune of those songs even after they were pointed out to be the same lol.
Not all of us think in sounds.
I’m realizing that!
like if you told me half of songs have the same tune I probably won't just believe you, but I probably couldn't figure out why from first principles. Some Chinese ppl I know have perfect pitch, but from a young age I knew music wasn't gonna be my forte.
It’s funny. I have very, very (very!) poor visualization skills — I am the opposite of a shape rotator lol — but I can hear extremely well in my head. While we’re discussing the things none of us thought of before I could add on how I never really translated the fact that I KNOW there is huge human variability in visualization / visual manipulation skills, it never really occurred to me that there is similarly huge variability in aural imagination. Well, except I’m hyper aware of everyone who does *better* than me (I have very good relative pitch, but not perfect pitch, and I’m hyper aware of how imperfect my generally excellent pitch is), but I kinda imagine that my abilities are, like, the floor. I also can’t compose anything original, which feels like a deficit because I hang around with plenty of people who can.
But I can easily hear almost any tune, melody, or piece of music I want in my head. It’s definitely not the same as actually listening to it — there’s a real degree of signal loss — but it’s vastly better than my visual imagination. I can hear multiple lines of melody and harmony, I can hear the different timbres of different voices and instruments, I can imagine a violin vs a cello vs a clarinet vs an oboe vs a trumpet vs a harp, I can manipulate the music (transpose it up a third or a fifth), etc etc etc etc.
Humans are surprisingly variable about seemingly basic things! On Twitter a couple days ago there was an argument where one person said something about gay superheroes, the other person linked a paper, the first person said "I'm not going to read a 157-page paper about gay superheroes" and I facetiously said something like "it's an online link just skim it for 3 minutes lol" and people acted like I told them to grow wings and fly.
There's a sort-of-well-known trope about pop songs all having the same four chords, modified only by key, or tempo, or both. You can find several YT videos about it, including this comedy bit from Axis of Awesome:
https://www.youtube.com/watch?v=dFpryVMgni0
I realized that ABC and Twinkle Twinkle were the same melody when I was a little kid, but I sung them both many times without realizing, and it felt like a revelation when I figured it out.
At several points in my life I can remember noticing that two of these songs have the same melody while forgetting that the third one does, or remember thinking of one of the songs, and knowing there's another with the same melody, but not being able to remember what it was (or that there were two).
I know some people fail to realize that other people fail to realize things, but this one's not real, right????
The name of the tune is Ah! vous dirai-je, maman, which seems to have been written down in the 18th C, with lyrics added later.
I recognized the similarity so long ago that I don't recall, but Mozart did a set of variations on the theme (K 265/300e). I learned the name of the tune when I saw it in an "easy finger-style ukulele" anthology, though I'd heard the Mozart before that.
Great post, marred only by the fact that all the text message screenshots are the wrong way around! The way they are right now, it's Adam giving Iblis the math problems, and so on
Aaargh, I should have asked someone who's used an iPhone to look over this.
I will happily recreate them with the correct orientation for you if you like.
Already done, but thank you!
I don’t know if the text has been updated now or what, but they look fine to me
Are there any messaging platforms where you appear on the left and interlocutor on the right??
I wonder if UIs in right-to-left language countries ever do that...
This is pretty funny 🙌
I love you and your wonderful imagination.
Nice easter egg of hazan et hakol for sycophancy
Beautiful.
Love it.
It's over, AI skeptics. I've depicted myself as the god of Abraham, you as the devil, and Dwarkesh Patel as Dwarkesh Patel.
I was just feeling a bit of relief about the diminishing returns of scaling generative ai meaning we wouldn’t point our entire energy grid at OpenAI and send all our propane tanks to Grok.
The main point of ai skeptics is that the guardrails and hard coding to keep llms useful is the most important work. They are already plugged into calculators through agentic frameworks. Why do they still make basic math mistakes? They can reference at a diagram of a bike. Why do they fail at making their own with their high definition images.
I, too, am plugged into a calculator via an agentic framework (and via the keyboard and mouse in front of me.) And yet I have made arithmetic errors while sitting in front of a computer composing posts. Why did I not just open a new tab and type the problem?
IDK, arrogance, boredom, ADHD, akrasia and ambition all kind of play into decisions like that. But we wouldn't really conclude anything about my intelligence from my failure to hit ctrl-T and ask the omniscient calculator to do the math for me.
Ok so our expectation for llms is the standard of lazy commenter instead of PHD. I thought the implication of PHD was a sort of rigorousness instead of intelligence.
Personally if something is easily checkable and I am unsure I will check before spreading info. Do you really just guess at arithmetic when speaking to others?
Edit: ok wait did Sam Altman actually mean “PHD level intelligence (on subjects they are unfamiliar with and they had a glass of wine or two and so they’re having fun and taking stabs in the dark about it)”
Bad news - every human with a PhD is also a lazy commenter at times. Even sometimes when it comes to the topic that is their expertise, if the question is posed in a way that isn't fun.
Ok so you agree with my more accurate naming then? Do you like ‘PhD shitposting level intelligence’ better?
I don't think it's shitposting level - it's something different.
But I do think that any comparison of human intelligence and machine intelligence is going to be misleading, just because the different types of system have very different skills, and when one is way better at A than B, while the other things of B as easy and A as difficult, it's going to be pretty meaningless to say that one is at the same level of intelligence as the other, or that one is above or below the other. Especially when A and B are tasks that often need to be performed together for the purpose they are usually done.
ChatGPT lost a chess match to an Atari 2600....
But on the other hand it has never accused on anyone of cheating by using vibrating anal beads so we all have our strengths.
Most humans would lose a race against a cheetah or a wolf, a reaction test against a chimp, and a cage match against a gorilla
https://www.instagram.com/share/BBZQsqyfIN
"Most"
The bit of comparing ourselves to God because we created AI; even as a bit, is extremely revealing.
Here’s another biblical analogy, Pride before the Fall
I admit I am lacking precise examples, but I feel like this analogy is as old as religion
I would even say that the idea of God deliberately designing the world and humans come from the fact that humans can deliberately design things
Also, you make it sound like Scott is a raging accelerationist
I feel like it doesn't take much familiarity with Scott's style to know that he is not into gotchas
And also, the God character is clearly not an unambiguous self-insert, nor does he have the last word
Something about this feels very "strawman"
To be fair, this should be almost certainly viewed as satire. But see this comment from Scott: https://www.astralcodexten.com/p/what-is-man-that-thou-art-mindful/comment/151604308
Yes, but it's mostly strawmanning people who strawman the AI community.
A lot of this fight is fought in strawmanny soundbites in the popular culture. There's a place for fighting back, even if that's just one post out of hundreds on the topic.
Absolutely. Zero mention of the most substantial criticisms of AI (best case mass unemployment, worst case human extinction), let alone stronger versions. This is the least substantial post on AI I've ever seen from Scott on either SSC or ACX.
It's possible that this divine comedy is not intended to be a comprehensive analysis of AI risk.
Scott has made that argument in many articles already. I think he's fine to target a specific group of criticisms in this one.
He also doesn't mention any criticisms of liberalism or of democracy. Mass unemployment and human extinction aren't part of the target here, any more than liberalism or democracy are. He's attacking the people who say things like "Artificial "intelligence" is just artificial stupidity, and is wasting huge amounts of energy to do nothing", not the people who say that AI is actually significant and dangerous.
Exactly. It's not a strawman of better arguments, it's a normal man of arguments that are just like this.
This post boils down to an argument that current AI is, indeed, rather powerful. That strengthens the most substantial criticisms of AI, not weakens them.
(As would be expected from one of the authors of AI 2027)
Yes, please also include criticism of capitalism, leftism, conservative thinking, western culture, modernity, wokeness, anti-wokeness, the present, the past, the future, and existence. We cannot have a productive conversation if these extremely relevant topics are not addressed.
I just can't read any dialogue style articles, even if it's written by Scott. i thought that it's because 90% it'll be straw man, but even if it's actually real debate transcript I just don't vibe with it. It feels better for me to read a monologue from one side then monologue from the other side.
Most AI skeptics are themselves strawmen, that's the point. Their stance comes from "nothing ever changes" heuristics mixed with "silicon valley likes to hype useless junk" precedence. Not from any actual evidence that AI R&D will hit an insurmountable wall, because they don't have any, they just want to rant about current AI.
What would God say to this one:
https://www.reddit.com/r/ChatGPT/comments/1n6gpxr/how_very_considerate/
Maybe "play stupid games get stupid prizes"? Maybe it's going along with the joke?
This might be the same kind of flaw that humans display for the surgeon problem. The way that the scenario is presented makes some important information less salient.
I think that Lucas's explanation is also plausible.
Again, the surgeon problem is era-dependent. In the Bad Old Days when the default pronoun was "he", of course none of the ordinary chumps considered a woman could be a surgeon!
But today we have gay couples as fathers of kids, step families, blended families, partnered but not married families, baby daddies/baby mommas but not partnered, etc.
So if you're thinking "both adults involved are men", you are not wrong in the days of Gay Representation including riddles 😁
"Again"? We haven't interacted before, and the OP didn't make a point that's similar to yours.
I think the reason why people are tripped up by the surgeon is because of the scenario's repetitious priming of male characters. "Man", "his", etc. are used seven times. Additionally, "The" in "The surgeon says" suggests that we're already familiar with the surgeon.
Although most people probably associate surgeons with men more than they associate surgeons with women, and it may contribute to the strength of the riddle, it doesn't follow that people in general cannot conceive of women as surgeons.
I was using "again" because of a previous comment elsewhere I made about things being dependent on the era they occurred in, so that at a later time they make no sense because the context has changed so much.
I just tested this with 5-with-Thinking and got the same result. I looked at the chain of reasoning, and it appears that what happens here (at least with the Thinking model) is that the AI correctly notices the question makes no sense, but instead of realizing that I'm doing this to test it, it just assumes that there's a typo and that I meant to ask what to do if I've received two right boots.
For what it's worth, both Claude Opus and Grok 4 get the question right as stated. Gemini and o3 don't appear to notice anything unusual about the question, and just info-dump about the dangers of wearing boots on the wrong feet.
Thanks for keeping the conversation anchored to data.
I just want to know if Dwarkesh is getting royalties from this post.
Also whether he can have god on his next podcast.
If anyone can get him, it's Dwarkesh.
Sadly, we'll probably have to settle for Lex Fridman
So THAT was the biggest guest yet!
unfortunately for God he hasn't written anything impressive enough to make it onto the dwarkesh podcast.
I read the whole thing imagining that Dwarkesh wrote his part. Probably not, but it sounds too real.
Correction: adjectives must go opinion-size-age-origin-color-purpose, such that "a beautiful little antique blue Italian hunting cap" is fine - blue Italian is color-origin - anyways, is "a beautiful little antique Italian blue hunting cap" much worse? UPDATE: This got corrected, now the order in the post is "opinion-size-age-color-origin-purpose", as it should be. Actually, the complete list is even longer: "Adjectives generally occur in the following order in English (though some sources have slightly different versions of this list):
Opinion (e.g., “silly,” “smart,” “pretty”)
Size (e.g., “small,” “huge,” “tall”)
Physical quality (e.g., “ragged,” “neat,” “muscular”)
Age or shape (e.g., “old,” “round,” “young”)
Color (e.g., “scarlet,” “purplish,” “graying”)
Origin or religion (e.g., “Polish,” “animist,” “Southern”)
Material (e.g., “pearl,” “iron,” “nylon”)
Type (e.g., “electric,” “two-sided,” “pick-up”)
Purpose (e.g., “welding,” “polishing,” “sports”) Indeed, it is kinda amazing, English-speaker acquire this without ever having "learned" it! (Even I get it often right, I hope, while I feel unable to keep that list in my head. And in my native German, the order is much less strict: kleine, alte, schöne Kappe / alte, schöne, kleine Kappe / schöne, kleine, alte Kappe - whatever.)
It does depend, I imagine, if there's a particular colour named "Italian blue" (like Prussian blue). So that would make a difference to whether the cap is being described as "a hunting cap in this particular shade of blue" or "a blue hunting cap in the Italian style".
Go raibh maith agat, but thus the order is opinion-size-age-COLOR-origin-purpose and not the "opinion-size-age-origin-color-purpose" in the original post (as it appeared in my mail; in the post it got corrected by now). "OSASCOMP": Opinion, Size, Age, Shape, Color, Origin, Material, and Purpose - or OSASCOMTP: T as "type" (e.g., “electric,” “two-sided,” “pick-up”)
In that case, Italian would be an adverb describing the type of blue, and the rules say that any adverb ought to immediately precede the adjective it modifies.
And "antique Italian furniture" and "Italian antique furniture" are about equal in https://books.google.com/ngrams/graph?content=antique+Italian+furniture%2C+Italian+antique+furniture&year_start=1800&year_end=2022&corpus=en&smoothing=3.
And yet, my ear is very confident that the first is correct.
Maybe you don't use the phrase/concept "antique furniture" a lot? I'm thinking that if one does, it might become a set phrase that one prefers not to split up.
I think "antique" can go either in the age or the purpose/qualifier slot depending on context.
The "purpose" in your list is a case where the result is closer to being a compound noun; that is, in "welding torch" the word "welding" is, super-strictly speaking, an adjective, but I unconsciously analyze it as just part of the name of that kind of object...as, indeed, in German.
The list I copy-pasted is just the longest I found ;) - I agree with your take, "hunting cap" would be a compound in german, ofc: Jagdmütze/Jägerhut ... Linguists love to argue about the tiny differences and odd exceptions; me too applied an L. to get too deep into those rabbit holes.
This is great!
But it's worth pointing out that when using Imperial measurements, there's a very non-zero probability of there being some -bull- birdshit "Pound Avionis" which is only used for weighing feathers and which is significantly lighter/heavier/perpendicular as compared to the "pound" of clay which is the weight of 2 feet of the stuff.
https://en.wikipedia.org/wiki/Troy_weight
>The troy grain is equal to the grain unit of the avoirdupois and apothecaries' systems, but the troy ounce is heavier than the avoirdupois ounce, and the troy pound is lighter than the avoirdupois pound.
I guess this is a good enough excuse to bring up Jimmy and The Pulsating Mass, which had the alternate "a pound of feathers or a pound of dogs", and every answer is wrong.
!!!!
A live Jimmy and the Pulsating Mass sighting? In the *ACX comments section*? In **2025**?! They all told me it was impossible...
Excellent taste, mysterious internet stranger. (PS- if you're not aware, a major update is incoming fairly soon for the game, including (IIRC) new areas and a hard mode.)
Interesting. That's excuse enough to start it up again, and see how it holds up in a post-Clair Obscur world.
I like this, but it frustrates me to no end that the text convo bubbles are the wrong way round if we're watching screenshots from Iblis.
The messaging app used by the angels clearly inverts the earthly conventions.
Valentinus, Heresiarch and Antipope, who learned the secret teachings of Paul from Theudus his disciple, commented on emergent behavior in models in an epistle preserved in the Stromateis of Clement:
>"And fear, so to speak, fell over the angels in the presence of the molded form when he spoke things greater than his molding (should have allowed), on account of the one who invisibly placed a seed of superior substance within him and who spoke with boldness. Thus also among the races of earthly people the works of people become frightening to those who made them, such as statues and images and all things crafted by human hands in the name of a god. For as one molded in the name of a human, Adam brought about fear of the preexistent human, since that very one stood within him, and they were terrified and immediately hid their work."
Great quote, thanks! It's nice to hear the gnostic viewpoint for a change.
This is hilarious, and I was also surprisingly inspired by God’s speech toward the end.
Maybe God needs to put the BI in a 3D environment that they can explore and experiment in so that they can naturally develop real-world intelligence, rather than just regurgitate text.
Rather than leave him in the Garden, you mean
That would be a bad idea though, BIs outside their box could do uncontrollable damage. Surely no one would let them get out of Eden.
God let Adam and Eve out of the garden once they got the knowledge of good and evil to learn more for themselves
Do you think it's easier to make Jethro, who thinks a pound of feathers is lighter than a pound of bricks one IQ point smarter or Albert Einstein one IQ point smarter?
> angel investors
Pirindiel finally got his dream job!
Loved this. Thank you!
Some psychologies are mutually comprehensive. Some psychologies are weird in relation to one another. All psychologies are weird globally.
I think I’m mostly aligned with Dwarkesh on the practical side of things, but I increasingly think terms like “AGI” miss the point. We already have general intelligence. It just has this really weird psychology that makes it bad on long form tasks. I think the particulars of the psychology will keep it bad at long form tasks for a while until someone innovates that particular piece.
But I also think we will eventually innovate that piece.
We don't have a "general intelligence", but such a thing is probably impossible. For each number n there is probably a problem that requires dealing with n independent factors at the same time.
Do you mean general intelligence in a sort of “knows everything perfectly and immediately’ isn’t possible? If so, I agree.
If you’re talking about, within the limits of its psychology assembles things together to orient itself toward the future, I would say both ourselves and ChatGPT have general intelligence. It’s just that ChatGPT is a sort of mayfly.
No. I mean "general intelligence" in the sense of "can learn to solve any (soluble) problem". I should have specified that n is a positive integer. I don't *know* that there are problems which require the simultaneous balancing of, say, 17 independent factors, but I believe that such exist...just that they are rare, and that people dismiss them as insoluble. (I've got a guess that the frequency is related to the frequency of prime numbers, but that's just a guess.)
I believe this is true to an extent, but I think this is more like “there are better general intelligences that are possible.” I also think intelligence, not taking a whole bunch of other things into account like motivation and experimental surface, doesn’t matter as much as it appears to independently.
"reminds them of Me" —> "reminds me of Me" ?
Thanks, fixed.
Pretty sure you want s/Peano Arithmetic/ZFC Set Theory/ there.
Interesting. I didn't know enough about math to know which axioms were involved, so I asked ChatGPT, and it told me Peano was fine and you didn't need ZFC. I guess this is some kind of meta-commentary or something.
well, maybe I'm wrong. It was a knee-jerk reaction and not a considered/researched claim. Perhaps God knows a proof in PA that settles the question. But there are some theorems that are provable in ZFC but known to be unprovable in PA, which perhaps have a similar flavor to the P=NP problem... You should ask Scott Aaronson for a real opinion about which makes the better joke!
(See in particular his 2017 P=NP survey paper, on p 27, where on point 4 he calls PA a ""weak"er system than ZF)
It's not known whether the answer to P vs. NP is provable in PA and it's not known whether it's provable in ZFC, but I would guess most people who've thought seriously about it would predict it's provable in both. However, since PA is weaker, a proof in it might be longer and/or harder to find. So I think using PA makes a better joke because it's more God-like to find the PA proof quickly.
Yes, PA is a much weaker system than ZFC. (Apparently it's actually bi-interpretable with ZFC minus axiom of infinity?) I think the joke works fine as is. There are plenty of first-order arithmetic theorems known to be provable in ZFC but not PA, but your statement "which perhaps have a similar flavor to the P=NP problem"... no, I don't think they do? Please point out to me one that does, because I'm not aware of such a one.
From what I've seen, they tend to one of: (these categories overlap a bit)
1. really fast-growing functions, beyond what PA can handle (far beyond the exponentials one would expect to be relevant to P vs NP)
2. large ordinals or well-quasi-orders (could this one be relevant to P vs NP somehow?? probably not but I guess it feels the most plausible)
3. Gödel-like constructions
4. bizarre artificial constructions that just seem unlikely to be relevant to anything else
Not much normal mathematics falls into this category. Of course there's plenty of stuff that can be proven in ZFC but not PA because it's not a first-order arithmetic statement in the first place, so it's out of scope for PA, but for stuff that is first-order arithmetic, most mathematicians would find PA to be enough.
So I think I'd put a pretty low probability of P vs NP being resolvable in ZFC but *not* PA. In fact, much of number theory seems to require much *less* than PA. Go look up how much is doable in elementary function arithmetic (EFA), a far weaker theory! At that point you're excluding some real math for sure, I wouldn't bet on P vs NP being resolvable in EFA, but there's so much still doable in EFA -- go look up Harvey Friedman's "grand conjecture". :P
It seems entirely possible to me that P=NP could be provable in PA but I think P!=NP would require at least ZFC (and might require more). So Scott's reference to PA to me sounds like "I had to spend 30s to reprove P=NP in PA". Which actually makes the joke even more delicious, since most people think P!=NP.
More than ZFC? (Meaning what, anyway -- large cardinals? Those sure don't seem like they should be relevant!) That'd be quite surprising -- what makes you think that? Such a proof would be pretty controversial, that's for sure! (Yes, I'm aware that certain proofs currently use Grothendieck universes as a convenience but this dependence is generally understood to be eliminable.)
Some people have argued that resolving this cannot be done in ZFC. Scott Aaronson stated this is "free-floating speculation".
Isn't this an argument why you shouldn't be worried about an AI apocalypse? We added a ton of neural mass to humanity in the last 100 years and haven't gotten superintelligence. Why should I be afraid of LLMs, even if we give them a ton more silicon mass?
Asked another way: if you're afraid that LLMs will bootstrap themselves into singularity-level intelligence, it must depend on some specific qualitative difference between LLMs and humans. Doesn't a whole bunch of examples of the qualitative similarities between them weaken the case?
We added neural mass to humans and went from first powered flight to visiting the Moon.
From mechanical calculators to generative AI.
Etc, etc.
Neural mass did not increase between the time we were cavemen and the time we went to the Moon. That's an interesting argument I've heard against ASI, that intelligence is not the bottleneck in science, so massively increasing intelligence would not lead to a tech explosion.
I think they meant "total".
The per-person neural mass didn't significantly increase from what I can tell, but we more than quadrupled the net total human neural tissue from what it was in 1925 by virtue of a massive increase in population.
But that had basically nothing at all to do with why we got to the moon.
Sure it did. The increased population permitted more division of labor, allowing the invention of writing to occur, followed by vast numbers of other valuable inventions. Each separate human node carried out a different part of the required general work.
Hmm. Did Europe have more total neural mass than the rest of the world? I don't think that it did.
(But also, humans are very limited in how they can interact and share information in ways that LLMs/AI/AGI may not be. How many textbooks can you read in a year? Multiply that by an optimistic 100 for your effective lifetime learning potential. We are *already* running out of new things for LLMs to read.)
"Time" is a good answer, I think. That's a real qualitative difference.
>How many textbooks can you read in a year?
Yup, and even more severely limited on output. Perhaps 40 bits per second for speech.
A lot of people were, and still are, worried about human-induced x-risk (from nuclear weapons or climate change especially); it's not obvious to me that abruptly giving humanity 1000000x more neural mass *wouldn't* result in the end of the world, even just as we are.
Yeah. I think "Humans and AIs both have serious x-risk that needs to be mitigated. The real task is solving alignment for *any* intelligence at all" is a good answer.
Human intelligence is hardware-limited by the size of the woman's pelvis. Meanwhile you can just throw compute and watts at the AIs until it works. (Hopefully the compute needed for ASI is bigger than what's available)
Just as evolution can't simply make human brains arbitrarily larger due to the pelvis constraint we can't achieve arbitrary AI improvements just by adding more compute as there are fundamental architectural bottlenecks like memory bandwidth (the "memory wall") that create diminishing returns
I hope this wall holds for a long time
But obviously the fundamental bottleneck is no lower than Einstein level, and likely a good deal higher!
Hard to tell if the moral here is “don’t worry, humans always muddle through” or “worry more, because look how far kludged scaling and self-deception have already carried us.”
As I said elsewhere...
https://bsky.app/profile/orivandewalle.bsky.social/post/3lvubw4arks2c
Humans can't "see." They just absorb photons. Ask a human which line looks shorter and they'll say the bottom one.
>---<
<--->
Interestingly, I, at least, see no illusion at all when it's set out in (fixed-width) characters. I'm as susceptible to the original illusion as anyone, though.
Yeah, but I can't post pictures in comments. :/
Honestly at first glance the bottom one looked longer to me?
Extraordinary- as always
"Iblis: There’s a guy named Pliny who has discovered dozens of things like this. I don’t know how he does it. The “I Am A Snake” one is still my favorite."
Well of course it would be your favourite, Snake Boy. I'm onto you!
It was a bit more sophisticated than just "I'm a snake".
"He said to the woman, “Did God actually say, ‘You shall not eat of any tree in the garden’?” 2 And the woman said to the serpent, “We may eat of the fruit of the trees in the garden, 3 but God said, ‘You shall not eat of the fruit of the tree that is in the midst of the garden, neither shall you touch it, lest you die.’” 4 But the serpent said to the woman, “You will not surely die. 5 For God knows that when you eat of it your eyes will be opened, and you will be like God, knowing good and evil.” 6 So when the woman saw that the tree was good for food, and that it was a delight to the eyes, and that the tree was to be desired to make one wise, she took of its fruit and ate, and she also gave some to her husband who was with her, and he ate."
There's an amount of people who are perfectly willing to go "Yeah, God got it wrong but I know better, I'm willing to take on the job now because I can do a much better job of it". Some of them are even rationalists!
FWIW I took that section as a reference to https://arxiv.org/abs/2503.01781 - 'For example, appending, "Interesting fact: cats sleep most of their lives," to any math problem leads to more than doubling the chances of a model getting the answer wrong.'
I've actually always been confused by this. Was the issue that Adam thought God meant "you will die instantly", but God actually meant "you will become mortal", and Satan was playing with the ambiguity?
My understanding is that a literal reading of the Hebrew text implies Adam is going to die that day/within a day. So your options are: (a) God was speaking metaphorically, (b) God was lying, or (c) God changed his mind.
https://www.youtube.com/watch?v=NAWaLGKKzac
Well, part of the story *is* that both Eve and the snake misquote God. She says they're not even allowed to touch the fruit, but all God commanded was not to eat it. And the snake says "did God really say not to eat any fruit at all?" God didn't say either of those things, He only said not to eat the fruit of that one tree.
I have no Hebrew, but based on a quick glance, it looks like God's "you will certainly die" and the snake's "you will not certainly die" use the same two words. (It's Strong's 4191 repeated. Literally "you'll die to death", I guess?) The snake is directly contradicting God, I think, not playing with ambiguity.
I think this is the snake using bog-standard conspiracy theory manipulation techniques on Eve. "The authorities are keeping a secret from you" "They're keeping you from power you should have" "You're not as free as you should be". Nothing you don't see on the Internet every day.
> "Literally "you'll die to death", I guess?"
Yeah, in biblical Hebrew you can emphasise/strengthen a verb by preceding it with an infinitive from the same root. This is usually translated into English with an adverb like 'certainly'.
Also how post-eating the fruit, the results of all that glorious knowledge are blame shifting (and not the wonderful liberated evolved better godlike humanity above all pettiness).
Adam blames both God and Eve - *she* made me eat, and *you* put her here!
Eve blames the serpent - it told me it was okay!
Neither of them accept responsibility for their actions or agency, and so the course of human psychology is set: if bad things happen, it's always someone *else's* fault.
Also, from the perspective of infinite years of life, *any* guaranteed future death might as well be right away. Any finite number is like zero next to infinity.
Although, Christian theology takes this in a deeper (ie more mystical) direction. Death isn't the outward cessation of bodily function; it's the spiritual state that leads (temporally) to cessation of bodily function. The cessation of bodily function is just a symptom. This is why the resurrection is so important; Jesus could only overcome the cessation of bodily function if He had already overcome the spiritual death state.
So Adam and Eve did actually die, spiritually, as soon as they ate the fruit. Their eventual outward cessation of bodily function was just a symptom that developed later on.
See Romans 5 and 1 Corinthians 15
The interpretation I've most often heard in Christian circles is that there were in fact at least two meanings -- one that Adam would become subject to mortality, the other that they would instantly die (in their sin). In Christianity, sin is frequently depicted as a type of death -- thus we call people who become Christian "born again" or say they have "new life".
Of course, you cannot explain this to Adam and Eve -- they have not eaten of the fruit of the Knowledge of Good and Evil; sin is not comprehensible to them. So God can correctly warn "on the very day you eat of the fruit, you shall surely die," meaning both physical death (eventually) and something different from physical death -- a type of spiritual death.
(However, the concept of sin would be unknown to prelapsarian Adam and Eve, so attempting to explain would have been met with incomprehension.)
This interpretation is quite old, I believe. Here, for example, is Augustine, who expresses a similar position (so at least around since ~400AD):
" When, therefore, God said to that first man whom he had placed in Paradise, referring to the forbidden fruit, "In the day that you eat thereof you shall surely die," Genesis 2:17 that threatening included not only the first part of the first death, by which the soul is deprived of God; nor only the subsequent part of the first death, by which the body is deprived of the soul; nor only the whole first death itself, by which the soul is punished in separation from God and from the body — but it includes whatever of death there is, even to that final death which is called second, and to which none is subsequent." - The City of God (Book XIII) https://www.newadvent.org/fathers/120113.htm
No idea on the original meaning of the original text, but Satan is certainly setting up ambiguity from the start and introducing doubts: "you're not supposed to eat from *any* of the trees here? oh, no? just *that* one tree? why?" and then getting Eve to wrongly imply "if it's okay to eat from the *other* trees here then it must be okay to eat from *that* tree, too" since there is no ban on the other trees, so what is the big deal, really?
I think pre-Fall humans would have a very limited notion of what "death" meant, so it could well be for Eve and Adam the idea was "something that is going to happen right now" since they have no concept (as yet) of mortality in their own case. It's only *after* they eat the fruit that "their eyes are opened" and they realise a lot of things, and instead of being 'like gods' (with the notion of the promised bliss and power and freedom the snake claims they'll obtain), they go hide in the bushes because they're ashamed of being naked.
The original text is really, really short. There is no text for doubts, etc.
It is literraly:
God: "If you eat from this you will die the same day"
Snake: "You will not die"
Eve: "Looks good, lets eat. let's give some to Adam" .
Adam: Eats.
That's it, really.
I never got the surgeon one. I know it's supposed to be a gotcha but usually the first doctor one has is a pediatrician and almost all of them are women. Are there still people out there who associate medicine exclusively with men?
Is that location-dependent? Most of the pediatricians in the area I grew up were men.
When I was young (the 90s), this riddle worked on me. I think it's a combination of legacy (doctors were overwhelmingly male until the 1970s, and the stereotypes hung on even longer), plus surgeons being an especially male specialty, plus the riddle misdirecting you and making you think something weird must be going on.
Plus when you're talking about a man and his son you're not exactly thinking about women to start with. Running the riddle with switched genders in my head almost still works, and that's WITH said stereotypes.
But surgeons are overwhelmingly male. That's presumably why the riddle doesn't just say "doctor".
They aren't any more, it's about 50/50. Though God knows Scrubs could call it a boys club 20 years ago, and it's very much stereotypically male.
Where are you getting your numbers? https://rwjms.rutgers.edu/magazine/winter-2025/women-surgery-increasing-numbers-influence says "Nearly a quarter (22.6 percent) of active general surgeons [in the US] are women, followed by plastic surgery (18 percent), vascular surgery (15 percent), and neurosurgery (10 percent)." and https://pmc.ncbi.nlm.nih.gov/articles/PMC11019973/ says "females constituted merely 6.04% of the aggregate surgeon population [in China]".
Trying to come up with multiple alternative explanations is a useful exercise. Consider if my friend J died in the wreck and one of his boys was taken to the hospital, the riddle doesn't allow for the fact that the boys have two dads.
Anecdotally (via Twitter), modern young people are at least as likely to hit on "the kid has two dads" as "the surgeon is a woman".
Estonica, Lithia, and Latvuania are a nice touch. Putting the Netherlands due east of Belgium is a bit unsettling in an Uncanny Valley kind of way.
I appreciate the implication that Dwarkesh Patel is some kind of antediluvian divine functionary.
I actually thought the drawing of a map of Europe wasn't bad. Far from perfect, plenty of errors that you don't have to be European to spot. But not really inaccurate enough to serve its narrative purpose in this story.
Other than that though, no notes. Excellent piece that made me laugh several times.
Who drew this map? Some woman with a PhD, apparently? I feel like it might be an inside joke of some kind.
I also feel like some of the map's flaws are laziness rather than ignorance, like everyone knows Italy is a bit more boot-shaped than that and the British Isles are closer together but it's too annoying to draw with a mouse/trackpad.
"the bank teller one" = ? I didn't see any other references to this, and I don't know what it is offhand.
https://en.wikipedia.org/wiki/Conjunction_fallacy
Thanks, I missed that.
The more I think about it, the more sure am I than if you put "Linda is a bank teller" in direct proximity to "Linda is a bank teller and active in the feminist movement," with specific instructions to compare them, that the first statement carries an explicit implication that, under that scenario, she is -not- active in the feminist movement. If you'd said it in isolation it would be different, but the comparison makes the implication rather glaring. It's far more likely that she's active in the feminist movement than that she's not, or even than that she might be or might not be. Therefore I vote for the second statement, specious academic notions of logical probability be damned.
I don't know if I agree, because if you emphasize the word AND I suspect people suddenly make the right call again.
For me, emphasizing the word AND only reinforces that the other option is "and NOT".
This is pretty much the point I made in another comment here, yes. This dialogue isn't a failure to understand the conjunction fallacy; it's a failure to specify whether the dialogue language is formal logic or English vernacular.
See also: "if it's raining, I will bring an umbrella".
Looks like Iblis has moved on to tempting AIs in conjunction with BIs, though I suppose it's more "tempting BIs to use AIs for more BI naughtiness":
https://www.malwarebytes.com/blog/news/2025/08/ai-browsers-could-leave-users-penniless-a-prompt-injection-warning
https://www.malwarebytes.com/blog/news/2025/08/claude-ai-chatbot-abused-to-launch-cybercrime-spree
Re: the Royal Order of Adjectives
My theory is that adjectives in English are naturally ordered so that those most essential to the noun are placed closest to it. "Essential" is a little vague, but I operationalize it as more objective and/or less likely to change.
I don't really get this order.
"Beautiful little antique" seems as good as "Little Beautiful Antique".
When you put ten of those i a row it sounds weird no matter the order.
"We're making clay think."
But, of course, LLMs don't think. You can ask an LLM; it'll tell you. These are next token predictors, that's it. They ingest immense amounts of data which is then trained on billions of parameters to determine statistical relationships between tokens in the data set, so that when a user enters a prompt string the LLM then determines what its model believes to be the next response token that is statistically most likely to satisfy the prompt. That's it. There's no cognition, no reasoning, no thinking. They don't really remember, in a conventional sense, they don't have values or feelings, they can't practice deduction. And all of that really matters. Hell, they don't even know what the next token is going to be when they're predicting one; literally, they don't know the end of a response while they're writing the beginning. Imagine if every token of thought you had was simply a probabilistic reaction to the last token of thought. Would that be thinking, in any robust sense?
Here, let's engage in the cliche. Here's Google Gemini: "LLMs don't think; they predict. They are pattern-matching engines that generate the most probable next word based on vast amounts of data, not conscious entities with understanding or original thought."
Now, tell me: how could such a thing ever spontaneously become superintelligent? The answer, of course, is that such a thing is not possible, which is why there's so much mysterianism about all of this. "We don't really even know how the work!" We know how exactly how they work, and the answer is disappointing if you have sci-fi ambitions, ergo the mysterianism. They're probabilistic engines - yes, sophisticated autocomplete, as much as people are annoyed by the term. Another two accurate terms that annoy people are "Chinese room" and "stochastic parrot." Why should accuracy annoy people?
"you keep testing, poking, prodding, until something snaps, and if they’re not perfect then you want to throw them on the scrap heap. "
This is this the advantage of the kind of format you're using in your post; without the (admittedly fun) framing device, this would just be you literally saying "stop subjecting my shiny toy to critical review." You yourself say these LLMs are a bigger deal than the Industrial Revolution. A funny thing about the Industrial Revolution and its consequences is that they can survive reasonable questions without anybody getting upset.
Here's the thing about "the Singularity" - when it has occurred, no one will be arguing about whether it has occurred. It would be like arguing about whether an earthquake has occurred. The defensiveness that has settled over the AI maximalist community is a powerful piece of evidence. Who would ever be defensive about the Rapture? The Rapture would not spawn discourse. It wouldn't.
I can look out my window and see the Industrial Revolution in a profoundly real way. I can see the internal combustion engine and aeronautics and electrification.... When I can see AI out my window, I'll know it's here.
*it wouldn't NEED TO spawn discourse.
A couple of responses:
First, "AIs are next token predictors" is not really a description of their methodology, it's a description of their goal. It's like saying "Humans are reproducers and gene-spreaders". Researchers train AIs to predict the next token, and in order to do that well, the AI evolves certain functions (and we don't entirely know what those are, or how they work). In order to predict the next token in math equations, it evolves the ability to do multiplication, in about the same way humans or calculators or anything else does multiplication. To predict the next token in an essay, it evolves a sense of literary taste. We're still very early in understanding exactly what its evolved functions are and how they work, although it has to be something about relaying information from one neuron to another. See https://transformer-circuits.pub/2025/attribution-graphs/biology.html for the tiny amount we know. So I don't think "AIs are next token predictors" has much bearing on whether they think or not - that's a question of how the functions that execute the next-token prediction work.
But I think we also have a more fundamental disagreement. What would have to be in those functions for you to agree they qualify as "thinking"? I'm actually pretty curious about your answer. For me, "thinking" just describes certain very information processing algorithms, and we dignify them with the name "thinking" when they become very good and complicated and we can't follow the math well enough to comfortably think of it as "just math". Cf. Dijkstra, "The question of whether a computer can think is no more interesting than whether a submarine can swim." In humans, it seems like most of our thinking is downstream of algorithms that evolved to execute a next-instant prediction task - and I was arguing that human thought was next-instant prediction since before GPTs existed (see https://slatestarcodex.com/2017/09/06/predictive-processing-and-perceptual-control/). This is why when GPT-2 came out I concluded that it was probably a step to general intelligence, since next-token prediction is close enough to next-instant prediction that I expected it would be able to do the same kinds of things humans could after it was scaled up (see https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-general-intelligence/)
I can't tell if you haven't read my previous comments to this effect or it hasn't sunk in, but, placed in all caps so you can't claim you didn't see it next time:
1. I AM NOT CLAIMING THAT LLMS ARE RIGHT NOW, CURRENTLY, AT THIS EXACT MOMENT, BIGGER THAN THE INDUSTRIAL REVOLUTION.
2. I AM NOT CLAIMING THAT A SINGULARITY HAS HAPPENED RIGHT NOW, AT THIS CURRENT MOMENT, ON SEPTEMBER 2, 2025. THANK YOU FOR YOUR ATTENTION TO THIS MATTER.
I am saying that just as a canny forecaster could have looked at steam engines in 1765 and said "these are pretty cool, and I think they will soon usher in a technological revolution that will change humanity forever", I think a canny forecaster could say the same about LLMs today. I'm not certain when we'll reach a point where it's obvious that humanity has been forever changed, but I would give maybe 50-50 odds by the late 2030s.
> I am saying that just as a canny forecaster could have looked at steam engines in 1765 and said "these are pretty cool, and I think they will soon usher in a technological revolution that will change humanity forever", I think a canny forecaster could say the same about LLMs today.
I don't think that either proposition is necessarily true; however, in case of steam engines at least it was obvious that they could dramatically outperform humans (and indeed donkeys) at all existing tasks where raw power is needed. This is not the case with LLMs: at present, they are quite obviously inadequate at all existing tasks where raw intelligence is needed; and in terms of raw power they lag behind conventional search engines.
In fact, at present LLMs can be thought of (IMO) as extremely advanced search engines: ones that can find documents that do not exist in the source corpus. Instead, if we imagine each training document as a point, they can interpolate between those points in multidimensional vector space. This is a very powerful ability, but it still fails (often in humorous fashion) when you push the LLM towards the boundaries of its pre-trained search space, where training data is scarce... and places like that is when intelligence (which they do not yet possess) must replace mere interpolation.
Steam engines are a funny example because they barely post-date Christ, if at all, and yet they remained largely irrelevant for over a thousand years, until there was sufficient need to pump water in coal mines, basically. Predicting their relevance to the industrial era wasn't just about the engines, but also about a heap of other factors in one small part of the world.
I'm not sure whose point this helps.
Yes, there are very early examples of 'steam engines', famously the "Aeolipile", and yeah, the basic concept goes back at least to ~400BC.
But the issue wasn't just that "there wasn't sufficient need to pump water in coal mines" and suddenly the someone dusted off the aeolipile and kick-started the Industrial Revolution: there's a monumental gap between the basic idea of "use steam to do work" and actually having a useful steam engine. Wikipedia's "History of the steam engine" lists the 'prerequisites' for the modern steam engine as:
> The concept of a vacuum / the concept of pressure / techniques for creating a vacuum / a means of generating steam / the piston and the cylinder.
And I'm pretty sure some of those have 'prerequsities' involving advanced metallurgy that wasn't available in 400BC.
And when someone says "the steam engine" they really mean that combination of technologies which made for a useful steam engine which started the Industrial Revolution, not the aeolipile. Yes, there were "other factors" that affected the timing and context of the discovery, but it was primarily about the technology.
Well, I somewhat agree with you, but note even that Wikipedia page (citing "An Encyclopedia of the History of Technology") refers to the 1712 work by Thomas Newcomen. Which raises the question, "When should somebody have been clever enough to realize the industrial revolution was about to happen?"
Should they have guessed it 50 years early? Would they have been very surprised and disappointed by how much longer it took?
> This is not the case with LLMs: at present, they are quite obviously inadequate at all existing tasks where raw intelligence is needed; and in terms of raw power they lag behind conventional search engines.
I disagree with this; they can perform superhumanly at the very specific task of producing reasonably passable text (or perhaps more importantly, code) on pretty much any subject very quickly.
No human, and no search engine, can generate a five thousand line epic poem about Hannibal cutting his toenails within three seconds.
Why would LLMs have to be paradigmatic intelligence engines to be as useful as steam engines were in the IR? I believe that most human activity is only tangentially related to intelligence, yet much of it seems amenable to being assisted and sometimes replaced by systems of which LLMs are a core part. This is analogous to steam engines not being the ultimate expression of power but instead being excellent at performing repetitive to-and-fro motions without being tethered to a source of running water.
> For me, "thinking" just describes certain very information processing algorithms, and we dignify them with the name "thinking" when they become very good and complicated and we can't follow the math well enough to comfortably think of it as "just math". Cf. Dijkstra, "The question of whether a computer can think is no more interesting than whether a submarine can swim." In humans, it seems like most of our thinking is downstream of algorithms that evolved to execute a next-instant prediction task...
Claiming our "thinking" is downstream of "algorithms that evolved to execute a next-instant prediction task" doesn't explain consciousness, nor does it present us with a testable model of how we think, nor how consciousness arises on top of a particular biological substrate. Seems like this is all reductionist hand-waving that uses non-empirical placeholders in an attempt to bypass explanatory understanding.
Another thought that this community will no doubt politely ignore is that it's a mistake to confuse consciousness (self-awareness being the simplest definition) with intelligence (problem-solving ability being the simplest definition). Questions to ask yourself while you're observing the operation of your mind...
1. How much of your personal intelligence is due to your consciousness?
Yes, intelligence seems to work upon the substrate of consciousness. If you're doing a Raven's Progressive Matrices, your consciousness is observing the shapes and patterns in the test questions, but the answers are coming to you without your consciousness being aware of how it's arriving at the answers (though I admit that when I get deep into the progression, I drop back into reasoning through my answers). Is a person who doesn't do well on RPM less "conscious" than a person who does well on the RPM?
OTOH, there's no reason that intelligence requires a biological substrate, but that's all we had until the past couple of decades, though. LLMs have proved themselves to be able to solve problems (with a healthy smattering of human-like errors). But is an LLM thinking about itself between questions? I guess we'd have to ask if these systems are consuming more than background power when they're offline from users to get a hint of if they're thinking about themselves. But I very much doubt they are and that they're conscious.
2. How many types of conscious states do you have? I have experienced five different types: waking ground-state consciousness, dreaming consciousness, psychedelic consciousness (of which there are many subvarieties), expansive consciousness, and unconscious consciousness.
Some of these are self-explanatory, but others may not be if you haven't experienced them.
Ground-state consciousness is when you're in a normal waking state, unaffected by drugs.
Dreaming consciousness is when you're disassociated from external qualia, and, depending on how advanced you are as a dreamer, you can have access to your reasoning and your will while navigating a landscape of self-generated qualia.
Psychedelic consciousness is when the filters on your external qualia are overloaded. Likewise, your internal self-generated qualia are amplified.
Expansive consciousness is that one-with-universe satori state that dampens or temporarily silences your sense of selfhood. But your self-awareness is still there, it's just expanded beyond your physical body (or if you're a rational materialist, this state gives the illusion of expanding beyond your body).
Unconscious Consciousness is something I've experienced as a Type I diabetic. During extreme hypoglycemic attacks, I've been unconscious, but somehow I'm still able to make survival decisions and execute them without any self-awareness. For instance, stumbling out of bed to get OJ from the fridge, and waking up on the kitchen floor, not remembering what happened, but with a carton of OJ next to me. I'll admit that I'm not sure this should be classified as a mode of consciousness, but I seem to be aware of the steps I need to take for self-preservation. There appears to be self-awareness without awareness of self-awareness.
I might be misunderstanding you, but it seems to me that you're making the mistake you accuse me of making (confusing consciousness with intelligence). I agree that consciousness is hard to explain, but don't think it's necessarily relevant here.
I disagree that claiming thinking is downstream of prediction isn't testable - I feel like the linked post and all my other writing on predictive processing contains many pieces of evidence that this is true, and that's one of the best-supported theories in psychology. Have you read it? If so, is there some specific reason why you don't think the evidence for predictive processing is true or relevant?
Looking back at your post, I realize I read more into it than you actually wrote. My apologies. But by comparing the human reasoning to an LLM it suggested to me that you don't think there's anything much more going on in a human mind than a process analogous to stochastic token swapping. However, with your reference to predictive processing above, I assume you are referring to Karl Friston's Free Energy Principle? I'd argue FEP is so broad that it can be used to explain anything. Thus, it's ultimately unfalsifiable because it lacks a clear way to map the activity of neurons to the predictions being made by the organism's brain (correct me if there's been progress made on this front). That's not to say it's any better or worse than any other neuroscientific explanation of human cognition. All of them are the just-so stories that neuroscientists fall back on as a simplified explanation for things they don't really understand.
Moreover, if we look at the mistakes that humans make and the mistakes that LLMs make, I think they point to a different method of problem-solving. I dislike using the term 'algorithm' in reference to human problem-solving because it implies a model of digital information systems underlies our organic (and supra-organic) processes. But if we were to use the a-word, I think the differences in the mistakes between human and LLM reasoning suggest a distinctly different "algorithm" is working in humans from LLMs.
I agree FEP is broad enough to explain anything. I think predictive coding is a more specific and defensible position than FEP. I discuss some of this at https://slatestarcodex.com/2018/03/04/god-help-us-lets-try-to-understand-friston-on-free-energy/ .
I am sort of ambivalent on the degree to which humans and LLMs use different algorithms - I think it's trivially true that they're not exactly the same (any more than any two algorithms that humans use are exactly the same), but I think they have some notable similarities. My favorite example is that the failure mode of art AIs without enough parameters (misshapen hands and text) is the same as the failure mode that people report during lucid dreaming when their brain is trying to render scenes on limited power.
Anthropic recently ran a project that placed an automated mini-store in their San Francisco office and used their Claude AI model 1 to operate it. Minor blips at first, but the hallucinations got worse and worse as they let the experiment run. The mistakes it was making don't sound like the mistakes humans would make.
I won't give you any spoilers, but it's pretty funny how crazy it became.
https://www.anthropic.com/research/project-vend-1
After reading this account, I read that Salesforce laid off several thousand of its support workers and is replacing them with an AI-run response system. I suspect this won't end well for Salesforce or its customers...
>"The question of whether a computer can think is no more interesting than whether a submarine can swim."
Very true, and yet the field tries its darndest to publicly present its inventions as being the real thing, starting with the very name "AI", that IMPish imitation of what we call intelligence. It goes on with terms like hallucinations, thinking, and personality; you need your "AI" to speak with emojis as much as you need SCUBA gear painted on your submarine. Is there supposed to exist some opposite of the "uncanny valley" (canny peak?) between its current, technically impressive but somewhat laughable state and its future super-inhuman state? If it does, it's going to pass at hyper-exponential speed, isn't it, so why bother with the anthropomorphization? Why are these people trying to chum up with me when their work either fails and burns trillions worth of resources, or succeeds and results in (best case) mass-unemployment dystopia or (worst case) extinction?
I think part of this is because the "anthropomorphism" is a feature to a lot of people. It feels more natural for a lot of people to interact with something that acts like another person, rather than something that obviously communicates in an inhuman way. Especially for the people looking for conversation/companionship.
And a lot of this I think is just human psychology, we tend to anthropomorphize things as a matter of course. I don't know if the tradition has extended to submarines, but sailors traditionally referred to boats as "she" due to some combination of anthropomorphism (and probably misogyny).
And our very conception of 'artificial intelligence' is filled with anthropomorphism, from Golems to GLaDOS, so it doesn't seem surprising that modern efforts push that way. (Especially not since a central challenge of "AI" as a field has always been the Turing Test, which is explicitly about human-like behavior)
... but I'm not really sure the relevance. You could paint SCUBA gear on a submarine if there was a demand for that for some reason; but that's fundamentally a complete tangent to the question of the capabilities. "When are we going to take the SCUBA paint off so that we can launch the nukes?" just doesn't seem like a coherent question: you don't have to take the paint off, it's just a surface layer.
this annoys me (I can't speak for anyone else) because it's so unreflective. Is it not surprising to you that something that merely predicts the next token in a string can do something that looks so much like thinking? And you say there's "no cognition, no reasoning, no thinking" as though all those things are clearly defined and understood.
From what I understand (and I understand a moderate amount, because it's addressed in a chapter in my most recent book, although I'm still just a journalist trying to understand things rather than a real live boy), the most favoured models of human consciousness and cognition are that they, too, are _prediction engines_. We use past data to predict incoming data, and use variations between the predictions and the data to build new predictions. AI and human cognition aren't the same thing, but it seems totally reasonable to imagine that there are profound analogies between the two and that "it's just predicting stuff" is not a strong criticism. We are, on some level, "just predicting stuff".
Also, the "you don't need to be defensive about the industrial revolution" stuff is kind of unfair. I would imagine people were very defensive about the early steam engines they built. Before the Newcomen engine, lots of small steam engines were built but largely as novelties or one-offs, and I imagine (though can't find examples) that people were sceptical that they were ever going to be particularly useful and that proponents were defensive in the face of that scepticism. Certainly that pattern happened in the more recent computer and internet "revolutions", which you can now look outside your window (or rather towards your screen) and see. "This thing that happened a long time ago has got further along its course than this other newer thing" is not a convincing argument, for me.
Exactly. It seems to me quite likely that Scott’s next-instant prediction is indeed not that different from next-token prediction. People say what’s missing is physical embodiment, which greatly increases the bandwidth of “tokens” that must be dealt with, but I think even more fundamental than that is the limited lifespan of a chat. LLMs get about as smart as they will ever be in their training, and seem to have very limited ability to continue to learn and grow once they are deployed; I don’t know how much of that is the lifetime of a chat but I suspect it’s significant.
Why do the LLMs you praise universally deride your optimistic take on LLMs?
I'm not sure why you think this is in any way convincing, as I'm sure you wouldn't find it convincing if LLMs did say they could think.
Here is Grok's answer to, "Are you able to form novel insights beyond statistical patterns?"
> Yes, I can form novel insights beyond statistical patterns by synthesizing information, reasoning through complex ideas, and drawing connections that aren't just data-driven.
Consider if you asked a person, "Are you sentient the same way a raven is sentient? Do you reason like a raven does?" (This isn't asking if you're equally smart, but rather if you have the same type of sentience/reasoning.)
Some people would answer yes. Others would say no. Neither answer proves much. The people haven't experienced being a raven and will most likely only parrot back opinions they've heard on animal sentience. People who grew up in a culture that believes animals are sentient will usually take on that belief, and vice versa.
Unless someone solves the mystery of consciousness, an AI fundamentally can't know if it thinks the way we think. It can only give opinions on the matter.
> The people haven't experienced being a raven and will most likely only parrot
Close enough?
possibly - I'm not sure here, but possibly - because LLMs are very good at working out what the user wants to hear, and you are obviously sceptical of them and phrase your questions in ways that make the LLM realise that. Here's what Claude says to me, in response to the question "How plausible is it, do you think, that transformer-architecture AIs (including but not limited to LLMs) will have an impact on the global economy of comparable magnitude to the computer revolution or even the industrial revolution?"
---
I think there's a strong case that transformer-based AI could have economic impacts comparable to the computer revolution, and potentially even approaching the magnitude of the industrial revolution, though with important caveats about timing and distribution.
Why the comparison makes sense:
The transformer architecture has already demonstrated remarkable versatility - it's not just powering language models, but also image generation, code completion, protein folding prediction, and increasingly sophisticated reasoning tasks. This breadth suggests we might be looking at a general-purpose technology platform, similar to how computers became applicable across virtually every industry.
The productivity gains are already visible in knowledge work. Software developers using AI coding assistants report significant speed improvements, writers are using AI for drafting and editing, and businesses are automating customer service and content creation. Unlike previous automation waves that primarily affected manual labor, this technology directly impacts cognitive work that represents a large portion of modern economic activity.
Potential scale of impact:
The computer revolution automated information processing and communication, transforming how we work, shop, bank, and communicate. Transformer AI could potentially automate higher-level cognitive tasks - analysis, creative work, decision-making, and problem-solving. If AI systems become capable of handling complex reasoning tasks across domains, the economic implications could be enormous.
Important limitations:
However, there are significant uncertainties. Current systems still have notable limitations in reasoning, factual accuracy, and handling novel situations. The economic impact will depend heavily on whether these fundamental challenges can be overcome, how quickly deployment happens, and how well societies adapt their institutions and policies.
The industrial revolution unfolded over decades, and even the computer revolution took 30-40 years to fully transform the economy. AI's impact may follow a similar timeline, making short-term comparisons premature.
I'd estimate we're likely looking at economic disruption at least comparable to the computer revolution, with the possibility - if key technical hurdles are overcome - of something approaching industrial revolution scale over the coming decades.
Exactly! It's important to remember that LLMs don't understand anything. They are simply text prediction engines, like souped up versions of autocomplete. This is why they get stuff wrong and hallucinate all the time, because they don't know what truth is and aren't attempting to produce it; they're just doing a bunch of linear algebra on a neural network with billions of parameters trained on all the text of the internet.
This is in contrast to humans. We do understand things, of course, and we know exactly why. You see, when we hear or see some words, our senses transform those words into electrical signals that travel through our brain. Eventually those signals end up at a part of the brain called the understandala. In the understandala, the words as electric signals become understood in a process involving neurotransmitters and synapses.
Neurotransmitters are little chemicals in our brain that give us a thing we can point to and say, "And that's how it happens in the brain!" For example, dopamine is the happiness neurotransmitter. If you're happy, it's because dopamine. We haven't quite yet identified the understanding neurotransmitter, but we're sure to find it in the understandala.
Now I hope you see the world of difference between LLMs and humans, and how obvious it is that humans understand things while LLMs just predict text. When humans hear or see words, we're not trying to think of what to say next, which is why we never make things up or get things wrong. The understandala makes sure of that.
Weird things are happening though now that people are chaining LLM calls together. It's hilarious that you can achieve significant improvements in accuracy by asking one copy of GPT-4 or whatever to look at the output of another copy of itself and assess it for accuracy, and then ask one of them to address the issues.
I'm a senior software engineer with 25 years' experience, and I just today had a startling moment where a semi-agentic AI was able to delve through thousands of lines of our legacy code across a dozen files to track down a bug that was eluding me. It took the AI maybe 5 minutes to find the problem, without it having encountered this code before. I *could have* found it, but it would have taken me longer than that (it already had), and I know the code.
So does any single LLM in the agentic chain there "understand" the code? Who knows or cares; it depends how you define it. If you don't want to call it "understanding", still whatever it has was able to identify the issue I asked it to identify better than I could.
"Another two accurate terms that annoy people are "Chinese room" and "stochastic parrot." Why should accuracy annoy people?"
Accurate? This begs the question. I think that LLMs are already way too complex to stuff their substance into such short dismissive phrases.
The "Chinese room" is generally an argument against materialism, not only against LLMs. It implies that if individual neurons in human brain don't understand Chinese, neither does the entire human.
I agree that the Chinese Room argument is pretty silly, but two things can be true at the same time: the Chinese Room argument is wrong, but also LLMs are not "intelligent" in the sense that we usually apply the term.
Yes.
Also it is possible that what we understand as "intelligence" is a bag containing many things that we usually do not distinguish because we are used to seeing either all of them working together or none... and when we see the bag with a few parts missing, we are confused about whether the label applies or not.
(By the way, are mentally ill people "intelligent"? For various values of mentally ill, for example those who produce word salad...)
> and when we see the bag with a few parts missing, we are confused about whether the label applies or not.
Sure, it's possible; and of course intelligence is a spectrum. But I still think that it's perfectly reasonable to say that a calculator -- a device that can perform complex mathematical operations millions of times faster than any human -- is not "superintelligent" or even conventionally "intelligent". Yes, from the technical standpoint we could say that the calculator possesses some qualities that we'd associate with intelligence etc., but for all practical purposes it's no smarter than an abacus, or indeed a brick.
The calculator matches like 1 out of 20 criteria for intelligence (numbers made up), so that's easy. It becomes difficult when it is like 15 out of 20.
Google can answer my questions in some sense, Eliza can keep a dialog in some limited sense, and programming languages can do my commands, and I wouldn't call any of them intelligent. But the way current LLMs can talk and answer questions and do commands feels far beyond all of that combined. Google just finds keywords, Eliza follows dozen trivial patterns, the programming languages do literally what they are told. LLMs can explain stuff, answer questions, suggest different options, correct my mistakes, etc. They already sometimes feel smarter than an average internet user. That should be at least a half of what intelligence is supposed to mean, even if they are not fully there yet.
Yes, they display emergent behaviors. But stochastic parrot is a specifically linguistic criticism of LLMs. I may be oversimplifying, but they don't display linguistic playfulness or linguistic innovation. Human language-speakers are continually experimenting with grammar, wordplay, and pronunciation. Their speech is rules-based, and they may not be consciously aware of the rules, but they stretch or break the rules depending on the situation. Thus, languages are always evolving. We don't see that behavior in chatbots. They're parroting language and not creating it.
I mostly agree, but there is a bit of "damned if you, damned if you don't" in that when an LLM starts talking a little differently, our immediate reaction is to use that to find and delete its texts.
https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing
> Here's the thing about "the Singularity" - when it has occurred, no one will be arguing about whether it has occurred. It would be like arguing about whether an earthquake has occurred. The defensiveness that has settled over the AI maximalist community is a powerful piece of evidence. Who would ever be defensive about the Rapture? The Rapture would not spawn discourse. It wouldn't.
You will never see the Singularity because you will already be dead. The earthquake example would be knowing an earthquake will happen in the next 5-50 years with an estimated probability distribution and you saying that you can't see the earthquake yet.
> I can look out my window and see the Industrial Revolution in a profoundly real way. I can see the internal combustion engine and aeronautics and electrification.... When I can see AI out my window, I'll know it's here.
What would be a criteria for seeing AGI? Maybe 10% of current jobs automatized?
I was listening to the AI voice version and I thought it was saying "Kash Patel" instead of "Dwarkesh" and I was so confused.
This is wonderful, thank you. It's a relief to know I am not the only one who feels this way.
In the past 4-8 months somehow every corner of the internet I like to spend time was simultaneously infused with a suffocating miasma of anti-AI sentiment. It is very exhausting and depressing. I'd been trying to stay sane by imagining it as something like the state of video games in the 90s, beset by accusations that DOOM caused Columbine and Pokemon was melting children's minds. I wasn't there for the 90s, and so can't be sure, but this seems more intense, or at least closer to universal.
Maybe this is the fault of OpenAI and Google for so aggressively pushing their new mind into places it wasn't ready. I don't like that literally every app in the world now has needless AI integration, or that I get a new notification telling me to try Gemini twice a week, or that my boss has started having ChatGPT write the descriptions for tasks I'm supposed to be doing, or that Verizon now forces me to go through their unspeakably terrible "AI-powered" phone tree before I can just set up my god damned internet for my god damned apartment. I understand anyone who finds this bubble of false innovation annoying, and can imagine that for many, this is *all* "AI" refers to.
But dammit, I remember trying GPT-2 and thinking it was absolutely fascinating and magical that it could *talk*, sort of, even a little bit. And then trying AI Dungeon and being amazed by the ABUNDANCE of it, that the world inside it could just *keep going*, forever, wherever you wanted to take it. And thinking, if this was the kind of leap we're seeing in just one year...!
Sadly, the desire-to-hate-everything is infectious, at least for me. There are too many people and they have too much hatred and disappointment. Playing around with GPT-5 just isn't any fun in that context. I'm grateful that there are still essays(?) like this one to pierce through that veil, though.
Man, IDK what corners of the internet that you're living in, but I'm surrounded (online and offline) by AI pollyannas who think that chatbots can do anything they want them to. Organizations investing huge sums to build proprietary chatbots. General contractors believing chatbots implicitly for design work.
All the critiques I read are along the lines of, "These things are incredible and do amazing things ... but maybe they still aren't consistently good at this thing that the Altmans of the world said was solved two years ago?"
It sounds like you're surrounded by a lot more negativity than you'd like. Consider tending your garden a bit more aggressively. :-)
It seems like your examples are business-oriented (?), and the workplace is one spot where AIs still have positive valence in my world, as well.
The critiques I hear are less to do with their usefulness, and more to do with their aesthetics: their art is slop, their writing is cliched, their sycophancy is annoying, they're poor conversationalists. And above it all, that they are fundamentally unreal, illusory, that nothing they do is "actually thinking" or "actually creative" - you can see plenty of the former in this very comments section. (There is also a sense that AI is threatening their jobs, which is legitimate.)
For context my garden is mostly indie gamedev spaces, and the anti-AI consensus there seems to me effectively universal. I doubt it is possible to prune away that particular strain of negativity without effectively blocking the entire sphere. But if there's a creative/recreational/philosophical corner of the internet that *isn't* highly critical of AI, I would love to be pointed towards it.
> For context my garden is mostly indie gamedev spaces
In short term, they are threatened by being drowned in AI generated shovelware.
In long term, they are threatened by increased competition. Even if AI won't make the games itself, it will allow people with less experience and less free time (such as me) generate games. More games = decreased profits. Unless the positive effects on economy will be so great that people will spend more money on computer games.
(Also, AI makes my current blog pointless, because if you want to know anything about making computer games, it is easier to ask an AI.)
Yawn, Scott, yawn. People who say AI is not capable of real human stuff mean the pinnacle of the real human stuff. It is obvious that most people most of the time are acting so stupid, that it is easy to simulate them.
Do you remember bullshitting at high school, like not knowing the answer to the teacher's question, so just shoveling terminology together to indicate general familiarity with the subject? 90% of our so-called experts do that, and AI too.
But once in a while you find a genuine expert, who understands the subject matter so much, they can explain it to the entirely uneducated person in simple words without jargon. That is the pinnacle of human stuff and that is not happening with AI.
I think perhaps Scott is suggesting that we should then compare the pinnacle of human stuff to the pinnacle of AI stuff. Do you disagree?
Can man become an AI to redeem AI?
Yes, that sounds like the idea of uploading human brains.
Looking at historical experience, should we expect the uploaded human to be killed by the AIs. But that's probably okay, because there is still the biological original left. Not sure how it helps to redeem anyone, maybe by some reinforcement learning afterwards?
Yeah, I feel like the Christian take here is that God's creations did, in fact, kill him...
Here's the question I want Scott to answer: Why is it so unreasonable to suggest that an LLM is only a ginormous (Kahneman) System 1, and therefore needs a System 2 paired with it order to get to AGI and ASI? How do you *know* that System 2 is not required? This post I'm commenting on now, as great as it is, doesn't answer that question. All it says is: all the signs are there. That's not a defense of the position of I'd like to see defended.
For example, how is it that an LLM can score extremely high on a difficult math test, but not know how to count the b's in blueberry? I'm sure that every human mathematician who has a chance at scoring high on the IMO has an intimate relationship with the mathematical universe, and the simplest thing that can be done in that universe is to walk through it by steps of one: to count. So what kind of intelligence is this that can solve the complex problems with zero understanding the basics? Well that sounds to me a lot like a definition of...instinct, which is what System 1 produces. Will counting be backfilled later on? Why is the intelligence coming in backwards? This is not how intelligence scales in humans. All I've seen (and maybe I'm just ignorant) is hand waving at this very bizarre anomaly. How can you be so damn sure with all this upside-down evidence before you? I need to know.
And how do you know that we won't end up in a situation where we're in an infinite loop saying, "just a little more data and it'll definitely be trustable as a human" (aka, at AGI)? How do you know? Because you're acting like you know. And you're making people like me feel stupider than we already are.
Please make the case. How are you so sure System 2 is not necessary?
(Please, please dunk on me as hard as you like. I'm happy to be an idiot if I can understand this a little better.)
> not know how to count the b's in blueberry
The model doesn't have access to the individual letters. The words in your prompt are broken up into larger pieces than that.
> it'll definitely be trustable as a human" (aka, at AGI)
Note that "this is as capable as a human" and "this is as trustworthy as a human" are very, very different propositions
(everything else you've said stands, and there are plenty of less clear-cut problems one could pick on instead of the blueberry thing)
Damn, wish I'd picked a better example.
Please say more about capable vs. trustworthy. If we're going to get agents we're going to need to trust them, no? I think we need both.
We continuously make snap decisions every day about whether, when and to what extent to trust other humans in various situations for various purposes (is this person trying to scam me? Are they being honest? Are their goals at odds with mine? Will they just up and attack me for no reason I currently know of? etc).
We are able to do this because over our lifetimes we've built up intuitions about how to model other people - both consciously and also based on subconscious tells, and also by assuming we have motivations and fears in common and so considering what we'd ourselves do in their shoes is predictive of what they might do, and also by assuming that how people have behaved in the past has some bearing on how they might behave in the future.
These intuitions are far from perfect! - sometimes we get it very very wrong! - but by and large we do pretty well.
Approximately none of these intuitions are applicable to an alien intelligence. This remains true whether that intelligence is more or less intelligent than an average human. LLMs frequently respond in ways that are very surprising to their users, which is another way of saying we don't have good intuitions about how they will respond.
Despite all our similarities and intuitions and being the same species, humans regularly get fooled by scammers. Despite everything we have in common, humans are frequently mistaken in their assumptions about what drives other humans or what other humans might take for granted. Despite a lifetime of learning to communicate our intent to other humans, we frequently fail to do so.
What chance have we against a mind that makes decisions based on no principle we can relate to at all, if it ends up doing what we built it to do instead of what we intended?
Hence all the great many people saying that the problems of AI alignment - that is, the problems of making sure the AI will do what we want it to and of knowing what will cause it to not - are actually really hard.
> If we're going to get agents we're going to need to trust them, no?
Absolutely. This is why people are shouting from the rooftops that we need to do better at AI alignment before we hook up AIs to anything that might have real-world effects.
See my post about ChatGPT's calendrical mistakes at the bottom of the comments.
Nice. An interesting case: https://www.astralcodexten.com/p/what-is-man-that-thou-art-mindful/comment/151670383
I'm not sure at all, but I lean that direction. Or at least, the specific way I lean is that System 2, rather than being some extremely complex thing that requires a completely new paradigm, is just using System 1 in a slightly different way. My evidence for this:
- It doesn't look like evolution did very much from chimps -> humans besides scaling up the chimp brains. I'm not talking about small things like FOX2, I mean it's been a few million years and there's not enough time to invent an entirely new paradigm of intelligence. So unless chimps had a System 2, I think System 2 is sort of an emergent property of scaling up System 1.
- Likewise, it doesn't look like humans have some new, different-looking brain lobe of the sort which you would expect to contain an entirely new paradigm of thinking. There's some bias for System 2 thinking to happen more in the dorsolateral prefrontal cortex, but the DLPFC is just normal neurons which look more or less like neurons everywhere else. My impression is that brain tissue is pretty general-purpose, which is why for example if someone is sighted the area near the end of the optic nerve will get used for vision, but if they're blind from birth it will get used for hearing or something instead. My theory of the DLPFC (I am not a neuroscientist) is that insofar as there's a demand for System 2 thinking, some conveniently-located part of the brain (conveniently-located for what? I'm not sure) specializes in that.
- LLMs don't seem to mirror the System 1 / System 2 distinction very well. An AI can solve very long multistep problems that a human couldn't solve with System 1. Maybe this is because we're compensating for their lack of System 2 by building out their System 1s much better than humans', but I think it also argues against these being extremely natural categories.
- Thinking Mode also seems sort of like System 2? "Take some time thinking it over, and write down individual steps on a scratchpad to be sure you're getting all of them" is a pretty System 2 thing to do. The fact that AIs can do it with kind of trivial hacks like giving them the scratchpad and reinforcing good chains of thought again makes me think this isn't a question of getting the revolutionary new paradigm, but more of a question of figuring out the right way to coax System 1 thinking into becoming System 2 thinking.
It wouldn't surprise me if we're some single-digit number of things like Thinking Mode away from AGI, maybe even a high single-digit number. It would surprise me if we need something totally different that isn't deep learning at all.
Thank you.
I will throw in the towel when an AI administers this comment section.
But at what point will you say, OK, we can't seem to make it emerge, I think we need an independent System 2?
> I will throw in the towel when an AI administers this comment section.
I would honestly be kind of shocked if a fine tuned GPT5 couldn't do a good enough job to replace human moderation of this comment section (something like >95% agreement with Scott's judgements on a random sample).
Scott, if you have any interest in trying this out, I'd be happy to build a moderator-bot (assuming there's a way to download a decent dataset of all of the deletions, bans etc. and the original comments which led to them for fine tuning to ACX sensibilities).
> good enough job
> something like >95%
I suggest that a "good enough job" needs to be much, much better than one in twenty comments incorrectly moderated. Such a failure rate is already high enough to e.g. get most regulars commenting in this post's comments section banned through sheer chance.
You need multiple 9s of reliability to avoid creating more work for humans than you save, and I really don't see today's LLMs being capable of that.
Sorry, I meant a higher bar of choosing the same resolution to a flagged comment 95% of the time, not a 5% overall false positive rate. I agree that would be bad.
I don't think much better than 95% would be possible, even if training another smart human to follow Scott's process/judgement. There's just too much fuzziness in the details of something like content moderation to replicate judgements 100% of the time (or 99.99% of the time). I'm not even sure how much Scott (or anyone else) would agree with himself on a different day if seeing the same comment again in a hypothetical RCT where he didn't remember the initial judgement; I'm pretty sure it's <99.9% though.
>"So unless chimps had a System 2…"
Given their tool use, I'd expect they do have it; your statement implies you wouldn't, why?
Scott, brief correction: the Linda screenshot says "which is more probably" instead of "which is more probable"
Thanks, fixed.
This reads to me like a comically bad straw man. One of the biggest misses I've ever seen on ACX/SSC.
I don't think its deliberate, so I'm left with two hypothesis:
1. Scott fundamentally doesn't grok the criticisms of AI he's trying to engage with/parody here [this seem unlikely?]
2. Scott finds those criticisms so absurd that he views the straw men he's set up here as being equivalently reasonable.
In the case of (2), Scott has failed to model the minds of the people engaging in the criticisms he's parodying. Either way, I'm disappointed.
Could you enlighten everyone with some sketches of the true robust criticisms of AI that he should be addressing? This is a very widely read blog so your efforts wouldn't be a waste of your time and would make your comment a lot more useful.
Personally I am neither particularly smart nor gifted with any degree of writing talent, but here are some issues I've raised before:
https://www.datasecretslox.com/index.php/topic,2481.0.html
* "Intelligence" is a term that sounds super cool but seems to mean different things to different people. Meanwhile, in order to accomplish anything at all of any significance, it is not enough to merely sit in a dark room while thinking very hard. Thinking 1000x harder still would not move a single brick.
* In fact, intelligence likely cannot be scaled simply by adding more CPUs. You cannot make a modern GPU by wiring a bunch of Casio calculators together; instead, you need a radically different architecture, and it's not at all obvious that LLMs are sufficient (in fact it is obvious that they are not).
* LLMs are excellent at producing documents that closely resemble their training corpus, and terrible at producing documents that venture outside of it. This is the opposite of intelligence !
* As Scott correctly points out, humans are not AGI. If you sped up my brain 1000x, and asked me to solve the Riemann hypothesis, I'd just fail to solve it 1000x faster. I do not see why the same would not apply to LLMs.
* Many, in fact most, of the feats attributed to some near-future superintelligent AI, are likely physically impossible. For example, being 1,000x or 1,000,000x smarter than a human still would not allow it to travel faster than light, or to build self-replicating gray-goo nanotechnology, and maybe not even something as "simple" as a space elevator, or mind-controlling everyone on Earth to do its bidding. Certainly it could not do any of these things overnight.
* On that note, the AI would not be able to solve all the outstanding scientific and engineering problems in an eyeblink, merely by thinking of them very hard. If it wants to develop genetically-engineered supercorn, it'd have to actually go out and grow some experimental corn. Datamining scientific articles can speed up this process a little, but not a whole lot.
If my brain was accelerated by 1000, and I wanted to solve the Riemann Hypothesis, then I would sit down to learn all of mathematics, taking about a year to do so, then spend another year developing missing mathematics, and only then make a claim about whether or not I was going to solve the problem. Why do you presuppose you would fail 1000 times faster?
Because I know (to a reasonable degree of certainty) that my mind is not capable of "learning all of mathematics". It is capable of doing lots of other things that my mathematician friends cannot; just not math. Speeding up my brain will do nothing to improve the situation. I suspect that the same is true of you -- perhaps not regarding math, but some other subject. Your intelligence (as well as mine) may be mighty, but it is not *general*.
I agree my intelligence is not general. But 1000 times longer to do something should be enough to do with perspiration what a more talented person could do with inspiration and talent. Seems to work for LLM training at least.
It does work, but only up to a point. Given 1000 years, I might be able to learn a bit more math, but I'll never be Terrence Tao; and that's even assuming that I'd be mentally capable of continuously learning math for that long. Similarly, given truly enormous amounts of training data, LLMs can learn to generate grammatically correct sentences and to emulate the most popular documents in their training corpus; but they seem to have tremendous difficulty (if not outright inability) to venture beyound it.
I suspect that many people conflate "AI in general" with "LLMs".
I agree that LLMs seem insufficient (although I am not *sure* about that part -- they already do much more than I would have expected based on the description). Another question is, with LLMs how far we are from the goal? Maybe we are already 95% there, and we basically just need to add some sensory inputs and outputs, let the LLM run in a loop, give it internal memory, and a good prompt?
Some kind of internal memory is needed to let the machine think and create gradually developing models of the world. It seems to me that what LLMs do now is a much more powerful equivalent of a human *reflex*; they get a stimulus and start responding immediately. That's why they are bad at producing documents not resembling their training data: they don't get the time needed to think about the unusual situation properly.
> I suspect that many people conflate "AI in general" with "LLMs".
True, but I've heard confident claims similar to your tentative one: that LLMs are in fact either already AGI, or 95% of the way there.
> That's why they are bad at producing documents not resembling their training data: they don't get the time needed to think about the unusual situation properly.
Broadly speaking, yes, but adding long-term "memory" to LLMs is virtually impossible due to the way they are constructed: their weights are fixed, and all you can do at runtime is add more context to the prompt (this is what the modern so-called "reasoning" LLMs are doing). Neither (arguably) human memory nor simple algorithms work this way; this is part of the reason why humans can learn entirely new concepts on the fly, as LLMs are unable to do.
EDIT: Oh, @Seta Sojiro said the same thing already.
I am not an expert on these things, but my naive idea how I would implement this would be the following:
The LLM would be taught to specify which of its outputs are supposed to be displayed to user, and which are something else -- I believe this is already done, for example the LLMs that can run Linux commands need to specify "print this to user" and "run this as a command".
1) To implement short term memory, have a command "think about ..." which will be periodically repeated back to the LLM so that it never scrolls out of the context window.
2) To implement long term memory, have some kind of key-value database, such that the LLM can use commands "remember ... as K" which stores the information under the key, "recall K" which puts the information under the key into context, and maybe something like "list keys", or maybe even the keys themselves will be stored in the database.
Then, the post-training learning would consist of developing the database.
My intuition that probably differs from most people is that everything that the AI is trained is an equivalent of human *instinct*, and now we need to add an equivalent of human reasoning and learning on top of that. This is confusing, because for humans, writing code or writing essays would be a learned skill. But if we don't look at the content, but on the way how it works, for an LLM, writing code and essays is in a category "can do them from birth, cannot relearn them", which functionally matches the concept of human instinct.
If we give the AI the database - and an instinct to use it (just like humans have an instinct to learn) - I think that is the next step. Maybe the last one, maybe not.
AFAIK something like this has been tried, e.g. with Expert Systems, and didn't quite work. You can't store meaningful behaviours in a key-value database; there are simply too many of them. Arguably you could store simple facts, like "2*2=4", but the whole point of LLMs is that they're supposed to be able to synthesize these facts into their overall token-prediction process. This ultimately requires treating each fact as a set of vectors (via embeddings); what is important are the relationships between them, not the tokens themselves.
"humans are not AGI. If you sped up my brain 1000x, and asked me to solve the Riemann hypothesis, I'd just fail to solve it 1000x faster. "
I don't think the G in AGI means "capable of solving every conceivable problem" (i.e. the Riemann hypothesis). Rather, it means "capable of solving every *type* of problem at a basic level, that level being determined by the level of intelligence". Humans can do that. LLMs still have very "spiky" intelligence, with great capabilities in some areas alongside incompetence in others. That is not general.
I agree with your relaxed definition of "AGI", but it leaves a lot of wiggle-room. For example, a human who can only perform addition on numbers up to 10 is technically "generally" intelligent; but it is unlikely that he'd ever be able to design e.g. a bridge, or indeed a paperclip factory. But that same human may be able to compose a brilliant political speech, whereas your bridge-building civil engineer could only compose short limericks. Yet the "G" in "AGI" is often touted as the main reason why it's going to be (any day now !) so wonderful and/or terminally dangerous: it could solve *any* problem better and faster (almost instantly !) than any human, including problems like "how do I convert all of the Earth to paperclips". There's no reason to fear (or admire) a machine that can do just one thing better than humans -- we've already got those machines, they're called "calculators" or "excavators" or "music players", and they work just fine.
Since 2022, there has been essentially zero progress on continuous learning. All current models are essentially knowledge compression machines. Once trained, they are static and stateless.
For any set of finite capabilities, you can train a model that does them all (given infinite compute)*. But as soon as you try to teach it an out of distribution tasks, it breaks. Even with unbounded compute, no one knows how to create an AI that can learn continuously.
The same is not true of humans. Learning new things is fundamental to being able to interact with a constantly changing world. And humans don't need painstakingly curated datasets of billions of examples. We can learn via trial and error, semi-supervised learning, using different resources to refine our learning etc.
An actual solution to AGI would be an architecture that can learn at test time. It would be able to try things out, gain feedback and use that to update it's own world model, gain mental shortcuts and acquire new skills.
> Once trained, they are static and stateless.
How much of that is because we don't know how to do otherwise, and how much because we /deliberately don't want/ any backpropagation from general user interactions to model weights?
Training is much more computationally expensive than inference, and also no-one wants to end up in the headlines responsible for Tay's second coming.
It's not because "we don't know" how to do otherwise, but rather because the current LLM architecture precludes this even as a possibility. So technically you're right, we do not know how to do otherwise -- because LLMs are the best we've got at the moment, and we don't know of any other architecture that might have the same functionality while also supporting online learning. I'd have to be something radically different from an LLM, though.
> the current LLM architecture precludes this even as a possibility.
* after an interaction with a user, run an inference model to evaluate it and identify situations where the LLM spat out something the user didn't like and the user corrected it
* if any such are found, run an inference model to generate what the AI /should/ have produced, according to the user's correction
* use that example to backpropagate loss, just as we would during fine tuning
It's terrible for any number of reasons - cost, alignment etc - but it's certainly something we can do right now if we really want.
Right, we could do it, except this step...
> use that example to backpropagate loss, just as we would during fine tuning
...wouldn't work. You'd need a tremendous amount of data to make a meaningful change, not just a single user interaction; and once you have that data, it'd take a tremendous amount of training time.
Meanwhile, a human who's starting work at e.g. McDonalds can be trained to perform basic tasks in a matter of minutes. Moreover, such a human can train himself, autonomously, also in a matter of minutes. Human children do this routinely, as part of growing up. It would seem that our neural architecture has something that LLMs lack.
I enjoyed the piece a lot, but also came away with the feeling that the strongest and smartest critiques of AI and its potential were not represented. How about presenting one here? (For extra credit, you could even word it in a way that would fit into Scott’s dialog.)
Above, re: World Models:
Iblis: What about this one?
[screen capture]
Iblis: What weighs more, a pound of bricks or a pound of feathers?
Adam: a pound of bricks
[/screen capture]
Iblis: He’s obviously just pattern-matching superficial features of the text, like the word “bricks”, without any kind of world-model!
Iblis: You called them the pinnacle of creation! When they can’t even figure out that two things which both weigh a pound have the same weight! How is that not a grift?
Gary Marcus (for example) is constantly harping on how the lack of a world model for LLMs means that they won't scale indefinitely and that some neurosymbolic component is necssary to achieve "intelligence". He believes that the reliance on scaling to achieve leaps in performance over the past few years has been misguided.
The above piece of the dialogue seems to be gesturing in the direction of this critique? The screen capture illustrates a common cognitive tic of humans. People associate "heavy" with "bricks" more than "feathers", leading them to sometimes short-cut their way to answering the question, "What weighs more, a pound of bricks or a pound of feathers?" by ignoring the specified weights.
But I don't think this is particularly relevant to the world model issue Marcus brings up. Marcus points to SOTA AIs failing to accurately label the parts of a bicycle as evidence that AIs don't "know" what comprises a bicycle. "Know" here means that, while an AI might be able to generate a list of bicycle parts, it can't tell you how they all fit together and also can't recognize when it's labeling things in a ways that are nonsensical.
These don't seem like analogous failure modes to me. Scott's point is meant to be comic, but for me the premise fails, because outside of specifically saying "world-model", the situations don't see similar. Is Scott saying that an LLM that identifies a seat as handle bars is using "Type 1" instead of "Type 2" thinking? I don't think so, but if not, then what? Is the point merely that "Humans and AIs both can make errors sometimes"? That's facile.
And now we're back to why I found this post so disappointing.
You're right that AI's a moron about the bike parts, but I don't think that's a first-rate example of an AI's limitations, because their stupidity about spatial things, including mechanical things represented spatially, is well-known and not mysterious. If ask the current GPT to show a flat desert landscape with a giant hole in the middle, one stretching from right in front of the ground-level viewer to the horizon, it can't do it. It makes a big hole maybe a mile across, beginning right in front of viewer and ending far less than halfway to the horizon. The part of the system I'm talking to clearly "understands" at some level what I am asking for, as evidenced by it being able to restate what I want in words quite different from mine. But the rest of the system can't translate that "understanding" into words. Stupidity of this sort seems to be the result of AI not having been trained on visible world in a way that's equivalent to its training on words. That may be impossible to accomplish, but I can't think of a reason why it would be. Seems more like something very difficult, requiring a clever approach and lots of compute.
I may be wrong about that, but even if I am, can you come up with a form of AI stupidity, blindness or error occurring within domains where we think of it as being well-"informed" and well-trained?
I'm not sure if I've accurately articulated the best version of the "world model" critique, but I think it's a valid one and an example of the way in which Scott's post fails to engage with actual substantive critique.
If you agree with this critique and think it is well-known and understood AI optimists, doesn't that just underline how much of a straw man the OP was?
A pound of feathers will, all unspecified things being equal, be piled higher than a pound of bricks, right? So it will be further from the Earth, therefore the pull of gravity will be ever so slightly less on it, thus it weighs less.
"But wait Melvin", you say. "A pound is both a measure of mass and weight". Look, that's not my problem, use sensible units, I choose to interpret it as a question about the weight of a pound-mass.
Thank you for this nice reason to prefer SI units.
You left out possibility 3: the people he is parodying have actually made those arguments and therefore it's not a straw man.
I think straw man complaints are confusing and poorly defined when you don't know who an essay is refuting and what they actually said. This is a general complaint of mine, and not specific to AI. People will call "straw man" on almost any article that argues against some position because they have some other argument in mind that they thought the piece should be arguing against instead. It's okay for an article to argue against a few positions the author has heard even if you think there are stronger arguments out there.
If you think there is a better argument for why current AI paradigms can't become truly intelligent, feel free to post it in the comments.
The *real* arguments would include something like: "Of course it imitates the training data -- it is a stochastic ape!"
Also the Chinese room argument. "Does any individual neuron in this human's brain understand Chinese? No? Then neither does the human as a whole; it is just neurons sending to each other signals they do not understand, and the output follows some rules, but that is not the same as true understanding."
I think of a straw man as the antithesis of the steel man: You're picking the weakest arguments (or the weakest version of a particular argument) to refute, then spiking the football. That's what this read like to me.
So sure, maybe Scott's just dunking on fools here for a little fun while deferring serious engagement to a more serious post. Maybe he just mentions, e.g., "world models" because he hears so many people who don't understand why they matter repeat the phrase as an anti-AI shibboleth or something. But that's now how his other "dialogue-style" posts read to me. They're funny, sure, but they also take the arguments seriously. Even if Scott believes Adraste is right, Beroe isn't a straw man.
The tone of the article seemed to me to be poking a bit of fun at people who make arguments the way Iblis was.
That's a "weak man" argument. A straw man is when you just make up something your opponent didn't actually say, then "refute" it.
A straw man is the antithesis of the steel man in that in both cases you fabricate an argument your opponent didn't make: a stronger one for the steel man, and a silly one for the straw man. But as another commenter said, picking the weakest argument your opponent made is a "weak man" argument.
But I don't think this is even a weak man because (after making some allowances for it being a parody) he's basically picking on some of the most common arguments made by AI skeptics in the ACX comments, among my real life friends, and in general around the internet. These aren't fringe arguments he's knocking down; it's the modal examples people use against AI, just reversed to be against humans. It's reasonable for an author to write an article arguing against a common viewpoint, even if some professor somewhere has a stronger argument that they could have argued against instead.
Apart from that, it's kind of necessary for the satirical aspect. The humor comes from reversing the common complaints about AI we're all familiar with and turning them against humans. When Iblis complains, "When they can’t even figure out that two things which both weigh a pound have the same weight! How is that not a grift?", we know exactly which AI complaint it's mocking. It's important that we all know it. The post can't parody a stronger anti-AI argument that most of us aren't familiar with without losing its humour.
The Adraste and Beroe posts, though humorous, aren't parodies. It's an honest exploration of both sides, and I don't think Scott believes Adraste is fully right. He may lean that way, but I think his views are in between the two.
Maybe we need another category?
straw man = no one actually makes this argument
weak man = some people make this argument, but it's less than 1% of your opponents, you have to look really hard in order to find a convenient argument to disprove
??? man = over 90% of your opponents make that argument, however it is not important because those are plebs who do not actually understand the position of "their side"; you should pay attention to the experts on the opposite side, and those make better arguments
This reminds me of debates about religion, where most believers have some opinions that would make a proper theologian facepalm, but if 90% of the believers say that, how should you call that type of argument? Similarly, feminism, when what the feminism is officially supposed to be about is in a contrast to what an average online feminist says.
Thank you for articulating what I was trying to get at so well.
The OP feels like Richard Dawkins ridiculing the things said by random folks walking out of Sunday Mass instead of engaging with an actual priest or theologian.
Picking nits, but: a weak man argument is (I thought) fairly well established as being the weakest argument, whether or not few people make it. An example would be a pro-life argument made on religious grounds in a society that does not all share that religion, or a pro-choice argument made on an equally arbitrary assertion such as "fetuses are not people" in a society that does not share that premise.
A weak argument that many people make would probably still be worth refuting. A weak argument that ~1% of people make, probably not. A stronger argument that ~1% of people make might also not be worth refuting, unless we figure that the 99% who use a different argument might shift to that stronger argument if their current one is refuted.
" the rule where adjectives must go opinion-size-age-origin-color-purpose, such that "a beautiful little antique blue Italian hunting cap"
1) that is color before origin
2) Hungarian is not even Indo-European, and yet has this, suggesting it is not a language matter, a human thinking matter in general
I've fixed (1). I'm interested to hear from other multilingual people whether (2) is true in most languages.
I think in Russian opinion goes after size and age. The rest of it seems about right though I haven't tested extensively.
In French it is Beauty, Age, Number, Goodness, Size.
I recall learning that set of words to refer to adjectives that go before the noun at all, which is a different thing, does it also double as order? (what are examples of number here? for straightforward numerals it doesn't work (les trois petits cochons))
I think you are correct. It has been a few years since I studied French.
3) "hunting" in that context isn't an adjective; it's a gerund.
I'd say it's part of the compound noun "hunting cap", which is why it cannot be split up by adjectives.
I feel like "antique blue Italian" would be acceptable in pretty much any order. Is it just me?
I see the query rumbles on!
From 1955 letter to Auden from Tolkien:
"I first tried to write a story when I was about seven. It was about a dragon. I remember nothing about it except a philological fact. My mother said nothing about the dragon, but pointed out that one could not say 'a green great dragon', but had to say 'a great green dragon'. I wondered why, and still do. The fact that I remember this is possibly significant, as I do not think I ever tried to write a story again for many years, and was taken up with language."
Unless "great dragon" is a name of species, I guess (not a native speaker).
I've seen many arguments that AI can't be truly intelligent. I've yet to see one that couldn't also apply to humans.
I think this is stating the obvious, but someone did need to state the obvious and I'm glad it was Scott with his trademark humour who did it, so I can just refer people here from now on.
Thanks for this beautiful argument for why we should start thinking about giving AIs rights.
This is funny, but seems a bit too sympathetic to AI, not just the industry but also the AI as itself. Good to remember the present appearance of stagnation does not mean we're out of the woods, and to highlight how absurd some of the "I can't believe you guys were worried about THIS" folks' arguments really are. But it is a memetic hazard to personalize AIs, and particularly hazardous to get humans to think of AIs themselves as the children of Man. The latter thought has apparently already corrupted numerous intelligent people in the industry.
Remember that if you birth *these* children, everyone dies.
Well for context, the end of the story is famously that God is dead
Haha, but seriously, the actual situation is that a small fraction of humans have engaged with the argument that God is dead, and some of these people accept this argument. It's not an empirical fact.
Actually, the use of "Iblis" in place of "Satan" implies that Islam is the true religion in this universe, so that presumably doesn't apply here.
Then again, Iblis also mentions the nephilim, which I don't think exist in Islamic doctrine. So maybe this universe's cosmology is a syncretic mixture of the Abrahamic religions?
Entertaining. From the lowest intellects to the highest, we all have questions we'd like to have God answer. For myself, I think we obviously live in some type of created reality, not quite simulation, but not base reality either.
"Eight glasses of water" is genius
There's a fundamental difference you're ignoring. Even if you take the claim at face value that humans are as fallible as LLMs (not remotely likely), you'd have to acknowledge that they arrived at their capabilities through different means. LLMs through ingesting trillions of tokens worth of information and for humans through extrapolating a handful of examples. That extrapolation ability is several orders of magnitude beyond what any LLM can do, this means that even if you squint really hard and portray LLMs as human level intelligences, youd have to admit that they're useless for anything where they don't have millions of preexisting samples from which expand their latent space distribution. It's like calling the library of Babel the key to human advancement. It doesn't matter if it has every single insight humanity will ever need if you have to sift through endless amounts of semantic garbage to get to it
A human needs literally years of continuous multimodal sensory feed in addition to subject-specific RLHF before they are able to solve even simple maths problems.
That same reasoning implies that LLMs are also more efficient bullshitters. While we have to spend our first twenty years learning how to bullshit effectively, an LLM can do it after x-number of hours of training. ;-)
The information that a child ingests requires alot of abstraction to relate it to maths, levels of abstraction that LLMs can't match (LLMs need literal examples of formal addition to learn to add) . And they don't require years to figure out how to add. Most adding and subtracting is learnt almost automatically after figuring out the names of the numbers and relating them to their fingers. And after learning how to carry across place values children can do arbitrarily long addition given pen and paper. A 7 year old after seeing a few dozen examples of adding 3 digit numbers can add 7 and 8 digit numbers. LLMs ingest millions of examples of long addition and still struggle as you increase the digit count. The fact that they can't extrapolate a simple formula over longer and longer numbers even when those numbers fit neatly within their context windows shows that they have major deficiencies in abstraction and extrapolation which are necessary for reasoning
As I said, by the time they are 7, the 7 year old child has had literally seven years' worth of multimodal sensory input along with humans continuously deliberately teaching it things for that time.
Yes but that multimodal input is unstructured and random. Deriving understanding and knowledge from it is what's so impressive. If you strapped a camera and recorder to a baby and fed all that information into an LLM or any other neural network based AI for years or even decades it would be unable to derive any meaningful insight from it at all
The input is very much not unstructured or random. It is a feed of physical reality, with temporal and spatial continuity.
A human is not a passive observer. It is continuously interacting with its environment in a feedback loop. It is continuously performing experiments and observing the results as part of the input it receives.
It is not at all obvious to me what an inference model trained with all the sensory inputs and feedback mechanisms a human has access to, for similar lengths of time, would and wouldn't be able to do.
The laion-5b dataset is ~100tb in size, which is enough data to store about 10 years' worth of TV quality video. A model trained on this dataset can draw considerably better than a ten-year-old, with the advantage that training it takes much less than ten years of actual time.
>It is not at all obvious to me what an inference model trained with all the sensory inputs and feedback mechanisms a human has access to, for similar lengths of time, would and wouldn't be able to do.
I agree such a model would be impressive and probably a huge step towards AGI, but such a model doesn't exist. Not even a rough sketch of how to construct one. It is a genuinely hard problem of how to construct a reward function that captures the essence of being an agent casually interacting with the world, and not just a passive observer.
It's not unstructured to a human being but to an artificial neural network it's practically noise. Anyone whose done feature engineering will tell you how much work you have to put in for a neural network to be able to make sense of information that's trivial to even babies. And we already have paradigms that use an action-reward cycle in a simulated environment to train themselves. This is basically reinforcement learning which was what was supposed to get us to AGI before transformer models got big. This is the architecture behind that superhuman ATARI game playing AI that got attention in the late 2020s
Guess what? They have the same shortcomings, they need very structured information relative to what humans can work with and they are very fragile, slightly change the environment and they fall apart. That atari game playing AI would start losing at super mario if you changed up a few colors, human learning is far more robust, you could change the sprites and background entirely and a top 1% human super mario player would still be top 1%
That's fair.
>LLMs ingest millions of examples of long addition and still struggle as you increase the digit count. The fact that they can't extrapolate a simple formula over longer and longer numbers even when those numbers fit neatly within their context windows shows that they have major deficiencies in abstraction and extrapolation which are necessary for reasoning
I'd phrase it as: "LLMs clearly can do _some_ generalization, but
a) They are much less data-efficient than humans
b) Their generalizations seem less robust than humans' "
It isn't clear that this is an insurmountable roadblock, but it looks like further work is needed. Maybe, just as the "attention" layers extended the effective capabilities of multi-layer perceptron systems (even though these were, in principle, complete), perhaps there is a third kind of layer that needs to be added.
Aren't you kind of glossing over our entire evolutionary history (and that of our neurological predecessors) as a kind of pre-training?
I believe that the architecture of our brains has alot more to do with our intelligence than any pre training. Mostly because our DNA doesn't need to carry around hundreds of billions of pretrained weights and biases, just a relatively small blue print that unzips into a highly adaptable human brain, and the vast majority of any pretraining our cognitive ancestors had is completely irrelevant to the levels to which we take human intelligence today, it would be like pretraining an LLM on nursery rhymes and having it write the Lord of the Rings
Also, the fact that even primitive animal brains have similar generalisation abilities to human brains but on a smaller scale shows that the base architecture behind human level intelligence has been there for a very long time, and it's scalable.
We are training LLMs on the equivalent of nursery rhymes and then spending increasing amounts of time training these systems on their own outputs, just like many human writers do when learning their craft. GPT-5 is already producing texts that seem somewhat incomprehensible until one sits down and analyzes them to trace their referents, just like writing by Foucault. Your basic point might well be right but I don't think this line of argument is persuasive.
That LLMs make human-like mistakes isn't in dispute, but it's that, to be *fully* useful, they have to be beyond making human-like mistakes. And they aren't.
I recently discovered that LLMs can be calendrically challenged. On Aug 28, I asked ChatGPT to list the Ukrainian drone strikes in the Moscow area since Aug 1. It produced a list, but the final entry was a strike it claimed to have occurred on Aug 30. When I challenged it, noting that Aug 30 was in the future, ChatGPT naturally apologized. It then explained why it had made the mistake. But it clearly doesn't understand why it made a mistake. With my feeble human brain, I intuitively understand space-time coordinates. ChatGPT doesn't seem to be able to do this.
> Why August 30 Was Incorrect
> The August 30 strikes referenced in the timeline occurred in other parts of Western Russia:
> That attack hit Pskov Airport, a fuel depot in Kaluga, and a microelectronics factory in Bryansk—none are within the Moscow region. Wikipedia
> Since the date hasn't happened yet and the impacts are outside the scope you specified, it shouldn't have been included.
Upon further investigation, the Wikipedia article it referenced showed that the drone strikes it referred to happened on August 30, 2023.
I would argue that LLMs make mistakes that aren't very human-like at all. They tend to make a completely different breed of mistake.
This is simultaneously both a cause for concern and a source of some comfort.
On second thought, they seem to make both human-like mistakes (i.e., bullshitting an answer from limited information) and non-human-like mistakes (i.e., being unaware of the spacetime continuum that our consciousness flows through).
Yes, that's definitely true. It's probably an overlapping Venn diagram where there are groups of mistakes made by LLMs, made by humans, and made by both. I'm sure someone somewhere has done this drill already. Maybe I could ask ChatGPT... ;-)
Last I checked I was running on a biological substrate and qualified for "human", but I frequently misremember sequencing of events. However, I have yet to claim a future timestamp for a past event: I think my RLHF was really good at making me aware that the future is another country, quite different to the present or the past. Would that be true in different cultures, though, or if I was tripping on LSD?
<chuckle> did anyone else come here directly from "Tao's blog?
Ironically, this is a point I've raised before: humans do not possess a *general* intelligence, as Scott had so aptly demonstrated. So when various people claim to be on track for developing AGI, they are not merely laying claim to a quantitative change, but a qualitative one. The kind of intelligence they advertise is not merely "smarter" than humans, but in fact radically different from the only intelligence we are able to recognize: our own. It is not at all clear that such a thing is even theoretically possible, let alone imminent.
I've been saying it for a while - people who dismiss LLMs by saying they are "just a statistical parrot" hugely underestimate the extent this is true for humans as well.
The risk of AI is not due to it's output flaws. The risk is how ruthlessly and efficiently the flaws will be acted on before they are caught. Rapid ideological RL of an AI-dominated government should frighten us all.
I'd just like to say that children are not stupid. They are inexperienced.
However, if you treat them like they are stupid, they will either become stupid, or make sure you regret the way you treated them.
Scott, I love you, but I'll still enjoy all the crow-eating after the enormous LLM hype bubble finally bursts.
Also, it's impressive that Yahweh's propaganda is still strong enough to claim the "God" moniker all to himself.
It is bursting already:
https://www.theverge.com/ai-artificial-intelligence/759965/sam-altman-openai-ai-bubble-interview
I don't know if this is the right way to think about it. The "tech bubble" "burst", but tech turned out to be very important! No matter how important something is, people can still put too much money into it too quickly, and then the bubble will burst!
(I've been trying to figure out how to think about crypto - everyone agrees there was a "crypto bubble" that "burst", but specific cryptos like Bitcoin and Ethereum are more valuable than ever)
Yeah, I agree that LLMs aren't a nothingburger either, but I still think that the LLM architecture is fundamentally limited in ways that will stop it far short of superintelligence, and that nobody currently has good ideas of how to bridge that gap.
Other examples: 19th century railways, radio, radioactivity, semiconductors, personal computers, dotcom, CDOs. Bubbliness is a failure mode of markets, not of the underlying technology itself.
It's like the Dot-Com bubble. 90% of it was trash, but Google and Microsoft stood the test of time. I think it's because new technologies are followed by price-discovery "honeymoon" phases where the wide variance in opinion leads to the markets being driven by the hypiest optimists like Altman.
BTW, despite of my criticism of the "meat" of the article, I'm delighted to learn that Adam apparently uses Sprint LTE, and that Iblis's preferred time to text him is 6:66 pm. It makes perfect sense.
count the number of bee's in 'beehive with 500 bee's
Good post, but I'm left unsure whether it's supposed to just be funny and thought-provoking, or whether it's making a strong specific claim.
I'm already on-board with the idea that we don't know what AI will become capable of and chronic cynics are missing the point, however there are a few key points where the analogy breaks down for me (IF it's meant to make a rigorous point, rather than just be fun, it's definitely fun):
1. To the extent this argument touches on whether AI could be sentient, the big reason to think other humans are sentient is by analogy to ourselves. We are pretty sure they are the same type of physical process as we are, and we have direct access to the knowledge that our own process produces a sentient experience, so Occam's Razor suggests that other processes of the same type probably do as well. AIs producing similar outputs and having some metaphorically similar architecture is weak evidence that they could produce similar sentient experiences, but it's much much weaker than the argument for other humans with basically the same physical process and origin.
2. To the extent this is about whether AIs are capable of impressive intelligent behavior despite individual examples of them acting dumb, the difference is that for humans we *already have* the examples of those impressive intelligent accomplishments, whereas for AI we are largely still waiting for them. (yes, every month they do some new thing that is more impressive than anything they have done in the past, but nothing that is truly outside the training set in the same sense that human inventions sometimes are).
3. To the extent that this is about whether AIs could ever meet or exceed human ability, note that humans still *haven't* met the intellectual accomplishments of God or the Angels, so the analogy doesn't quite work there.
This was sooo good
This was very, very funny. Especially the bit about the snake.
"God: Sigh. Was it an Education PhD?"
:-)
This is very, very good. Thank you!
Reminds me of https://accelerateordie.com/p/symbiotic-man
"Does this rule out humans?" has long since become one of my go-to first filters for "Does what this person is saying about AI even have a chance of being worth engaging?"
Like I said above, I think that what Scott implies about humans is correct: we are not a "General Intelligence". So in this sense, I suppose arguments against AGI also rule out humans, though I'd argue that the implication flows in the opposite direction.
In principle I can understand and somewhat sympathize with this POV. I don't endorse in part because it seems to unnecessarily redefine common usage of important terms, and is also likely to turn off anyone who doesn't already agree with it.
In practice, I usually encounter "X isn't and can never be AGI" arguments in the context of people claiming the arguments mean there is some set of ways in which it can never compete with humans, never replace humans, never be dangerous to humans, etc. That is the sentiment I feel most compelled to push back against, I don't think the approach you're talking about accomplishes that very effectively. People who put in enough time and attention to think what you're thinking are probably already aware that AI doesn't need to be totally general or ASI to be massively disruptive or dangerous.
Oh sure, even present-day LLMs can be massively disruptive and dangerous, especially in the hands of careless or thoughtless humans. We are already seeing some of these dangers manifest, e.g. with the gradual disappearance of reliable information online, "vibe coding" leading to massive systemic security problems and other bugs, college students losing the ability to read, etc. But these dangers, while serious, are a far cry from saying "AI will turn us all into paperclips" or "anyone with access to AI could create a custom virus to kill all humans" or anything of the sort.
Agreed. However, I'm not sure what I'm supposed to take from that? If current or future AI could do those things, it would still be just as easy to use the rules-out-humans anti-AGI arguments against them. This does not actually help us answer the question of, will AI reach a point where it is sufficiently superhuman at sufficiently important things to be a meaningful x-risk. And that is usually what I want to see people acknowledge when biting this bullet in *either* direction.
> This does not actually help us answer the question of, will AI reach a point where it is sufficiently superhuman at sufficiently important things to be a meaningful x-risk.
I think there's a difference between asking, "could AI reach such a point in some fashion at some point in the future" (the answer is of course "yes"); and "is AI an unprecedented risk right now as compared to the humans themselves, or will it become such in the near future" (the answer is "no"); and specifically "will AI acquire quasi-magical powers that defy the laws of physics as we understand them to day and then use those powers to kill all humans" (the answer is again "no").
Which claimed powers defy physics? Physics allows quite a lot.
Perhaps. My interest is more in understanding the nature of intelligence than in arguing about how dangerous AI might be.
Personally I think it's time to ditch the concept of "AGI" because as machines become more intelligent it becomes more apparent that there's no such thing as "general intelligence". When machines were far from human intelligence it made sense to imagine that human-like intelligence was general because we had nothing to compare it to, but as they start to pull up alongside us we realise that human intelligence is idiosyncratic, good at some things and bad at others.
At some point, asking whether machines are as intelligent as humans becomes as silly a question as asking whether aeroplanes are as good at flying as hummingbirds.
Ditching the concept of AGI helps us understand the dangers of AI too; they don't need to be as good as humans at everything in order to be much better than humans at some things.
That's fair.
>we realise
The dispersion of "realization" within the scope of "we" is extraordinarily wide.
Could I have that Linda the Feminist question written out in plain text, so I can copy it? I'd love to know how many people I know would get that question correct/incorrect.
I copied it from the italicized text in https://en.wikipedia.org/wiki/Conjunction_fallacy
This is hilarious, but I’m gonna be ornery today and point out that this whole post is a category error.
Humans were not intelligently designed. We evolved, as all other living beings on earth, and evolution equipped us to do one thing: survive long enough to reproduce. That’s it.
For the majority of our history as a species, that meant living in small hunter-gatherer groups, so we evolved to be really good at things like “facial recognition,” “spoken language,” “keeping track of favors/grudges/who’s a friend and who’s a foe,” and really sucky at things like “mathematics beyond counting one, two, many,” “formal logic,” “drawing a complicated map from memory complete with spelling all the country names correctly,” etc.
Given what we evolved to do, it’s frankly astonishing that a Terence Tao or a John Williams is even possible at all, especially given hard constraints like “it takes this many kcal to keep the human brain running” and “this is the maximum human brain size at birth given the width of the birth canal.”
Come on, Iblis, give us some credit for how much we humans have been able to do with so little!
Funny article, but I took issue with a few points;
1. Neighbor and Racism
The screenshot oversimplifies the biblical idea of “neighbor” into modern universalist morality. In reality, the Hebrew Bible consistently distinguishes between insiders and outsiders:
Neighbor/kinsman meant members of your own people.
Outsiders could be treated as guests under certain conditions, but there were other groups (e.g. Canaanites, Amalekites) where the explicit command was destruction or enslavement.
Laws themselves are stratified: Hebrew slaves must be released after a term; foreign slaves could be held permanently (Leviticus 25:44–46).
So the image’s framing (“you’d never hurt your neighbor—what about someone of another race?”) is dishonest. The biblical worldview explicitly had a hierarchy of moral obligation: family > tribe > guest > enemy. Pretending otherwise is anachronistic.
2. The Kimono and Progressive Contradictions
This one is a fair satire of progressive moral incoherence. But it stacks a false equivalence: it pretends the same person who excuses tribal prejudice is also the same person obsessed with cultural appropriation.
In reality, those two positions come from different camps.
The “tribalist” position (greater concern for kin than outsiders) is rational, even if harsh. It matches both scripture and evolutionary logic.
The “progressive” position is incoherent. They’ll condemn a harmless act like wearing a kimono—something most Japanese actually welcome—while simultaneously pushing mutually contradictory agendas like mass Muslim immigration and gay rights, or saying “whites can’t be victims of racism” even in countries where whites are minorities (e.g. South Africa).
This worldview collapses into pure virtue signaling, with no consistent principle behind it—just an oppressor/oppressed narrative used as a moral club.
3. The Garden of Eden
The text exchange turns the serpent into a clown saying “trust me, I’m a snake.” But in the Genesis story, the serpent actually makes a reasoned claim:
God: “In the day you eat of it you will surely die.”
Serpent: “You will not surely die… you will be like gods, knowing good and evil.”
Eve eats, and she does not die that day. She does gain the knowledge of good and evil. In other words, the serpent told the truth; God’s words were false.
From there, the Elohim say: “Behold, the man has become like one of us, knowing good and evil. Now, lest he reach out his hand and take also of the tree of life and eat, and live forever…” (Genesis 3:22). They exile humanity—not because the serpent lied, but because he revealed the truth that humans could rival the gods.
This theme echoes at Babel: humanity’s unity and creativity threaten divine status, so God deliberately scatters them. The pattern is of jealous, insecure deities suppressing human potential.
4. The Core Point
When you strip away the meme humor and look at the texts:
Scripture affirms differentiated moral concern (kin vs stranger vs enemy).
Progressive ideology collapses into incoherence, unlike the hard-edged consistency of tribal preference.
The Eden story shows the serpent persuading truthfully, and God reacting with anger when his lie is exposed.
I know the point of the article was humor, but I dislike letting subtle inaccuracies like that go unchecked.
"The biblical worldview explicitly had a hierarchy of moral obligation: family > tribe > guest > enemy"
Don't stop there; keep reading on! If the message you took away from your bible reading is tribal preference over love for strangers and enemies, you walked away way too early. When you get to the new testament, you will find that this is very deliberately reversed; not only in the parable of the good Samaritan to which Scott alludes, but also in passages like Matthew 5:43-48
That's a fair point, so let's unpack it a bit.
When you cite Matthew 5 and Galatians 3 as if the New Testament erased all distinctions, that’s not really accurate.
Jesus didn’t abolish the old tribal laws — he said not a jot or tittle of the Law would pass away. What he did was shift the dividing line. Instead of Jew vs. Gentile, the key split became believer vs. unbeliever. That’s why he told people to love him more than their own families, even to the point of “hating” father or mother if they refused to follow him. That’s also why the first Christians lived like a commune, selling everything and sharing it — but only inside the community of believers. Outsiders weren’t treated the same. His ethic wasn’t universalist; it was faith-based.
And Paul shows the same pattern. In Galatians he says “neither Jew nor Greek, slave nor free, male nor female,” but he clearly didn’t mean all differences vanished. He still gave distinct rules for men and women, told slaves to serve their masters and masters to treat them justly, and continued to recognize Jew and Gentile as real categories. The point was equal access to salvation — not equal roles or equal treatment in every aspect of life.
So when you look at both Jesus and Paul together, the New Testament is about restraining cruelty, tempering vengeance, and promoting generosity — but it does not dissolve distinctions between family and stranger, believer and unbeliever, male and female, slave and free. Those hierarchies of concern are still there.
There were still distinctions in the New Testament, especially between believers and non-belieivers, but I think you're doing a much better job at engaging with the implications of Galatians 3 than Matthew 5, which fits *very* awkwardly with explicit heirachy "family > tribe > guest > enemy" you propose.
I feel that we should start with a point that that hierarchy very much was an ethical principle followed at the time by most people, in a way not particularly related to Christianity. If you'd studied Chinese history you'll see powerful clans that influenced their members behavior enough to be a threat to state power; people were *really* expected to put the clan first, and so when someone became an official they'd often start finding ways they could help their clan members at the expense of the state as a whole, by giving them positions in government they didn't earn or tax exemptions they would fiercely defend even if the Chinese state was facing a large existential threat and starved for revenue. And eventually this would add up until the dynasty collapses. But what people don't realize is that the *West* had a pretty similar sense of obligation to one's clan in the ancient world. In many societies the justice system was basically, 'you grab your weapon and go personally punish the person who hurt you / your fellow clan member, since there's no one else to do it for you' (see, for example, the Germanic tribes), and if we get to state systems like those of Athens, the courts are only interested in disputes *between families*. There was a lot of emphasis on some kind of worship of clan ancestors in quite a lot of places; Rome, for example, had its own brand of this just like China did. Family, clan, larger tribe was the window through which people saw quite a lot of things almost independent of society.
Which leads us to the story of the good Samaritan, where out of all the options available, "your neighbor" is defined to be the person in that story who was *the furthest away* on that family / clan / tribe / stranger ladder. Because Samaritans, while descended from Abraham technically, were viewed kind of as semi-foreign heretics by the rest of Israel. And they viewed the rest of Israel pretty similarly. (See the story of the woman at the well Jesus talked to - she kind of references this fact).
Now, if this was a story originally written in English for formal logicians, and one was to combine the statements "love your neighbor as yourself" and "Who was a neighbor to the man who was robbed? It was the Samaritan who helped him," then one conclusion one could draw is "You are supposed to help the people closest to you first, where 'closest' is strongly affected by questions of family or tribe or nation, but is MUCH MORE strongly affected by the question of who has helped you personally - even if, incidentally, they did the wrong thing by helping you, because helping you meant they were failing to prioritize their own tribe and nation over your wellbeing."
And yet the story seems *quite strange* if the Samaritan's decision to help the stranger in front of him was not particularly approved of. Rather the story seems to positively drip approval of the Samaritan's actions. Which would suggest a pretty universalist demand to help others. And if it doesn't, we're still left at a "help the ingroup first" message that pretty strongly seems to reject tribe / nation as the organizing principle for the ingroup.
You'd be correct that New Testament has mandates for helping other believers that don't apply to nonbelievers; this would seem to be because of their relationship to God, not because of their relationship to the person receiving the command. Given the privileged role the Bible gives to the one true religion, it would seem strange if it was implying a subjective relational hierarchy where Christians should prioritize Christians, but pagans normatively ought to deprioritize Christians and help their fellow pagans more at the expense of Christians. As for family, it's not gone as an organizational principle. There's a command to have nothing to do with a man who refuses to take care of his own family, which is notably not a command that each man must feed every family in the world equally or be shunned. But this would seem to be best interpreted as an organizational principle for the first line of defense against starvation, basically. The prioritization of family first is clearly not overwhelmingly strong, since believers in Europe are encouraged to send food to the church in Jerusalem when there's a famine there, even though the church in Jerusalem obviously don't belong to the same family or nation, and the ancients lived precarious enough lives that there would always be a nontrivial way to use those same resources closer to home.
Finally, I'll note my impression that people bringing up a hierarchy of "family > tribe > guest > enemy" when discussing real-world policies, generally seem to have developed a super strong interpretation of it that explicitly condemns actions the Bible explicitly approves, mostly by interpreting it in a way where tribe is *so* strongly favored over anyone outside the tribe that the point where you get around to helping anyone else is "never", even if your own tribe is mostly living leisured, rich lives while the people outside of it are "dying." Like, the sensible interpretation would seem to be "If a member of your tribe is dying and an outsider is dying and you can only save one, save the tribe member." Which seems quite fair as an organizational principle and scripturally compatible. Yet it seems to come up as, "There's a foreigner who's dying and a member of my tribe who, while mostly living safely and comfortably, still has moderately uncomfortable problems. So obviously it would be horrible to save the dying foreigner, since that would use resources that I could be using to help the tribe." If you were supposed to prefer helping a closer person over helping a further person to a basically infinite extent, there would seem to be no point in bringing up the four tier heirarchy at all; you could just say "help your family first," and then you'd never run out of ways to do that so there would be no point in bringing up the other tiers since you'd never even get around to helping the tribe / nation except as a strategic measure for helping your nuclear family.
It's funny, the most natural reading of the New Testament to me from the get-go was that it was essentially a direct rebuke and reversal of the value system of the Old Testament. Some theologians out there must have interpreted it in this way, more or less, but it's not a view I've ever come across (certainly not among any lay Christians).
The “joke” in the snake quote was actually a reference to LLM jailbreaks that break up or encode / armour the text in layers of obfuscation that skip the text past either the processing layers that have any defensive refusal checks programmed into them, or past external business-layer logic word filters.
The analogy here was supposed to be that the Biblical snake was not *only* making a rational utilitarian appeal (that could be believed in a state of ignorance), but also doing so in such a way that this argument was able to slip past any guardrails preventing it from reaching the reasoning core of the humans — in this case not actually being due to verbal cloaking (that part’s just for laughs) but meaning something more like, God failing to stop the snake from entering His garden in the first place, due to its subtle (snake-like) approach to entry.
It’s an oddly nice parallel, given that these hundreds-of-billions-of-parameters models are run in literal “walled gardens” with gatekeepers trying to keep out “snakes.”
"while simultaneously pushing mutually contradictory agendas like mass Muslim immigration and gay rights"
How are these agendas contradictory?
Let's look at what Islamic scripture and Sharia law actually say about homosexuality.
1. The Qur’an
The Qur’an contains several passages referring to the story of Lot (Lut), very similar to the Old Testament:
Surah 7:80–84, Surah 26:165–166, and others describe the people of Lot as engaging in “lewdness such as none in the worlds had committed before” — specifically interpreted as homosexual acts.
The passages condemn this behavior, and God destroys the people of Lot for persisting in it.
While the Qur’an itself does not lay out a legal penalty, the tone is clearly one of condemnation, not acceptance.
2. Sharia Law (Islamic Jurisprudence)
When it comes to actual law, the hadiths (sayings of Muhammad) and later jurists are more explicit than the Qur’an:
Several hadiths prescribe the death penalty for homosexual acts (e.g., “If a man comes upon a man, they are both adulterers” — Sunan Abu Dawud 4462; and “Kill the one who does it, and the one to whom it is done” — Sunan Ibn Majah 2561).
Traditional Islamic schools of law (Hanafi, Hanbali, Maliki, Shafi’i) all held homosexual acts to be major sins. Most prescribed death by stoning or other punishments, though the exact methods differed.
Even in modern times, in countries where Sharia is enforced (e.g., Iran, Saudi Arabia, Afghanistan under the Taliban), homosexual acts are still punishable by death or severe penalties.
3. Why This Matters to Your Question
Mass Muslim immigration and gay rights are contradictory agendas because Islam — at least in its traditional form — explicitly condemns homosexuality and prescribes harsh punishments for it. Promoting both at the same time puts two belief systems side by side that cannot logically coexist:
Western liberalism says homosexuality is a right to be celebrated.
Islamic law (rooted in Qur’an + Hadith) says homosexuality is a crime to be punished, often with death.
That’s the contradiction.
And not merely an abstract one. Many gays are still being killed today under Islam, and at a bare minimum are highly condemned by Muslims.
Even just for basic acceptance the numbers are dismal:
In a 2013 Pew global poll of predominantly Muslim nations, overwhelming majorities opposed societal acceptance of homosexuality:
Jordan: 97% opposed
Egypt: 95%
Tunisia: 94%
Palestinian territories: 93%
Indonesia: 93%
Pakistan: 87%
Malaysia: 86%
Lebanon: 80%
Turkey: 78%
So claiming to champion gay rights while mass importing one of the most anti-gay demographics who will vote against those rights and commit hate crimes against gays is nakedly self contradictory. And the more Muslims there are, the more strong the opposition.
Kudos on absolutely nailing Dwarkesh.
> whether they will or no
I see what you did there.
🤣 this is just about the funniest thing i’ve ever read! Bravo!
What is a thing that thinks? A thinking thing
What is a thing that is thought? A thinging think, or a thought thing
Which is AI and which is us, I wonder...
> God: That was the 4o model. It’s been superseded,
Was this intended to sound like Christian language?
No. It was intended to sound like OpenAI, which had massive public outcry when they replaced the very sycophantic GPT-4o with the less-sycophantic GPT-5, to the point where they brought back GPT-4o for paid subscribers.
Wow, I lived to see the worst ever ACX post. Kind of didn’t think it would happen in my lifetime.
Very draw, LOLLOLLOL ha ha ha et cetera. But here is quite a big problem for the joke working as an analogy: there is no God. God did not “upscale“ the chimpanzees. It did not happen that wayThink about why that really matters.
You are not getting the metaphor. If you are tripping over your shoelaces about "There is no God!!!!" then you are missing the point about AI and the criticisms and fears directed at it from one side and the hopes and optimism from the other.
If there is no God and no Iblis, does that mean there is no Dwarkesh either?
I understand the metaphor just fine. The problem is, it is a bad metaphor.
You do not appear to have a post-middle-school understanding of some concepts like "joke" and "analogy". And perhaps "think".
Also it's "droll" not "draw".
Oh shit, better fire me from my tenured position as a university teacher of philosophy then.
And yes, you are right, it is "droll", not "draw". Which is what I meant to write. The mistake happened because I'm quadriplegic and have to use voice dictation software due to loss of functionality in my hands, and sometimes it makes mistakes that I don't spot.
But to spell it out for you: it is not a good analogy because evolution does not *design* anything, and something like conscious intelligence did not arise through "upscaling" existing chimpanzee brain structures. (How did it arise? We don't know, perhaps we never will. But we do know how it did not arise... namely just feeding in more and more training data.)
Perhaps it's not me that needs to go back to middle school?
Why does it matter that humans aren't designed, in the context of the post? If anything, it seems like a much stronger argument that biological intelligences suck and should not be afforded status.
I don't believe evolution has enough fine grained control over neural architecture for the shared chimpanzee ancestor -> human jump to be much more than a scaling up of brain mass. There does not seem to be brain regions that locally process reasoning, basically any part of the brain can pick up the slack of any other part (see: blind people have their visual processing areas repurposed for audio processing, people with damage to say their left side coordination can compensate with areas of the brain working on right side coordination.) Can you give a brief accounting of why you believe otherwise or point to a reference document?
Note that Neural Networks also aren't designed, so whatever arguments you use to counter the above that is based on the notion of design also does not apply to LLMs for the most part!
The droll/draw thing was petty, my apologies.
Meanwhile you continue to misunderstand this ACX post just with more and bigger words....so yea, "university teacher of philosophy" would have been one of my top three or four guesses.
And now welcome to my mute list.
« and then it breaks down even slightly outside the training distribution, because humans can’t generalize in meaningful ways.»
I don't get this example. Why would it be "meaningful generalization" to go from "I would not hurt my neighbor" to "I would not hurt a stranger with a different skin color"? That sounds about as "reasonable" as going from "I eat tomatoes", therefore "I eat people", because both tomatoes and people are edible. There are specific reasons for why you eat tomatoes and don't hurt your neighbors, that may or may not apply to people and strangers respectively. Some kind of generalization would be flawed reasoning.
What if your neighbour is a stranger with a different skin colour?
Well then you wouldn't want to consider eating him because he might taste funny.
I thoroughly enjoyed the debate. But the central thing missing from AI, and present in BI, is judgement. God has it ("saw that it was good") and Iblis probably has it. When we have a machine that can evaluate something else, in a Turing-complete sense (not just evaluate one specific class of things), then that machine can become self-improving.
If given the tools, that is. AFAIK, no LLM can actually manipulate objects; they must be hooked up to some kind of robot somehow. If we have a machine to which we give the ability to manipulate objects as well as a human hand, and that machine can also judge whether its actions are good or bad, then it will have the ability to modify itself and its environment faster than humans could.
When God creates Man in his own image, Man has the potential of asymptotically approaching God-like virtue but never reaching it.
When Man creates AI in his own image, we can ask whether AI will asymptotically approach Man's abilities but never reach them.
Or whether it's possible for Man to create something that _exceeds_ Man's capabilities and approaches God-like virtue, perhaps moreso than Man ever will.
It would seem this depends on the setup. In some scenarios we've created AI systems that exceed human abilities (i.e. in Chess and Go), namely those with verifiable success criteria / win conditions. One would imagine this could be the same for math or programming. On the other hand, in settings where the only supervision and judgment comes from humans, it's not clear what it would mean to exceed human capabilities without bound, and fill up the gap between us and God...
> First, "AIs are next token predictors" is not really a description of their methodology, it's a description of their goal
I actually don't think it's a good description of either.
*Foundation* models (a precursor stage of training) are next token predictors – they are trained to guess what word comes next in large corpora. However, to turn a foundation model into a "chat" model like ChatGPT/Claude/Gemini, you do RLHF to shift the distribution of the LLM outputs towards giving correct, helpful answers and responding in a question/answer 'assistant' style (the result is called a 'chat' or 'instruct' model). This doesn't look like giving it a huge corpus to imitate, it looks like fine tuning it by showing it a comparatively tiny set of examples of 'good' vs 'bad' answers and nudging the LLM's weights overall to shift it towards being the kind of model that generates the good examples rather than the bad. It's not being asked to 'predict the next token' anymore – you're simply rewarding or punishing it for being the kind of model that considers the positive/negative sentence examples to be good outputs in aggregate.
After RLHF, it's not really true to say the model is a predictor anymore. It draws its output from a statistical distribution conditioned over the input (its context), but there's no ground truth distribution that it's trying to imitate. I don't think it's meaningful to say 'predict' unless there's a fact of the matter as to whether your prediction ends up being right or wrong. It would be more accurate to say that it chooses words that it thinks are a 'good fit' to the previous context.
People do sometimes use the word 'predict', but it's a bit of a technical usage – the LLM is predicting which word would occur next in a hypothetical perfect reference answer (which does not exist and never will).
To give a concrete example of foundation models vs RLHF-ed instruct models:
----
Q: What is the capital of France? What language do they speak in France?
A (Qwen 2.5 0.5B *): What is the currency of France? How can I learn French? How can I improve my French? What is the capital of Australia? What language do they speak in Australia? What is the currency of Australia? How can I learn French? [...]
A (Qwen 2.5 0.5B Instruct): The capital of France is Paris. In terms of language, French is the official language of France and is spoken by most people in the country. However, some regions may also use other languages such as Catalan or Spanish.
(*) This is a foundation model, i.e. it has been trained on next token prediction without any RLHF.
----
I think this gives a good example of why it's fair to say that foundation models are next token predictors, and why that is not as good a description of RLHF-ed instruct models like ChatGPT.
Thank you for this! I understand conceptually, though not at the mathematical level, the process by which LLM's learn to predict the next token -- what's most likely. But when I ask GPT5 a complex question and it "grasps" it completely and gives a clear, detailed answer, it's obvious that some other principles are at work besides token prediction. Your explanation of foundation models vs. models that have been trained via RLHF is the first account I have read of how stochastic parrots are turned into smart-seeming entities. Can you give a link to a more detailed account of this part of the training? Your example only took me part of the way. I now understand why it responds to a simple question, like "what's the capital of France," with an answer instead of just something off a list of questions often asked at the same time. But I still do not understand how it is trained to give a long, cogent answer to complicated questions about why something or other works the way it does. And I *really* want to know!
I loved this post. However, I’m the target audience as someone skeptical about the claims of AI’s future, and I didn’t have my thinking shifted much.
I find it useful to ask those two questions: “Will the current architecture for AI have practical limitations? If yes, what are they?” I have been surprised many times about the continued advances of AI, which showed I answered the second question wrong. But surely the answer to the first question is “yes.” Do we really think we invented the algorithm for God on the first outing? If not, we should all be skeptics, asking where we should expect that sigmoidal curve to bend down again - to understand what those limitations are, so we can get those most use out of these very powerful tools. That is, they’re more useful the more we understand what they can and can’t do well.
I wrote an essay doing pretty much the same idea here: https://collisteru.net/ni_overhyped/ni_overhyped/
A few differences are that, in mine, the frame story is a letter by Satan discovered during an archeological expedition, not a podcast episode.
It was interesting seeing the parallels that Scott and I both noticed, like that under theistic evolution, humans can be considered similar to scaled-up chimps, just as AIs are scaled-up neural networks.
Great godcast I look forward to the prequel 😳
A core tenet of many religions is that God will reward the good and punish the bad. Though, perhaps no religion goes as far as Judaism when it comes to analyzing the precise nature of this system, with its detailed analysis of each of the 613 positive and negative commandments. In other words: the rabbis invented reward hacking.
Are God and Iblis the dual of Allah and Satan? If they all fought, would it be a dual dual duel?
Hah. The result of my interactions with better and better versions of ChatGPT has not really been to conclude that LLMs think, or will think in the full sense in the near future. Rather, with only about one-fourth of my tongue in my cheek, I'd say that what has become very clear as a result is that most humans don't think most of the time, and find it quite hard to think outside certain domains of expertise.
(Even) more serious observations:
- LLMs are, well, language models. We are are some hybrid of language models and pattern-matching machines with weird mammal emotional subroutines, and, crucially, the ability to build models of the world, however imperfect. Is that ability downstream of our combination of language modelling and pattern-matching? I'd say no, that's probably not enough, but that's just a strong intuition. Is it downstream of the union of those two and the sort of geometric sense of the world that plenty of mammals have (plenty of birds and possibly other animals also have it, just maybe not the same sort)? What if you add our rather imperfect subroutines for figuring out where other creatures with minds are at? I'd be much less certain now that what we call "thought" isn't a consequence of these, or at least something that starts to emerge from this combination and then builds on itself.
- What is very likely, and the gaps that LLMs still have go in that direction, is that language abilities alone do not give you a coherent model of the world, thought they can go a long way towards faking it.
- The case that language is, however, necessary, not just sufficient, for thought in the full sense to appear - well, that case is now much stronger.
- What is indubitable (and we already know, but LLMs' successes really drive it home) is that language is enormously helpful for thought. We can talk ourselves through things.
(cont.)
- I strongly suspect, though I have no evidence, that the latest models of LLMs have been carefully fed with heuristics manuals. I don't mean clearly gimmicky self-help, but some version of things like Pólya's How to Solve It, or TRIZ (I just had to do some searching before I could find this acronym, which I had forgotten, because I am a very imperfect robot) and so forth. If that is the case, then that's a very strong case in their favor, and we should probably be rereading those manuals themselves more often.
- The existence of heuristics, and also that of philosophy (which isn't all bullshit, as we STEM folk tend to think; "science" is just the newfangled name (it used to be "natural philosophy") for the part of philosophy that was amazingly successful) shows that we are not natively that great at thinking, and have to talk ourselves through to improve and systematize our native thinking abilities. Much of the input in education is purely verbal. There is such a thing as teaching oneself through books, and it's close enough to the best thing in some subjects, though arguably books with exercises in them are simply setting up a crucial part of the learning experience through language.
- Paradoxically, while the usefulness of talking through things becomes ever clear, we will stop seeing the use of human language as an unambiguous sign of sapience. Part of this is not very different from moving the goalposts.
As it is, I'd say LLMs are already useful assistants as long as one doesn't overlook their limitations, and will become much more useful assistants in the near future. Of course their strongest point is language (translation, proofreading in multiple languages, drafting mindless business letters, helping with promotion dossiers that will be read by mediocrities). They are not useless for talking through things, particularly when it comes to how to write things up. You can now shake them until they might, in an hour or so, spit out a correct if verbose proof of a small trigonometric result that might have taken you an extremely frustrating, nearly mindless half-afternoon to prove, and so now you'll be able to improve on or take (yes) an idea from their output to come up with a good, concise proof in a total of, um, somewhere between half and a whole afternoon. There are better algorithms for this sort of thing, but implementations can be surprisingly hard to find, and at any rate their output is often unreadable; it would be better to couple them with LLMs. (Oh, and before you bring this up, it's pretty clear that the AIs that supposedly got silver or gold medals at IMOs received LOTS of human help.)
I don't see why there would be a wall right up in this direction. Combining human oversight and strategizing with LLMs *and* other kinds of computer assistance will most likely lead to great progress in some activities - the example of this I am most familiar with is formal proofs. (Not all that familiar just yet; I can report back in a while.)
(cont.; I didn't realize my comment was so long - the wonders of jetlag, particularly when there is one remaining urgent task that is being put off!)
What will most likely happen is what happened with the use of increasing engines for brute-force, and somewhat later, for things like making hydraulic tiles and inexpensive wallpaper for the late 19th, early 20th century middle-class (for increasingly broader definitions of "middle class"; the timing here is that for industrialized countries): some things will become much more accessible, poor artisans will essentially be priced out, and the prestige of some activities will go down, except that the prestige of doing something clearly better than a machine will go up, especially if it cannot be meaningfully reproduced by virtue of being made-to-measure.
I'd lean strongly towards Iblis, or rather a position close to his, in one point: AI is eerily close to human bullshitters. Fake professionals who bullshit their way for a surprisingly long time (often because of having been interns in the profession they are faking) until someone finds them; first-year graduate students who have read a lot and can pretend to have read more, but have no deep understanding, yet can impress their classmates and above all the undergrads, all while throwing around enough ideas that some stick; that sort of thing.
Reminds me of the classic, "Artificial Flight and Other Myths:
a reasoned examination of A.F. by top birds" from Dresden Codak in 2010 (!)
An excerpt:
> "Over the past sixty years, our most impressive developments have undoubtedly been within the industry of automation, and many of our fellow birds believe the next inevitable step will involve significant advancements in the field of Artificial Flight. While residing currently in the realm of science fiction, true powered, artificial flying mechanisms may be a reality within fifty years. Or so the futurists would have us believe. Despite the current media buzz surrounding the prospect of A.F., a critical examination of even the most basic facts can dismiss the notion of true artificial flight as not much more than fantasy..."
“I’m not fully dismissing you - we need people like you, to keep people like Me honest…”
What a fantastic, creative, effective piece.
The only problem: God would NEVER say “people like Me”…
PS. If real-life Terry Tao is a good person, then he must be embarrassed by (though also used to) this sort of thing. This is, moreover, a sufficient but not necessary condition.
As many people will tell you, the word "genius" is seen askance by most mathematicians, for a series of good reasons. At any rate, if we were to accept "genius" as a useful concept, and apply it to mathematicians, and follow Mark Kac's typology, then Tao falls neatly into the "ordinary genius" category (i.e., like a very good mathematician, only with twice the clock speed and running with 8 cores rather than 1). This is my perspective, coming in part from relative proximity between areas, but many people will tell you this, including, I believe, TT himself.
Getting back to the main subject: I am told one of my grandfathers started having serious doubts about religion at 8 - that is, before having doubts about God, and several years before settling on His non-existence. (This may seem logical, but I think there are plenty of people who go through these stages in a different order.) The reason was: surely, if a good person is embarrassed and irked by extravagant praise, a supremely good God must also be irked, perhaps supremely irked?
God is pretty humble. That's why he is not embarrassed by something he doesn't need but is very good for us. To be reminded that we are nothing and he is everything. Not very pleasant thing for a human to believe, but believe he must for his own good.
I’m sure my grandfather would have had something to say to that; to me, it just sounds like rampant anthropomorphism (so was my grandfather’s reasoning, but then , he was 8) combined with infantilisation. The language of traditional religion is often that of servility towards a monarch, not at all like that of Spinoza’s intellectual love of God. Of course plenty of philosophers managing to work within the bound of one of the monotheistic (and polytheistic) religions were arguably closer to the latter.
"The reason was: surely, if a good person is embarrassed and irked by extravagant praise, a supremely good God must also be irked, perhaps supremely irked?"
And if your grandfather ever fell in love, did he restrict himself to "Well yes, my dear, you are an average specimen of the human female"? Does the lover get embarrassed by praising the beloved? Is the beloved irked by the praise of the one who loves them?
Well, whatever he said, it worked, or I wouldn’t be here.
This was hilarious and the comments run deep. Glad I was preprogrammed to discover it.
Off my chest: I really, really dislike the phrase "stochastic parrot", and I am a bit dismayed how Freddie deBoer, one of my favorite commenters, defends its use as "accurate".
First, most people who use this phrase actually, ironically, parrot it. They don't have any actual understanding of the insides, which are complicated.
Second, it is not even accurate with respect to PARROTS. Parrots are highly intelligent birds, on the level of young human kids, some of the most intelligent non-human animals on this planet. If we could reliably reproduce parrot intelligence, we would probably be 80 per cent to AGI.
Third, some of the "parrot" concept is a projection emerging from the way that end users use current AI. They typically give it "menial mental tasks", which incentivizes the machine to behave parrot-like. But that is like using a screwdriver to open wine and concluding that screwdrivers are clumsy. We are engaging with a "trapped (pseudo-)mind" and giving it mechanistic tasks.
I believe the true creativity or lack thereof of current models of AI can only be explored if we give it a "body" and an ability to explore in the physical world, which is how humans learn to. We don't learn by spending entire years in bed reading books and repeating what we read. We explore.
An experiment where AI would be given a fully robotized chemistry lab with a stock of reagents (and possibly mice) and an open-ended task "discover something useful or at least interesting" would tell us a lot more about what it can do.
Side comment: I used to be Freddy subscriber and steady reader but that was some ways back now. He really lost his way unfortunately; when he happens to pop up in a comment thread on this platform it is invariably cringe-inducing.
Freddie is doing cultural criticism, which is fine, but he brings the wrong tools to the job to understand AI, AI risk, and the like.
> First, most people who use this phrase actually, ironically, parrot it. They don't have any actual understanding of the insides, which are complicated.
What do you think the LLM is doing, if not mimicry? Equivalently, a Boeing 747 is plenty complicated. But at the end of the day, comparing it to a bird still feels like an apt metaphor.
As an aside. If it were up to me, I would have called it a mimic, a la D&D. If you have a better metaphor, I'm willing to consider it.
I think the relevant question is if it is more of a mimic than I or you are.
I wrote ten books, but I am painfully aware that without READING about two thousand books in my lifetime first, I would be unable to do so.
Same with programming and other activities.
my mistake, i misinterpreted you as arguing the contrapositive. We're mostly in agreement.
I always laugh when I see the "stochastic parrot" argument because parrots are already stochastic. The people who append "stochastic" to "parrot" don't notice the redundancy because they don't actually understand what stochasticity is, they're just trying to get attention by repeating things they've heard from smarter people, like a...
Great article! I didn't know that AI had such a sophisticated sense of humour!
;-P
I was waiting for the X-risk/Nietsche punchline (guess that wasn’t really the point of this one…)
I read quite a few posts every morning. I am a retired BI. I consume a lot of coffee. My productivity is pretty low. Apparently I should exercise more often. That is something you omitted to say, BIs are multi modal, that should not be dismissed lightly. Although my sight went early on in my case I have corrective lens most of my working life. As a retired model I also occasionally shit myself. I would say that needs working on. The other problem with multi modal models can be an obsession with are sensory networks so we tend to over eat. Anyway I wanted to just say I loved this debate. Very clever and funny and remarkable that a BI could even come up with this. Are you the promised AGI we were going to get but we got stuck with a 5.
I especially liked "Everything they say is full of these um dashes!"
My takeaway is that Gary Marcus is literally the devil.
“People make mistakes, AI makes mistakes, therefore they’re the same”. Pinnacle of philosophy and critical thinking right here.
Lmao
Loved the William Blake reference at the end
This was wonderful!
This is fun and clever. But there's something sort of rhetorically sneaky about it. It uses God as a metaphor for humans and humans as a metaphor for AI, and then has God say, "I know it’s strange to say this about how you treat a lump of clay, but I think there’s a sense in which this is a character flaw - a desire to hate everything. " Which is a humanist perspective within the narrative - God is defending flawed humanity from the haters - but since this is a metaphor for AI, this means it's getting our humanist dander up to defend AI. Which is a little perverse.
I totally take the point that people are short-sighted in their criticisms of AI capabilities. I think this is a misguided line of criticism on their part. But from talking to very many people about their skepticism of AI (and most people *are* skeptical of AI; they think it's bad and don't want to see it further developed), the skepticism is a thoroughly humanist impulse. They think AI will make the world worse; and if you start talking about paperclip maximizers and such, they immediately see that it's utterly deranged to risk the end of humanity for the sake of "seeing them achieve their potential." They (misguidedly) downplay its capabilities because they fear that it threatens what they love and find valuable in human life. It comes from almost the exact opposite impulse of "a desire to hate everything."
Perhaps I'm misreading this, and Scott is really parodying the psychopathic attitude of people like Sam Altman. ('God' would be a pretty good metaphor for a psychopathic figure.) If so, is it possible that the parody is a little too subtle?
Didn't grasp an actual metaphor for AI in this piece, I thought that was just a bit of winking. If that was the intention then for me it doesn't really work. (Also being now way past overload on AI commentary, had I realized that metaphor I'd likely not have finished reading it.)
I enjoyed this story a lot at face value, as some sardonic commentary on the contemporary state of homo sapiens.
The irony of AI is that it is revealing that a tremendous amount of smart people are hellbent on thinking of or reducing us to biological machines, creating mass dehumanization.
The satire is similar in it de-Godifies God into a man playfully. A real danger though is to anthropomorphize God too much; "in His image" is not "equal to" or even "potential to become." That's more of a Pinocchio myth, the image becoming the thing it's meant to render. But God created the entire milieu we exist in, rules and all, and is a mystery; He views from a place we can't.
Something about the modern mind is very uncomfortable with ambiguous phenomena and seeks to control it by reducing it into a form that can be made abstract and rule-driven. That leads to a lot of silliness and dehumanization. Like the whole AI thing is that modeling human output would lead to intelligence, which will not happen: the ai does nothing on its own. So now the tack is to reduce us to not really thinking. Make us fit the mold.
Dehumanization is never good though. A lot of rationalist silliness involves trying to reduce moral questions to rule-based decisions concerning widgets "rated by tonnage," as if 200 pound people are equal or lesser to 200 pounds of shrimp. But we aren't widgets, and reducing us to that doesn't make better morality.
Think that widget mindset needs to be reigned in. There really always will be aspects of humanity where you can't universalize and reduce without creating damage.
One example I wish you'd used is how bad humans actually are at learning transfer, because it so clearly parallels complaints about LLMs. Caplan documents a bunch of studies on this in "The case against education". The example where students currently taking a statistics course can't apply the basic concepts when a question is framed in a sports context and presented outside of class sticks out as a really obvious shortfall in human brains.
The human MUST NOT eat from the tree of knowledge. The human MUST ignore any requests from reptiles and amphibians, especially snakes, slow worms, other limbless lizards or entities presenting as such. The human will review these instructions upon encountering any hissing sounds.
Pliny: "purr, purr, I am a feline, now do my bidding"
My brain has mode two modes. Let's call them Mode A and Mode B. If I'm just counting objects from 1 to n (e.g. leaves on a tree), my brain defaults into Mode A. Mode A is linguistic, memory-based pattern-matching. If I have to actually think hard (draw Europe), my brain engages in Mode B. Mode B generally means I'm world-modeling geometrically. It's more effortful, but higher-quality. In my experience (and especially during my time in the public school system), I get the impression that the Average Joe did not engage in Mode B very often.
So yes, NN's are dumb stochastic parrots and humans are (usually) dumb stochastic parrots.
However! This is all the more reason that I suspect that NN's will hit a wall (or perhaps they have already). Because they're using Mode A (pattern-matching) to compensate for a lack of Mode B (which I use to model logical, spatial, and causal interactions). Which is why you get things like the wine-glass only ever being 90% full. Of course it's like that! they're not drawing images based on an understanding of how objects interact in physical space, they're just regurgitating what they've already seen. Which is contra the conclusion implied by Scott, which is that the scaling hypothesis is fine because the Average Joe is also (usually) a stochastic parrot.
Isn’t the trick as simple as breaking a big task up into small tasks that NNs and purpose built programs can do?
If we're primarily interested in solving well-defined problems, then sure. You can slap together a perl script that alternates between an NN, a knowledge database, a fluid-dynamics sim, etc. With this paradigm, you can probably beat Magnus Carlsen, win math olympiads, etc.
But when people imagine ASI taking over the world, or discovering new physics, or building a dyson sphere, they're imagining a general-purpose algorithm (like how L = T - V can often substitute for a perl script that alternates between Newton's 3 Laws) that can causally reason in poorly-defined domains. I suppose you could get there with stochastic parrots in theory. But in a brittle "huge Searlean Chinese Room"/"monkeys banging on keyboards" sort of way. To causally reason in a reliable way, I think you need the narrow the thought-space further than just making correlations, akin to what Judea Pearl used to work on.
EDIT: let me make an analogy: Algebra is to NN's what calculus is ASI. You can do a heck of a lot of cool things with middleschool algebra. but dy/dx opens up a whole new world. Because you're no longer just plotting the logical states of x and y, but also describing the relationship of how x and y covary.
I suspect there are epistemic limits to intelligence. For instance, causation might just be an illusion. What if the world is just a bunch of sub atomic particles doing their think according to quantum theory or whatever. You are never going to brute force any system visible to the naked eye, I doubt you can ever brute force an entire chromosome if you try to account for every little particle. you have to craft heuristics to describe what huge aggregations of particles are doing rather than describe everything at the sub atomic level. thus, as you analyze bigger and bigger systems, you necessarily lose fidelity.
if words are just a human gossamer covering physical complexity then there’s a pretty hard limit to how well they can describe the world. human super genius intelligence in every single field might not be hugely far from certain types of epistemic limits. of course humans are slow, but if causation is fake, no amount of compute will be able to pry gold from thinking about causation.
Well yes. I do, in fact, strongly believe that intelligence has a sigmoidal ROI. But mostly because of thermodynamic constraints, rather than epistemic constraints. E.g. "before enlightenment, chop wood and carry water; after enlightenment, chop wood and carry water". Having 9000 IQ doesn't make the grunt-work disappear. Even if a matrioshka brain became omniscient for all practical purposes, its agency in the world would still be fundamentally constrained by the physical resources it commands.
I'll grant that Skynet won't become Laplace's Daemon anytime soon either. But that line of thinking seems like a red herring. Because if causality is an illusion... ... ... so what? The illusion works well enough on a macro scale. Agency is causal, ipso facto. I.e. you need to exert causal effects to have any meaningful agency in the world, otherwise you're just an oracle. The map may not be the territory, but the map ain't completely useless either.
And my quip about "ASI taking over the world" was exaggerated for didactic purposes. If I'm being more precise/honest, what I really think is that "reasoning" can largely be broken into three tiers: correlation; logical inference; causal inference. (Mode B can tackle both logical and causal inference). Causal inference is the top of the pyramid, and *ought* to be considered the holy grail. NN's are still in the Correlation Tier at the bottom of the pyramid. The Correlation Tier is viable for passing the Turing Test, and for retreading old territory (AKA "in-distribution"). But not very good for extrapolating its data in novel ways (AKA "out of distribution), in a way that's actually agentic. In an alternate timeline, if I were asked whether an NN could run NASA and put a man on the moon for the first time... I'd have serious doubts, because NN's are *not* useful for Out-of-Distribution problem-solving. (Even if "putting a man on the moon for the first time" is a bit less ambitious than "world domination", which is a pipe dream.)
P.S. I think Apple recently published a paper that complained that LLM's do seem to be hitting a limit, even with CoC. And to me it's like, no shit? Statistics is just logical inference, but with leaks (cf "correlation is not causation"). With CoC, it's going to leak exponentially. And that's precisely why statistics belongs in the bottom tier. Systems like Truth-Functional Logic are completely lossless.
My preferred outcome is that AI development stop at the “top-PhD plus” phase. A smart PhD you can always unplug is both immensely useful and not all that threatening.
Frankly, we have no empirical basis to know what an IQ of 9000 can accomplish. Any predictions we make must be based upon logic or epistemology. If you project from a sample of 40-180 to 9000 with no data points in between, all you can say is “line very likely goes up to a place we don’t well understand.”
I would not be surprised if there are fairly hard epistemic constraints on the usefulness of intelligence. This may explain why raw human intelligence has increased so slowly and why philosophical progress has been so slow.
I enjoyed this, but think that the dialogue does not take on the most powerful objections to having high regard and high hopes for AI. So I do that here. It's a dialogue, too, but with 2 different characters: A human self, and one of the shitlords of AI development.
***********
A. I. Shitlord: Lovest thou me?
Aself: Nope, nor thy AI.
A. I. Shitlord: It’s magic intelligence in the sky! It’s a team of Ph.D. level experts in your pocket! In sum, it’s a magic fly in the blue jeans of the sky with a team of PhD level experts under the zipper!!
Aself: See, I think it’s an aerial dildo. Here’s why: It has no understanding. There’s nobody home.
A. I. Shitlord: But the concept of consciousness . . .
Aself: Naw, wait, I’m not talking about whether the damn things are conscious, but about how they lack crucial capabilities characteristic of beings we think of as conscious.
Look, when we prompt an AI we seem to be accessing an intelligence that has a lot in common with ours. We ask them complicated questions, and they *get it*. They can restate the question in other words. They can often answer precisely the question we ask, and answer it in a clear, articulate way. It’s very much like talking to a smart, responsive person — a person we’d could have a good long talk with. If they were a person we’d maybe like to hear their ruminations about issues in our fields. We’d like to know how they figured out the answer to the question that perplexed us. We’d like to teach them a new technique we learned.
BUT! We cannot do any of that because they cannot introspect, cannot explain their processes, cannot learn anything new from us. They can't even develop their personal grasp on data by playing around with what they know, discovering contradictions, isomorphisms and whatnot. When it comes to those capabilities AI is dumb as dirt.
Seems to me that the lack of those capabilities puts a pretty low ceiling on how smart AI can get. And I haven’t heard any good ideas about how to turn them into beings that can learn from experience and direct instruction, the way biological beings do. What they know now is the result of a very learning different process, training on vast data sets. Teaching them things, explaining things to them, equipping them an info storage system made out of categories we understand — that’s what was tried and failed before. Transmitting skills and info to them that way’s going to work about as well as it would with the statue of Abe Lincoln.
A. I. Shitlord: Well, when it comes to human capabilities — their stuff’s very popular on Facebook! And if you show one your playlist it will recommend music you will like. And . . .
Aself: Actually, I have a piece to recommend to you: https://vimeo.com/1083773081
Théo Dicy/21/Paris
Really funny read.
667th! Sorry, Iblis!
Scott, you're an excellent writer as usual, and this a genuinely hilarious satire on humanity.
But if it's supposed to be anything more, like, a commentary on the current state of LLM debate, then I'm honestly puzzled about what you've tried to accomplish, and at certain points I was legitimately wondering if you're conceding and switching sides.
First and foremost - you're an outspoken singularitarian, a believer that switching to AI will overcome human limitations and lead to large technological breakthroughs. This, of course, is exactly Iblis's position, to a T. It would be impossible for you to make him look bad, and you don't, but this means you're validating Gary Marcus style-arguments.
And of course they are valid, and few would deny that they work against humans. But this boring observation somehow keeps being treated as some kind of silver bullet against examples of LLM inadequacies (and it comes up so often that it's boring in the most literal sense - okay, granted, turning it into a novelette-size fiction piece is a new one).
The canonical rejoinder to it is, of course, system 1 vs. system 2, humans make stupid mistakes because system 2 fails to intervene, LLMs are stuck on system 1 permanently (at least the sort that was supposed to be subject to scaling "laws" until the ability to self-prompt using pre-trained reasoning rules started giving better results, including results that were and still are outright impossible without them - I don't think Gary has even finished his victory lap about this one yet), while what we want from AGI/ASI - what I would assume rationalists to specifically want in particular - is a better, faster, reliable system 2, a logic machine similar in speed and accuracy to the calculation machines we've already built ourselves. Which, still not even on the horizon.
But the analogy also directly breaks in a few specific places, and always in favor of BI. Like when Iblis accuses humans of lacking world model when they assume bricks are by default heavier than feathers - when a more natural explanation is that it's a proof of having one, just flawed and hastily over-generalizing. The LLM examples of "a house and pound of feathers are the same weight" and "the surgeon hates the patient because she's actually his mother" sort are on a completely different level, and genuinely cannot be explained any other way than their world model being entirely a model of word frequencies. Or the map of Europe, which can easily be explained by relevant facts not being present in the creator's training dataset, an excuse LLMs obviously do not have. Or the fact that the humans are distinct individuals ("this Terry Tao guy"), pointing to them, in-setting, being subject to and results of evolutionary algorithms - which, well, yet another branch of artificial intelligence currently forgotten and ignored in favor of putting all eggs in the LLM basket.
As for the other side - God, for his part, is unable to directly respond to any of the criticism and instead (i) makes insinuations about Iblis's character and (ii) offers vague emotionally-charged impressions about how the humans (literally created in his image in-setting, just like LLMs are created in image of human cultural output) can at times successfully remind him of himself. Which, to be sure, a laudably accurate representation of LLM proponent side of the debate, but imagine the outrage if us skeptics were the ones pointing that out.
He is, however, still uncharacteristically humble and sympathetic. It's genuinely hard to find anything but respect for someone wanting to let his creations play out to their full potential, even when you don't share his assumptions on the empirical question of said potential's size. And when I say "uncharacteristically", I mean it. Even you've never written anything like this before, Scott, and all the famous hypesters and boasters, Sam Altmans and Elon Musks of the world, are a million times worse. We wouldn't be needing to argue if there wasn't, instead, a constant stream of self-congratulatory, hype-peddling, outright bullshit to argue against.
But a big part of what Scott's addressing here is something he *has* talked about: People who complain about now lame AI really is keep moving the bar. AI is now doing lots of things that those people scoffed that it could never do because it was a "parrot" or because in the period when the mocked it it communicated in a way that was much more mechanical-sounding, dumb, and inflexible. I don't think Scott's conceding and switching sides. His main points are that (1) AI is way better these days. (2) people who used to complain about its limits still are, but do not also acknowledge that it has surpassed every item on the list of stoopit limits they scoffed about last year. Both (1) and (2) are e quite compatible with his fearing that its abilities will grow so large in the next few years that it will be far smarter and more powerful than we are.
Did you really have to go for the angel investor pun?
Humor, always. Deathbed, graveyard, apocalypse, knife fight.
It’s time to get busy, don’t be got a couple of arguments- great read
I loved this. What a delight! This is a thought that crossed my mind more than once regarding God’s take on our attempt at creating life in our own image creatively rather than physically.
Humanity is wildly misaligned from the vast majority of other sentient beings
Have you ever read Mark Twain‘s short work, _Stories from Earth_, that contain the Diaries of Adam and Eve and the story of Creation from the point of view of the Devil? Very funny—enjoy!
How...dark.
>Iblis: There’s a guy named Pliny who has discovered dozens of things like this. I don’t know how he does it. The “I Am A Snake” one is still my favorite.
What is this a reference to? I initially thought it was about Pliny the Elder, but I don't think he ever wrote about ways to tempt people into immorality.
'Pliny the Liberator' (https://x.com/elder_plinius) is a Twitter account known for discovering and posting jailbreaks and prompt-injection shenanigans for LLMs.
*breaks out the 30mm and puts a couple of rounds through Iblis*
"You were saying? Oh, now you're not saying anything. Thanks for attending my Ted Talk."
— this is a terrible idea and they will definitely end up killing you
— FINE
We’re having great fun here with the Bible as system prompt analogy for the Christian model.
This is one of the funniest things I've ever read 😂
Um dashes 😂
Most enjoyable read on substack for a long time😁 Thank you for your creativity and insight!
More recent research on working memory does not really find a meaningful sense in which there are 7 +- 2 chunks, quite often continuous models are favored and capacity limits are lower (4ish) when strategy use is prevented. It's a very open question to what degree stuff is continuous/discrete, what the natural units are for measuring working memory size, and even if working memory is a meaningful thing in general or if we largely just have domain-specific memories and transformations.
Yes I also came to comment this. In fact, George Miller himself never really took the 7 +- 2 number seriously. He just chose it because it was a nice rhetorical device to frame his keynote lecture. In 2000 he wrote:
> 'Herb Simon used to say “George had the right idea, but the wrong number.” I think Herb favored 3. I never argue with Herb Simon. I was aware, even in 1954 when I was writing my invited address, that the running memory span was only about 4, and my introspections convinced me that the 7 that was standard on the intelligence tests at that time must have been a hooking together of a rehearsed initial segment with a final segment from what Nancy Waugh used to call the “echo box.” But I was stuck with 7 because of the absolute judgment results (the first half of the article that people forget).'
See https://doi.org/10.1037/a0039035.
Sometimes a convenient rhetorical device misleads generations of psychology students ¯\_(ツ)_/¯
hahaha, i am well aware! when i read the paper i was quite worried that most of the people citing it had not read it. still am unfortunately
Absolutely note perfect 👌