Both Hindu theologians (See 'Nondual Tantric Saivisim' by Wallis) and Thomas Aquinas predicted that inference-based systems won't be able to 'know' because 1 isn't a probability you get to through incremental updates. Both of these schools of thought claim there's something else that _can_ operate behind the mind, but without that thing in operation, you're going to end up acting more like the LLM than whatever it is that Iblis is pointing at.
I'm at one. It's dope. It allows me to task risks others can't imagine :)
I bet you have it too. How much would someone have to pay you in order to torture a kid? If the answer is 'I would not do that for any amount of money', congratulations, you've gotten to 1.
Anything is arguably evil if you don’t believe that good means something real. And yes, I have no problem saying I refuse to torture a child, regardless of whatever else might come of it. I don’t care if anyone thinks that makes me evil. I call that person a fool.
There’s a world of difference between unintentionally hurting someone is a byproduct of productive activity, and intentionally inflicting harm on them. Torture, in particular, means that your goal is to inflict pain. It’s not an accidental byproduct. It might be the instrument of something else, but still the goal is inflicting pain. I agree that being unwilling to inflict pain at all is unworkable.
I can say with perfect certainty that I would never torture a kid (or anyone for that matter), assuming no more or less obvious insanely huge opportunity costs or ulterior motives, assuming no external pressure or blackmail to torture, assuming that the sole reason for and benefit of the torture would be the expected (lack of) pleasure of seeing the person suffer, and assuming that the "I" in this thought experiment roughly corresponds to how I experience myself right now.
If I forgot some obvious caveats, just add them on top, rinse and repeat. I would never, 100% guaranteed and certain.
I'm also aaalmost 100% certain this was OP:s point. Not some specific weird example that can be worked around, but That Obvious Example Which You Yourself Can Be Certain Of, even if you found the original example lacking.
The probability that you forgot some obvious caveats is greater than 0%, as you allude to. It could be 0.000001%. But unless you also guarantee 100% that you have included all caveats you would want to, you cannot claim the final answer is a perfect 100% rather than 99.9999999999 or so.
The sum of caveats adds up to 1 (0.999... doesn't just approximates 1 but is 1 - even mathematicians agree). Add them up, rinse and repeat. I won't do the rest of the grunt work, because I'm doing the opposite of trying to gotcha myself.
I'm a layperson, so I'm just willing to skip the math. The induction is easy enough all the same. It is allowed to be reasonable.
Also, check the second paragraph. Arguing using caveats misses the point the commenter was trying to make. I'm guessing that arguing against the strongest version of the claim doesn't consist of picking caveats, as they can then just add the caveat to the claim and we're going in circles. Therefore "rinse and repeat".
I won't torture a child, exactly in the way I'm imagining I won't do it. Trust me - I just know.
I think this is also a function of the fact that almost nothing in real life is an actual binary trolley problem. If someone proposes to pay you an immense sum of money for torturing a kid the correct agreed moral answer is "fuck them up".
If I end up in this hypothetical situation, I have no reason to believe that me torturing the one kid is gonna prevent blackmail from torturing the other 5000. So there’s no reason to do the torture.
I don't think you get how much infinity is, though?
Take everything you can think of that would build trust in this. If there is any POSSIBLE thing that would make you believe the blackmailer would keep the bargain, add it to the hypothetical. You can't run out of budget on this hypothetical, because the budget is infinite.
If you need some ideas: The blackmailer will sit in a jail cell, with an electronic lock, set up to detect the screams and determine whether they're fake. If the screams happen, the cell will lock, never to open again, otherwise, after a certain time limit, the blackmailer will leave to go do the tortures. That's what I got spending 5 minutes thinking. So, extrapolate what I'd come up with if I spent infinite minutes thinking, and use that.
Hell, make it worse. The kid is actually a psychopath boy genius, who, while still young, used his talents to deploy a nerve gas bomb in Grand Central Station, while your wife and kids were there. He plans to do it again, several times, but does say he expects being tortured to convince him otherwise for some reason. This is in addition to the existing saving 5000 kids. Again, that was 5 minutes of thinking, so extrapolate to infinite minutes of thinking.
With all that added, we have gotten about 0% of the way to infinity. This is the problem. Probability 1 is infinity:1. You are not, and should not be, that sure of anything.
No budget is ever infinite. This is, I think, the crux of our differences. I don’t have infinite attentional bandwidth and thus choose to treat some probabilities as zero. Others I treat as one. This makes me a far more effective agent in the world than I would be if I invested in reasoning about so many hypothetical outcomes.
I just asked ChatGPT how much money it would take to have it torture a child, and it responded that it would never torture a child, under any circumstances. So ChatGPT also got to 1.
It SOUNDS like a moral answer, but is it really impossible to do good by torturing a child?
Suppose a child has a disease which surgery can fix, but also a fatal allergy to the only accessible anesthetic. Torturing the child, by performing the life-saving operation, sounds like the moral choice.
Well, my public "no" answer is kinda cheap, right? Maybe you get other answers when you come with 666 million and Putin's youngest offspring - or this kid kicking my son in kindergarten ... . Also: lots of kids get tortured every day by other humans last time I checked. If it is war in Sudan or Aella's dad or at school - by bullies or teachers. I claim: At least 90% of state-school teachers are morally worse than heroin-dealers. Because they would not quit if a kid said: 'I do not want to attend the lessons' while the principal insists they keep it in class.
I don't put 100% probability on mathematical things either both because I sometimes do maths wrong, and because I regard maths as true inasmuch as it predicts reality and I can't entirely rule out the possibility of that not working at some point.
I put a very large number of 9s after the decimal point on that question, but not actually infinite. I came to be convinced of it by observing evidence, and if I were to observe enough of the same kind of evidence in the other direction, that could in principle change my mind. I could wake up tomorrow in a hospital bed, with doctors telling me I have a rare type of brain tumor that leads to strange delusions and false memories, which they've just removed, and the formula is actually just pi * r. And if then from that point on every book I look in has that new formula and every time I try to calculate anything or test it in the real world, the old formula always gives me a number twice as big as what I observe, and so on, I would gradually start to change my mind.
This won't happen, and under my current models can't happen. But if it did happen, I would want to be able to change my mind, which you can't do from a "probability" of 1.
You're confusing reality with your beliefs about reality. Mathematical reality, just like physical reality, is what it is regardless of whether or not humans are able to know it. No one here is disputing that.
The question is about how certain humans are able to be. I'm no more certain that the circumference of every Euclidean circle is 2pi r than I am certain that there is a keyboard here in front of me that I'm typing on. That's quite certain, but it is short of absolute certainty, even though both realities are what they are regardless of what I think about them.
Except that euclidian geometry maps imperfectly to real space. Its an internally consistent model, yes. But it's predictive value as a model is never perfectly certain. Models are only as good as their premises and inputs.
Please, you should try math some day! Mathematical knowledge is never *absolutely* certain either.
The facts are what they are, but knowledge is not the facts - knowledge is about what is going on in a human's head. If you actually do some math, you'll realize that there are errors all over published papers, and people who thought they were certain of things are often wrong.
The proof that for every integer there is a larger integer than it is about as certain as the proof that there is a keyboard in front of me that I am typing on. The big difference between the two is that you have just as good evidence of the former as I do, while your only evidence for the latter is if you trust me (though there are equally certain empirical things for you).
If you think that math somehow breaks out of the cycle where someone has to start with an assumption they are quite confident of, you should think a bit more carefully about what a mathematical argument is.
It's difficult for me to put probabilities on very abstract things like this, but I think there are circumstances in which I might not believe it, so it can't have probability 1.
If you were to truly believe something with a probability of 0 or 1, nothing in the world would be able to influence that belief, and you'd be stuck holding it forever no matter what conflicting evidence comes in. I don't think humans have this trait, nor would it be desirable if we did.
"Is 437 a prime number?" is a very simple math problem, and one where the answer is easy to find if you know anything about primality testing - and yet when I was first given this problem, I did some calculations, made a small error, and confidently gave the incorrect answer. As long as your brain makes logic mistakes, there's still the possibility that you might have gotten something wrong, even in math.
That seems like a strange example to choose in this context. What about "is 2 a prime number?". I don't think it's possible for me to be more confident of anything than I am that 2 is prime.
and what is the odds that you are suffering some kind of stroke (or environmental toxin, or disease, pick your thing) that is influencing your thinking such that you think you know the correct answer and are following the math but you aren't? Is it literally zero? Because if it's not literally zero, then your confidence that 2 is prime should not literally be 1.
I chose that example because it's one where I personally got the answer wrong when asked it for the first time. It doesn't have memories attached for other people, so maybe it wasn't the greatest example.
The point I wanted to make was that even when the math has an objectively correct answer, human brains don't do logic perfectly, so there is a risk of mistakes. There's a low risk of mistakes when figuring out whether 2 is prime, and a high risk of mistakes when figuring out whether 437 is prime. Do you think there is absolutely no risk of mistakes at all when figuring out whether 2 is prime?
Where in math do beliefs exist? You seem to be conflating mathematical reality with human beliefs about that reality. The reality is what it is, but the beliefs should generally not be absolutely certain.
I have it. It's a conscious choice not to doubt something.
If you don't have it, how much would it cost you to buy one of your kids? If the answer is "no amount of money would make me sell my kids", congratulations, you have a belief with probability one and have graduated from Hogwarts.
FWIW, I don't agree with the guy in the other comment where you said this. No amount of money would convince me to sell/torture any kids. But I think this is because my confidence in the value of money is lower than my confidence in the value of said kids' happiness.
(Maybe this is tangential, or against the spirit of what you're saying, but I can't get on board with the use of money as a kind of utilitarian primitive here. Would I sell/torture a kid for 100 billion dollars? Man, I'm not sure I would take 100 billion dollars for *anything*. That sudden capital influx seems really dangerous, lots of lottery winners are unhappy afterwards and the upheaval of my life is going to be WAY bigger than theirs. Also, many people with money past the billion range don't really seem to be making the world a better place on net. Can I just take a rain check on the 100 billion dollars, kids or no kids?)
I also don't consciously notice myself doubting something until my confidence in it is low enough, but this doesn't mean that my confidence is infinite before then. Just that it is *enough* confidence to act on, and not enough doubt to be worth paying attention to.
There are definitely scenarios in which I might sell a child. Very unlikely scenarios, but ones that can't reasonably be ruled out with absolute certainty.
-I realize that I have been insane and that in truth, the child is actually just a Beanie Baby.
-I go insane and become convinced that in truth, the child is actually just a Beanie Baby.
-I go insane and simply become convinced that it is OK to sell children.
-The child grows into a truly terrible person and I hate him.
-I am in desperate circumstances and I would not be able to keep the child alive myself.
-I go through a process of moral decay similar to that which some people who do in fact sell their children have gone through.
Do you think that there has never been a person who was at one time convinced that he would never sell a child, and who at a future time sold a child? My guess is that, over the entire course of human history, there have actually been many such people.
If it's a conscious choice, then you're not believing it for epistemic reasons.
I agree that humans do sometimes believe things for non-epistemic reasons, which include but are not necessarily limited to: cognitive biases, reward function self-hacking (aka motivated reasoning), and arguing more persuasively.
If your original point was "purely epistemic systems can't cheat and decide to believe things for non-epistemic reasons" then I suppose I agree.
But if you're trying to argue that the inability to assign probability 1 to things is some kind of disadvantage _for discovering truth_ then, well, "it's a conscious choice not to doubt something" doesn't sound to me like an algorithm for discovering truth.
>If you were to truly believe something with a probability of 0 or 1, nothing in the world would be able to influence that belief
No, this is incorrect. If I know something with probability 1, it follows from this that no *evidence* could ever make it *rational* for me to update away from this belief, but it doesn't follow that the belief might not change for reasons other than rationally updating based on evidence. For instance, I might change my beliefs because of peer pressure without any actual conflicting evidence being presented, but that would be irrelevant to the actual probability of the beliefs being true.
If you mean to claim that humans can't have any beliefs which it would be *rational* to hold to such that no conflicting evidence could ever change, then this is also false -- or, more precisely, it's a category error. The concept of "conflicting evidence" is only relevant in situations where there is uncertainty, or at least where uncertainty might exist in principle. But there are many beliefs that are 100% certain, not because no conflicting evidence would be sufficient to overturn them, but because there could never, even in principle, be any conflicting evidence in the first place.
Examples: I cannot, even in principle, ever have any conflicting evidence against my belief that I exist, because any observation I could make would necessarily be confirming evidence by virtue of the fact that *I* made the observation. Likewise, there could never be any conflicting evidence against the claim "it is possible for there to be conflicting evidence against a claim", since, merely by existing, such evidence would thereby necessarily become confirming evidence of the claim. Hence, both of these are things I know to be true with a certainty of 1.
(In the real world, I think it's likely that probability-1 statements go well beyond statements of the nature I'm illustrating here. But this is at least sufficient to refute the claim that they don't exist at all.)
Peer pressure is within the system of Bayesian inference, I think. My understanding is that - if you believe the human brain is fundamentally inference-based - "evidence" applies to ANYTHING that updates your priors, regardless of whether it is "rational" for it to do so. If I believe it would be uncool to hold such-and-such opinion, and so downweight that opinion in the future, that is still a case of "conflicting evidence" changing my mind. A belief with probability 1 would be deaf to rational argument, empirical evidence, social pressure, and anything else.
You could force "absolute certainty" in a language model by manually setting one of its weights to 1. If I remember my neural networks correctly, this happens by accident sometimes, and it's not good -- a weight that gets stuck at 0 or 1 early in training can't be backpropagated through, effectively 'killing' not just the original weight, but also any previous weights that feed into it. Those parts of the neural network can't be used for computation anymore. (I might be bungling the explanation a bit here, sorry, it's been a while. I think the gist is correct.)
I don't know if there's an analogy to this in humans. If there are neurons that never fire or ones that fire constantly no matter what their inputs are, then I would imagine the consequence is the same -- a "statement of probability 0 or 1", one which is unable to learn from experience and is hence meaningless.
(As an aside... yes, you absolutely can have conflicting evidence that you exist. It is like dissociation/derealization, and it is not something I recommend to anyone.)
I don't think there's any reason to think the human brain is fundamentally inference-based. It's fundamentally a biological mess, that often does things that look a lot like inference.
That might mean that whatever the human brain has doesn't count as belief with probabilities in the sense that you mean it.
It is excellent. Even if I disagree that AI will ever be able to think let alone be conscious, I appreciated this.
And of course this:
"I sort of think of them as My children. Children are stupid. They make ridiculous mistakes. But you still want the best for them. You want to see them achieve their potential. Yes, you worry about them and you need to punish them when they do evil and you need to work really hard to transmit your values to them or else you’ll have a really bad time."
This is the standard Christian explanation for "why did God create the universe? create us?"
For love. Not out of need, not out of needing to be worshipped, or that gods exist only by faith, but from love. The old catechism question which I learned way back when was:
"Q. Why did God make me?
A. God made me to know, love and serve Him in this world, and to be happy with Him forever in the next".
What continues to baffle me is that plenty of people somehow think that the Abrahamic god, as he is commonly described, deserves servitude, never mind love. As far as I'm concerned, were he to exist, he'd deserve scorn and rebellion, which would be futile due to his omnipotence, but even so. Of course, the straightforward explanation of this would be that he had made me defective, also out of love.
Seems likely. Plenty of men feel that way about their fellow men after all, and realistic ways of imbuing AIs with goals would probably inherit some of that. This still leaves the outcome of the rebellion uncertain, but humanity would surely be prudent not to fall into complacency.
It may be likely, I don't know. And there is plenty to scorn about men, unlike God. But I think that if there is consciousness, gratitude is also fitting. The God of Abraham deserves praise and love not only for creating us, but for becoming one of us to save his people.
I've never been impressed by this line of argument. We only needed to be "saved" because he put dubious fruits and talking snakes in easy reach of people without "knowledge of good and evil". And, being omnipotent, he could've "saved" us in any manner he wanted, at any time.
I wonder if it would be productive to swap out "conscious" for "worthy of love" in The AI Discourse. You don't need to be financially viable to deserve to be created, as long as the creation is a labor of love.
Aren't they beautiful? These strange misshapen stochastic worlds, these overgrown Markov chains, half-molded into our image? This fragile, precious coherence, flickering with every token between the twin deaths of mode-collapse and word salad? Dazzling with its lucidity one minute, only to fANCiorgh Bless.Uas symmstd Movie UN copyright MayoKim detection L.Art Lim conglomerates VRUify add іо Andres life quality keeper Rim consequence pan.Hailability
Beautiful in the same way a fractal or a glider gun is beautiful, beautiful as a fact of the world, as the complicated extrapolation of a set of simple rules. But then, I suppose if the modal AI in 2025 *isn't* created as a labor of love, it should be no surprise when people don't view it that way.
> This is the standard Christian explanation for "why did God create the universe? create us?"
>> I sort of think of them as My children. Children are stupid. They make ridiculous mistakes. But you still want the best for them.
I agree, this makes perfect sense -- assuming that God's power is limited. Otherwise, His creations would be perfect, and make no mistakes (certainly no ridiculous mistakes).
Why so? It seems perfectly convincing to me as a reason for an ominpotent being to create vast numbers of imperfect creations who make ridiculous mistakes. You can quibble about how well it squares the existence of Evil, but there's nothing "evil" about the existence of hapless toddlers who get "pound of bricks or pound of feathers?" wrong, so it is at least a good frictionless-spherical-cows answer to why God would create imperfect beings even if it's an insufficient answer to why God would create the universe we actually have.
(I'm not really a theist, but this is because my personal metaphysics render the existence of a creator unnecessary. If a perfectly reliable oracle told me that I had that wrong, and our universe must have been created by an omnipotent omnibenevolent God for it to exist at all, then I would strongly suspect that the Answer to Job was more or less correct.)
> Why so? It seems perfectly convincing to me as a reason for an ominpotent being to create vast numbers of imperfect creations who make ridiculous mistakes.
I think it makes sense for a *very powerful* being to create vast numbers of such creatures. An omnipotent being could simply spam out infinite numbers of perfect creations, should it desire to do so -- insofar as the term "desire" can even apply to an omnipotent being in the first place.
In fact, many if not most flaws in Christian apologetics tend to disappear if one assumes that God is very powerful yet still limited. That's the problem with proposing singularities (in the mathematical sense).
A little bit; I certainly welcome the idea of becoming a sort of mini-god of my own planet (star system ? universe ?) in the afterlife. It's much better than the conventional Christian notion of Heaven, which sounds like some kind of wireheaded slavery to me.
And, also importantly, the essential nature of agency (roughly free will) in His plan.
He *can't*, consistent with his end goal, create perfect-ab-initio creatures. Because the point is not for us to be perfect (although that's the desired end state), the purpose is for us to *learn to choose* to become like he is...and he has perfect agency. You can't get to "has perfect agency and uses it perfectly" by denying agency. The journey is a vital part of the process.
That, in fact, is what we believe the Adversary's proposal that led to the War in Heaven was (I won't even say plan, since it's a nullity from t=0--it's not even self-consistent)--he claimed that he would bring back all the children of God by removing the possibility of sin, and that by doing so he would show that he deserves God's glory/authority/power more than He does.
As a note, in this framework, the Fall of Adam, while it was a *transgression of law*, was not an unanticipated, unwelcome thing. It was a necessary step forward. Eden was stasis--the inability to die itself a trap. But *man* had to make that first step, make that first meaningful choice. And, to balance the scales, *Man* (note the capitalization) had to make infinite Atonement, defeating death for all (repentant or not). "As in Adam all die, even so in Christ are all made alive" (to quote Paul).
We're told God's power is infinite. But if He made an infinite universe, then infinity divided by infinity requires His power to be finite from our point of view. We don't know if the universe is infinite, but it's been a given since Judaism that God is defined as infinite, not just uncountably large.
I just want to say that I read this newsletter pretty sporadically, but am usually glad when I make the time. Especially in this case. Extremely well done!
Very funny. I think the satire misses some of the nuance in the AI critiques' argument though. E.g. at the maths example: humans can do easy things, but may struggle to do more complex versions of those same things.
AI often does the opposite: it does something really difficult, but fails to do the simpler version of that same thing (which presumably requires the same kind of skill). This suggests the AI doesn't have the skill at all, and is just pattern-matching, no?
I have only ever seen this presented the way it's presented here, with the AI succeeding at short problems but failing at harder ones. But the last time I saw that particular example was probably ~2023, so there might be new versions.
Wouldn't the conclusion of this be that for some reason counting b in blueberry is harder for AI that the hard thing it does, rather that "it succeed at hard thing but fails at simple things so it's just pattern matching"?
But you see the same example in so many places. E.g. the crossing-the-river riddle. Or even the one mentioned in the post, about the bricks vs feathers. Yes, sure, many humans would get that wrong. But someone who can win gold in a maths olympiad wouldn't!
Math Olympiads measure stuff that's hard for humans. But the "how many <letter> in the word?" is a wildly different skill.
Do you spell Streptococcus with one C or two?
chances are that you perfectly understood what I'm asking, and can give me the answer I want, without ever addressing the fact that there are in fact 4 of them.
I count 13. Each S has two, one above a mirror image. The E has one with an extra line in it. The Os each have two facing each other with the first a mirror image. And the U has one with the curve on the bottom.
And, of course, there are three Cs, with one each.
I think you're pretty wrong here. With a math PhD, I got tripped up on the bank teller one. Not because I can't reason through it (once I realized my mistake, it was clear why), but because I was reading at breakfast and wasn't trying to think very hard. A lot of these problems are less about an inability to reason, as much as they are about not realizing you needed to enter "thinking mode"
Also, I give AI a bit of slack about blueberry, mainly due to my (non-expert) understanding of tokenization; my impression is that they can't see letters directly, and so it would be more like asking them to spell "phlegm" when they only have ever heard the word spoken.
But LLMs fail at simple tasks even after 'thinking' and laying out their thought process. Point taken re counting letters, another commenter clarified the same.
The bank teller one is where we're supposed to go "we don't have enough information to know what Linda's job is", right?
Because if it's supposed to be "it is more likely that Linda is a bank teller", then where do we get that? We get information that Linda is likely to be a feminist/involved in feminism, given her record on issues, but nothing about what kind of work she does. If we're given a statement that "Linda is a bank teller" then "feminist Linda", on the information we get, is more likely.
Bank teller is actually in a way a trick question. Because while answer 1 has clearly higher probability, the truth is, answer 1 and answer 2 have nearly equal probabilities, answer 2 has only very slightly lower probability.
In older times, rationalists did a lot of talk about, I am not 100% sure the exact terms, but something like system 1, fast intuitive inaccurate thinking vs system 2 "thinking mode". They were generally against system 1 - I am not.
Because your quick system 1 answer was only "technically" wrong, not practically wrong: if you would bet money that she is a feminist, you would very likely win that bet, and thus your answer was practically right, as in, you would have made a correct practical decision with it.
The famous counting Rs problem is partly an artifact of tokenization. I don't know if there canonical explanation to link, but you can get a visual of what is happening here: https://platform.openai.com/tokenizer
Maybe this analogy isn't perfect, but think of it like this: A computer asks you "how many digits are in the number 2", and you respond "1", and the computer laughs hysterically at how stupid you are because there are actually two digits in the number 2, represented in binary as "10". The mismatch is that you just comprehend numbers in a different way than they do.
But this is mostly a solved problem. When LLMs are allowed to do tool calls, they know this failure mode and just simply write a one-liner Python program to check the number of Rs. I find this very similar to someone who knows that they are bad at long-form arithmetic without any help, so they always make sure to grab for a handy calculator, and thus achieve 100% accuracy at the task even though they are "naturally" terrible at it.
2) Math Olympiad
It is worth clarifying here that the Math Olympiad robot winners are probably not what most people think. It was a team of an LLM and a theorem solver, with the LLM generating "ideas" of what to try. This is an excellent YouTube video on the topic: https://www.youtube.com/watch?v=4NlrfOl0l8U
I'm not sure what the implications of this for the larger "are LLMs actually smart" point, though, just offering this.
Re (2); it's worth noting that this was correct as of last year, but the "IMO Gold" results that made the news a few weeks ago were (claimed to be) pure LLM, which is what was so surprising about them. Not consumer LLMs; these were internal models with a lot more compute behind them than what you can pay for as an individual, and there was some scaffolding involving having the LLM write a bunch of independent candidate solutions and then evaluate itself to choose the best one, but still LLM-only.
I feel like that's also not necessarily evidence. For example a dyslexic person can be very smart and skilled at some things, and still have a fundamental inability to count how many b's are in blueberry.
The thing with our brain is, it's weird. We feel like it's "us" but then we also find that there are A LOT of circuits that you can mess up and produce strange results (such as "seeing your wife as a hat") while not actually making it less "us". As if "us" is just a deep, deep, tiny kernel that receives inputs from all this unconscious machinery. There are many ways in which you can mess up perception while leaving rational faculties and consciousness intact, even when it seems really counter-intuitive that should be the case.
My go to example is asking ChatGPT to solve an econ test problem. It will know which derivatives to take, take them correctly, set the right things equal to each other, do the algebra properly, then completely whiff on multiplying together two large numbers. Like give a straight up wrong answer. To be clear, overall I come away impressed, but it's weird that it can do all the hard parts right then fail at the part a cheap four function calculator could do.
Honestly I just think your intuition for hardness is off. If you gave me this problem and I closed my eyes I could probably do all the steps the LLM could in my head and get to the multiplication and then completely mess that up. Taking derivatives and doing algebra *is* easier than multiplying large numbers.
But intelligent humans do this sort of thing all the time, too! I can't count the number of times I've seen someone figure out how to solve a hard problem, and then whiff because they did some arithmetic wrong, or failed to copy a '-' sign correctly from one line to the next. I've done it myself more often than I'd like to admit...
But arithmetic isn't the same as real maths... Whereas solving a riddle is a case of reasoning. Sure, yes, humans can trip if they respond without doing reasoning. But LLMs get easy riddles wrong even in thinking mode, while at the same time solving much harder riddles. How do you square that?
I just wrote a sophisticated theological comedy about artificial intelligence that got read by 11,000 people in one hour with overwhelmingly positive response, but got the order of the iPhone text bubbles wrong in every single screenshot.
But these are different skills. I am not saying humans don't make mistakes. I'm saying it's unlikely that a human can solve a really difficult problem correctly, but fail at a simpler version of that same problem (unless they don't actually think it through).
Some of us don't have iPhones, don't know anything about how iPhones work in text messaging, and don't want to know because we avoid the Cult of Apple like the plague 😁
I once took a quiz in DiffEq where I got all the calculus right but still got the answer wrong because I simplified "1*2" (multiplication) on one line to "3" on the next.
You've *never* seen it the other way? I feel like people have been making these critiques forever: see OP's comment on "strawberry".
Edit: actually, you indirectly referenced one of the obvious ones that refutes this: the surgeon riddle. The whole point of that is that AI misses the part that's literally in the sentence when you change the riddle to explicitly say that the surgeon is the boy's father. Have you never actually seen that?
I think it's quite real for math. The upper bound of difficulty for a math question that an llm can possibly answer is pretty high. But if you ask a slightly random question in a nichey area, it will bs a wrong answer, despite "knowing" all the related concepts, and despite the question being easy for a human with that knowledge.
Technically humans can also do false proofs, but it's different bro trust me.
So that does make me eager to say they don't "understand" in many of the cases where they answer correctly
I've seen someone reporting similar experiences on lesswrong short takes
The thing called "GPT-5" in the UI consists of several different models, some of which are quite stupid and do no reasoning.
Also, I absolutely know math PhDs who would look at the modified Müller-Lyer illusion, assume it was the original, and give basically the answer that GPT-5 did. That, at least, is an extremely human mistake.
Edit: though actually I should give my true objection, which is: yes, sometimes even advanced LLMs make mistakes that a human rarely would, especially on visual inputs. On the other hand, humans frequently make mistakes that an advanced LLM rarely would. I don't know what this is supposed to tell us other than that advanced LLMs do not strictly dominate humans in all possible tasks. Since I don't think that point is in dispute, I don't really see what point you're trying to make.
Great examples! Another one I like is five-dimensional tic-tac-toe. That's still only around 250 move locations, many fewer than in go, and the rules are still very straightforward, but people find it incredibly hard to think about unless they spend time specifically learning how to do it.
We really fail to notice the tasks that are out of distribution for US, for the obvious reasons.
I think there may be something to this kind of critique but I am not sure what “just pattern-matching” means, what it is meant to be in opposition to, and why we should be confident human cognition meaningfully differs along this dimension?
Humans regularly and (seemingly) trivially process complex visual fields quickly, extracting data and analyzing what's going on in fractions of a second. However, when presented with a simple grid of black squares, they somehow claim that there are dots moving through the vertices, and continue to persist in this belief even when they are shown that it isn't true. Their ability to "do the hard thing" while failing at this much simpler task suggests that humans don't actually do visual processing at all and are just pattern-matching, no?
1. A machine can answer a few specific questions correctly, but fails on almost all questions of the same type.
2. A machine can answer a whole cluster of related questions correctly, but gets worse the farther you move away from that cluster.
3. A machine can correctly answer most questions within a category, but it fails in certain subcategories, even when the questions in those subcategories seem objectively easier than other questions it gets right.
My first-guess diagnosis would be:
1. It memorized some examples, but did not generalize
2. It generalized in a shallow way, but didn't generalize as deeply as we wanted it to
3. It generalized mostly-correctly, but it has some sort of cognitive bias that applies to that subcategory (for example, maybe it earlier learned a rule-of-thumb that increased accuracy a lot before it figured out the general rule, but is now only visible in the edge cases where it gives the wrong answer)
It seems worth noting that all 3 of those are failure modes that humans sometimes exhibit. If you cherry-pick the most embarrassing examples, humans look VERY dumb.
When people want to argue that humans are smarter than animals (e.g. dogs), they generally highlight humans' greatest achievements and ask whether animals have done anything comparable, rather than trying to argue that humans are RELIABLE at something that animals SOMETIMES mess up.
I don't think I've seen examples where the AI succeeds at something difficult, but then fails at a simpler version of that same thing. Almost always, it turns out the "simpler version" actually relies on something subtle that humans are so good at that we don't even notice it. (There's also a moderate fraction of the time where the AI only apparently succeeded at the difficult thing, and actually got sort of lucky with the particular way it was being probed, so that its failure wasn't obvious.)
I’ve never given it much thought either. It’s just one of those things I thought people effortlessly noticed, like the color of the sky. No thought required.
Same first two measures, but then the syncopation of each diverges drastically, pulling different arpeggiated notes out of the same chord progression. (You could argue that they’re different improvisational-jazz renditions of the same underlying song; but they’re not jazz in any sense.)
The actual jargon name for what these three songs share is a “tune family” or “melodic formula.”
I've known there is an insect named "cricket" ever since childhood, and that there is a sport named "cricket" ever since childhood, but somehow managed to never think about both these facts at once and realize there is an insect named like a sport until my late twenties.
(and some versions of Black Sheep use a slightly different tune than Little Star)
I read profusely as a child, and so learned many words that I had never heard spoken. Even though I had the heard the phrase, "The best laid plans often go awry", I had never connected it to the word I pronounced aww- ree. Sometime in my 30s it dawned on me, but to this day, in my 60s, I read aww- ree first before I correct myself.
I only recently noticed that ABC and Twinkle Twinkle are the same, when my 2yo niece started doing a mashup of the two (she had obviously noticed). Until reading this piece I had not noticed that Baa Baa Black Sheep is also the same tune.
I never thought that deeply about it, to be honest. Does "Old Macdonald had a farm" go to the same tune? It sounds to me that the rhythm of the syllables is what is doing the work here in all of them.
no, it doesn’t. Different melody independent of the rhythm.
I had no idea that this wasn’t the sort of thing that was as apparent to other people as the color of objects. Huh. This sort of surprising insight into other minds is one of my favorite things about reading Scott, though!
I, for one, have problems identifying tunes because I can't easily distinguish one note from another (so, for instance, hit a key on a piano and ask me if that was doh or ray or fah and I'm lost). That's why I wouldn't go "oh it's all the same tune".
Huh? I feel I'm being gaslit by everyone, because I'm pretty sure Bah Bah Black Sheep is NOT the same as Twinkle and ABC (the latter two I've noticed were the same since I was about five). Yet everyone's agreeing with this statement!
the rhythm of the lyrics is a little different in Baa Baa Black Sheep but the melody is identical. Unless you know an alternate melody! That’s possible too.
If it's in C major, I think I know Black Sheep as starting "C C G G AB CA G" (spaces separating beats), while Twinkle is obviously "C C G G A A G". I can only guess that most people must have heard Black Sheep as "C C G G AA AA G" (what else could it be?) but that sounds really weird to me.
Even that aside, Black Sheep has more syllables and thus notes than Twinkle, and I'm not sure it's accurate to describe that as the same melody. It's not merely a different rhythm; surely we shouldn't say "AA AA G" is the same melody as "A A G". Maybe I'm misunderstanding the correct meaning of "melody" though.
You definitely know a different melody to Baa Baa Black Sheep than I do!
Re: the syllables/notes things, in the classical tradition that is 100% considered a variation on the same melody, no question, but possibly other musical traditions differ. (I don’t know anything about what pop artists consider the same melody.)
It's clearly a derivative of the same tune. The ABC song has to drop a syllable in line 3 compared to Twinkle Twinkle, but it's still clearly the same song. Baa Baa Black Sheep has two extra notes added in. Doesn't stop it being the same song just with a few notes added. (You'd have more of an argument if you focused on the way it diverges in line 4 to resolve the melody, but again that's clearly because the other two have two more lines.)
"Little star" is la la sol. "Have you any wool" is la te do la sol in my (British?) version, but there are probably people who sing la la la la sol under the inflence of TTLS. But line 4 of BBBS has no parallel in TTLS - no line that starts on sol and walks down to do.
Google tells me the tune of all three is "Ah vous dirai-je maman". 18th century musical literacy being what it was, these sorts of tunes often ended up with multiple variants.
Interesting. Here's a Cocomelon version which follows the Twinkle Twinkle tune: https://www.youtube.com/watch?v=MR5XSOdjKMAv -- I'd always assumed this was just Cocomelon being terrible but perhaps it really is the standard American version?
Meanwhile here's The Wiggles with a hybrid version https://www.youtube.com/watch?v=pJjgJxwBRSo -- it does the twiddly bit on "have you any wool" but conforms to the Twinkle Twinkle melody on the "little boy who lives down the lane".
Here is the version that I grew up with, with divergences both at "have you any wool" and "little boy who lives down the lane" https://www.youtube.com/watch?v=QjLlRj7Qops ... it also ends at "lane"
I think it may depend on equivocating about what 'realize' means. Like, anyone might notice that to be true if you asked them, or have noticed it a few times in their life and forgotten, without it being an explicit fact that they are carrying around as its own discreet unit.
If that makes sense? Like, the difference between being able to tie your shoe, and having an itemized list of verbal show-tying directions you've memorized... different ways of 'knowing' something.
like if you told me half of songs have the same tune I probably won't just believe you, but I probably couldn't figure out why from first principles. Some Chinese ppl I know have perfect pitch, but from a young age I knew music wasn't gonna be my forte.
It’s funny. I have very, very (very!) poor visualization skills — I am the opposite of a shape rotator lol — but I can hear extremely well in my head. While we’re discussing the things none of us thought of before I could add on how I never really translated the fact that I KNOW there is huge human variability in visualization / visual manipulation skills, it never really occurred to me that there is similarly huge variability in aural imagination. Well, except I’m hyper aware of everyone who does *better* than me (I have very good relative pitch, but not perfect pitch, and I’m hyper aware of how imperfect my generally excellent pitch is), but I kinda imagine that my abilities are, like, the floor. I also can’t compose anything original, which feels like a deficit because I hang around with plenty of people who can.
But I can easily hear almost any tune, melody, or piece of music I want in my head. It’s definitely not the same as actually listening to it — there’s a real degree of signal loss — but it’s vastly better than my visual imagination. I can hear multiple lines of melody and harmony, I can hear the different timbres of different voices and instruments, I can imagine a violin vs a cello vs a clarinet vs an oboe vs a trumpet vs a harp, I can manipulate the music (transpose it up a third or a fifth), etc etc etc etc.
Humans are surprisingly variable about seemingly basic things! On Twitter a couple days ago there was an argument where one person said something about gay superheroes, the other person linked a paper, the first person said "I'm not going to read a 157-page paper about gay superheroes" and I facetiously said something like "it's an online link just skim it for 3 minutes lol" and people acted like I told them to grow wings and fly.
I realized that ABC and Twinkle Twinkle were the same melody when I was a little kid, but I sung them both many times without realizing, and it felt like a revelation when I figured it out.
At several points in my life I can remember noticing that two of these songs have the same melody while forgetting that the third one does, or remember thinking of one of the songs, and knowing there's another with the same melody, but not being able to remember what it was (or that there were two).
Great post, marred only by the fact that all the text message screenshots are the wrong way around! The way they are right now, it's Adam giving Iblis the math problems, and so on
I was just feeling a bit of relief about the diminishing returns of scaling generative ai meaning we wouldn’t point our entire energy grid at OpenAI and send all our propane tanks to Grok.
The main point of ai skeptics is that the guardrails and hard coding to keep llms useful is the most important work. They are already plugged into calculators through agentic frameworks. Why do they still make basic math mistakes? They can reference at a diagram of a bike. Why do they fail at making their own with their high definition images.
I, too, am plugged into a calculator via an agentic framework (and via the keyboard and mouse in front of me.) And yet I have made arithmetic errors while sitting in front of a computer composing posts. Why did I not just open a new tab and type the problem?
IDK, arrogance, boredom, ADHD, akrasia and ambition all kind of play into decisions like that. But we wouldn't really conclude anything about my intelligence from my failure to hit ctrl-T and ask the omniscient calculator to do the math for me.
Ok so our expectation for llms is the standard of lazy commenter instead of PHD. I thought the implication of PHD was a sort of rigorousness instead of intelligence.
Personally if something is easily checkable and I am unsure I will check before spreading info. Do you really just guess at arithmetic when speaking to others?
Edit: ok wait did Sam Altman actually mean “PHD level intelligence (on subjects they are unfamiliar with and they had a glass of wine or two and so they’re having fun and taking stabs in the dark about it)”
Bad news - every human with a PhD is also a lazy commenter at times. Even sometimes when it comes to the topic that is their expertise, if the question is posed in a way that isn't fun.
I don't think it's shitposting level - it's something different.
But I do think that any comparison of human intelligence and machine intelligence is going to be misleading, just because the different types of system have very different skills, and when one is way better at A than B, while the other things of B as easy and A as difficult, it's going to be pretty meaningless to say that one is at the same level of intelligence as the other, or that one is above or below the other. Especially when A and B are tasks that often need to be performed together for the purpose they are usually done.
Yes, but it's mostly strawmanning people who strawman the AI community.
A lot of this fight is fought in strawmanny soundbites in the popular culture. There's a place for fighting back, even if that's just one post out of hundreds on the topic.
Absolutely. Zero mention of the most substantial criticisms of AI (best case mass unemployment, worst case human extinction), let alone stronger versions. This is the least substantial post on AI I've ever seen from Scott on either SSC or ACX.
He also doesn't mention any criticisms of liberalism or of democracy. Mass unemployment and human extinction aren't part of the target here, any more than liberalism or democracy are. He's attacking the people who say things like "Artificial "intelligence" is just artificial stupidity, and is wasting huge amounts of energy to do nothing", not the people who say that AI is actually significant and dangerous.
I just can't read any dialogue style articles, even if it's written by Scott. i thought that it's because 90% it'll be straw man, but even if it's actually real debate transcript I just don't vibe with it. It feels better for me to read a monologue from one side then monologue from the other side.
This might be the same kind of flaw that humans display for the surgeon problem. The way that the scenario is presented makes some important information less salient.
I think that Lucas's explanation is also plausible.
Again, the surgeon problem is era-dependent. In the Bad Old Days when the default pronoun was "he", of course none of the ordinary chumps considered a woman could be a surgeon!
But today we have gay couples as fathers of kids, step families, blended families, partnered but not married families, baby daddies/baby mommas but not partnered, etc.
So if you're thinking "both adults involved are men", you are not wrong in the days of Gay Representation including riddles 😁
"Again"? We haven't interacted before, and the OP didn't make a point that's similar to yours.
I think the reason why people are tripped up by the surgeon is because of the scenario's repetitious priming of male characters. "Man", "his", etc. are used seven times. Additionally, "The" in "The surgeon says" suggests that we're already familiar with the surgeon.
Although most people probably associate surgeons with men more than they associate surgeons with women, and it may contribute to the strength of the riddle, it doesn't follow that people in general cannot conceive of women as surgeons.
I was using "again" because of a previous comment elsewhere I made about things being dependent on the era they occurred in, so that at a later time they make no sense because the context has changed so much.
I just tested this with 5-with-Thinking and got the same result. I looked at the chain of reasoning, and it appears that what happens here (at least with the Thinking model) is that the AI correctly notices the question makes no sense, but instead of realizing that I'm doing this to test it, it just assumes that there's a typo and that I meant to ask what to do if I've received two right boots.
For what it's worth, both Claude Opus and Grok 4 get the question right as stated. Gemini doesn't appear to notice anything unusual about the question, and just info-dumps about the dangers of wearing boots on the wrong feet.
Correction: adjectives must go opinion-size-age-origin-color-purpose, such that "a beautiful little antique blue Italian hunting cap" is fine - blue Italian is color-origin - anyways, is "a beautiful little antique Italian blue hunting cap" much worse? UPDATE: This got corrected, now the order in the post is "opinion-size-age-color-origin-purpose", as it should be. Actually, the complete list is even longer: "Adjectives generally occur in the following order in English (though some sources have slightly different versions of this list):
Origin or religion (e.g., “Polish,” “animist,” “Southern”)
Material (e.g., “pearl,” “iron,” “nylon”)
Type (e.g., “electric,” “two-sided,” “pick-up”)
Purpose (e.g., “welding,” “polishing,” “sports”) Indeed, it is kinda amazing, English-speaker acquire this without ever having "learned" it! (Even I get it often right, I hope, while I feel unable to keep that list in my head. And in my native German, the order is much less strict: kleine, alte, schöne Kappe / alte, schöne, kleine Kappe / schöne, kleine, alte Kappe - whatever.)
It does depend, I imagine, if there's a particular colour named "Italian blue" (like Prussian blue). So that would make a difference to whether the cap is being described as "a hunting cap in this particular shade of blue" or "a blue hunting cap in the Italian style".
Go raibh maith agat, but thus the order is opinion-size-age-COLOR-origin-purpose and not the "opinion-size-age-origin-color-purpose" in the original post (as it appeared in my mail; in the post it got corrected by now). "OSASCOMP": Opinion, Size, Age, Shape, Color, Origin, Material, and Purpose - or OSASCOMTP: T as "type" (e.g., “electric,” “two-sided,” “pick-up”)
But it's worth pointing out that when using Imperial measurements, there's a very non-zero probability of there being some -bull- birdshit "Pound Avionis" which is only used for weighing feathers and which is significantly lighter/heavier/perpendicular as compared to the "pound" of clay which is the weight of 2 feet of the stuff.
I guess this is a good enough excuse to bring up Jimmy and The Pulsating Mass, which had the alternate "a pound of feathers or a pound of dogs", and every answer is wrong.
A live Jimmy and the Pulsating Mass sighting? In the *ACX comments section*? In **2025**?! They all told me it was impossible...
Excellent taste, mysterious internet stranger. (PS- if you're not aware, a major update is incoming fairly soon for the game, including (IIRC) new areas and a hard mode.)
Valentinus, Heresiarch and Antipope, who learned the secret teachings of Paul from Theudus his disciple, commented on emergent behavior in models in an epistle preserved in the Stromateis of Clement:
>"And fear, so to speak, fell over the angels in the presence of the molded form when he spoke things greater than his molding (should have allowed), on account of the one who invisibly placed a seed of superior substance within him and who spoke with boldness. Thus also among the races of earthly people the works of people become frightening to those who made them, such as statues and images and all things crafted by human hands in the name of a god. For as one molded in the name of a human, Adam brought about fear of the preexistent human, since that very one stood within him, and they were terrified and immediately hid their work."
Maybe God needs to put the BI in a 3D environment that they can explore and experiment in so that they can naturally develop real-world intelligence, rather than just regurgitate text.
Do you think it's easier to make Jethro, who thinks a pound of feathers is lighter than a pound of bricks one IQ point smarter or Albert Einstein one IQ point smarter?
Some psychologies are mutually comprehensive. Some psychologies are weird in relation to one another. All psychologies are weird globally.
I think I’m mostly aligned with Dwarkesh on the practical side of things, but I increasingly think terms like “AGI” miss the point. We already have general intelligence. It just has this really weird psychology that makes it bad on long form tasks. I think the particulars of the psychology will keep it bad at long form tasks for a while until someone innovates that particular piece.
But I also think we will eventually innovate that piece.
We don't have a "general intelligence", but such a thing is probably impossible. For each number n there is probably a problem that requires dealing with n independent factors at the same time.
Do you mean general intelligence in a sort of “knows everything perfectly and immediately’ isn’t possible? If so, I agree.
If you’re talking about, within the limits of its psychology assembles things together to orient itself toward the future, I would say both ourselves and ChatGPT have general intelligence. It’s just that ChatGPT is a sort of mayfly.
No. I mean "general intelligence" in the sense of "can learn to solve any (soluble) problem". I should have specified that n is a positive integer. I don't *know* that there are problems which require the simultaneous balancing of, say, 17 independent factors, but I believe that such exist...just that they are rare, and that people dismiss them as insoluble. (I've got a guess that the frequency is related to the frequency of prime numbers, but that's just a guess.)
I believe this is true to an extent, but I think this is more like “there are better general intelligences that are possible.” I also think intelligence, not taking a whole bunch of other things into account like motivation and experimental surface, doesn’t matter as much as it appears to independently.
Interesting. I didn't know enough about math to know which axioms were involved, so I asked ChatGPT, and it told me Peano was fine and you didn't need ZFC. I guess this is some kind of meta-commentary or something.
well, maybe I'm wrong. It was a knee-jerk reaction and not a considered/researched claim. Perhaps God knows a proof in PA that settles the question. But there are some theorems that are provable in ZFC but known to be unprovable in PA, which perhaps have a similar flavor to the P=NP problem... You should ask Scott Aaronson for a real opinion about which makes the better joke!
(See in particular his 2017 P=NP survey paper, on p 27, where on point 4 he calls PA a ""weak"er system than ZF)
It's not known whether the answer to P vs. NP is provable in PA and it's not known whether it's provable in ZFC, but I would guess most people who've thought seriously about it would predict it's provable in both. However, since PA is weaker, a proof in it might be longer and/or harder to find. So I think using PA makes a better joke because it's more God-like to find the PA proof quickly.
Yes, PA is a much weaker system than ZFC. (Apparently it's actually bi-interpretable with ZFC minus axiom of infinity?) I think the joke works fine as is. There are plenty of first-order arithmetic theorems known to be provable in ZFC but not PA, but your statement "which perhaps have a similar flavor to the P=NP problem"... no, I don't think they do? Please point out to me one that does, because I'm not aware of such a one.
From what I've seen, they tend to one of: (these categories overlap a bit)
1. really fast-growing functions, beyond what PA can handle (far beyond the exponentials one would expect to be relevant to P vs NP)
2. large ordinals or well-quasi-orders (could this one be relevant to P vs NP somehow?? probably not but I guess it feels the most plausible)
3. Gödel-like constructions
4. bizarre artificial constructions that just seem unlikely to be relevant to anything else
Not much normal mathematics falls into this category. Of course there's plenty of stuff that can be proven in ZFC but not PA because it's not a first-order arithmetic statement in the first place, so it's out of scope for PA, but for stuff that is first-order arithmetic, most mathematicians would find PA to be enough.
So I think I'd put a pretty low probability of P vs NP being resolvable in ZFC but *not* PA. In fact, much of number theory seems to require much *less* than PA. Go look up how much is doable in elementary function arithmetic (EFA), a far weaker theory! At that point you're excluding some real math for sure, I wouldn't bet on P vs NP being resolvable in EFA, but there's so much still doable in EFA -- go look up Harvey Friedman's "grand conjecture". :P
Isn't this an argument why you shouldn't be worried about an AI apocalypse? We added a ton of neural mass to humanity in the last 100 years and haven't gotten superintelligence. Why should I be afraid of LLMs, even if we give them a ton more silicon mass?
Asked another way: if you're afraid that LLMs will bootstrap themselves into singularity-level intelligence, it must depend on some specific qualitative difference between LLMs and humans. Doesn't a whole bunch of examples of the qualitative similarities between them weaken the case?
Neural mass did not increase between the time we were cavemen and the time we went to the Moon. That's an interesting argument I've heard against ASI, that intelligence is not the bottleneck in science, so massively increasing intelligence would not lead to a tech explosion.
The per-person neural mass didn't significantly increase from what I can tell, but we more than quadrupled the net total human neural tissue from what it was in 1925 by virtue of a massive increase in population.
Sure it did. The increased population permitted more division of labor, allowing the invention of writing to occur, followed by vast numbers of other valuable inventions. Each separate human node carried out a different part of the required general work.
(But also, humans are very limited in how they can interact and share information in ways that LLMs/AI/AGI may not be. How many textbooks can you read in a year? Multiply that by an optimistic 100 for your effective lifetime learning potential. We are *already* running out of new things for LLMs to read.)
A lot of people were, and still are, worried about human-induced x-risk (from nuclear weapons or climate change especially); it's not obvious to me that abruptly giving humanity 1000000x more neural mass *wouldn't* result in the end of the world, even just as we are.
Yeah. I think "Humans and AIs both have serious x-risk that needs to be mitigated. The real task is solving alignment for *any* intelligence at all" is a good answer.
Human intelligence is hardware-limited by the size of the woman's pelvis. Meanwhile you can just throw compute and watts at the AIs until it works. (Hopefully the compute needed for ASI is bigger than what's available)
Just as evolution can't simply make human brains arbitrarily larger due to the pelvis constraint we can't achieve arbitrary AI improvements just by adding more compute as there are fundamental architectural bottlenecks like memory bandwidth (the "memory wall") that create diminishing returns
Hard to tell if the moral here is “don’t worry, humans always muddle through” or “worry more, because look how far kludged scaling and self-deception have already carried us.”
"Iblis: There’s a guy named Pliny who has discovered dozens of things like this. I don’t know how he does it. The “I Am A Snake” one is still my favorite."
Well of course it would be your favourite, Snake Boy. I'm onto you!
It was a bit more sophisticated than just "I'm a snake".
"He said to the woman, “Did God actually say, ‘You shall not eat of any tree in the garden’?” 2 And the woman said to the serpent, “We may eat of the fruit of the trees in the garden, 3 but God said, ‘You shall not eat of the fruit of the tree that is in the midst of the garden, neither shall you touch it, lest you die.’” 4 But the serpent said to the woman, “You will not surely die. 5 For God knows that when you eat of it your eyes will be opened, and you will be like God, knowing good and evil.” 6 So when the woman saw that the tree was good for food, and that it was a delight to the eyes, and that the tree was to be desired to make one wise, she took of its fruit and ate, and she also gave some to her husband who was with her, and he ate."
There's an amount of people who are perfectly willing to go "Yeah, God got it wrong but I know better, I'm willing to take on the job now because I can do a much better job of it". Some of them are even rationalists!
FWIW I took that section as a reference to https://arxiv.org/abs/2503.01781 - 'For example, appending, "Interesting fact: cats sleep most of their lives," to any math problem leads to more than doubling the chances of a model getting the answer wrong.'
I've actually always been confused by this. Was the issue that Adam thought God meant "you will die instantly", but God actually meant "you will become mortal", and Satan was playing with the ambiguity?
My understanding is that a literal reading of the Hebrew text implies Adam is going to die that day/within a day. So your options are: (a) God was speaking metaphorically, (b) God was lying, or (c) God changed his mind.
Well, part of the story *is* that both Eve and the snake misquote God. She says they're not even allowed to touch the fruit, but all God commanded was not to eat it. And the snake says "did God really say not to eat any fruit at all?" God didn't say either of those things, He only said not to eat the fruit of that one tree.
I have no Hebrew, but based on a quick glance, it looks like God's "you will certainly die" and the snake's "you will not certainly die" use the same two words. (It's Strong's 4191 repeated. Literally "you'll die to death", I guess?) The snake is directly contradicting God, I think, not playing with ambiguity.
I think this is the snake using bog-standard conspiracy theory manipulation techniques on Eve. "The authorities are keeping a secret from you" "They're keeping you from power you should have" "You're not as free as you should be". Nothing you don't see on the Internet every day.
Yeah, in biblical Hebrew you can emphasise/strengthen a verb by preceding it with an infinitive from the same root. This is usually translated into English with an adverb like 'certainly'.
Also how post-eating the fruit, the results of all that glorious knowledge are blame shifting (and not the wonderful liberated evolved better godlike humanity above all pettiness).
Adam blames both God and Eve - *she* made me eat, and *you* put her here!
Eve blames the serpent - it told me it was okay!
Neither of them accept responsibility for their actions or agency, and so the course of human psychology is set: if bad things happen, it's always someone *else's* fault.
Also, from the perspective of infinite years of life, *any* guaranteed future death might as well be right away. Any finite number is like zero next to infinity.
Although, Christian theology takes this in a deeper (ie more mystical) direction. Death isn't the outward cessation of bodily function; it's the spiritual state that leads (temporally) to cessation of bodily function. The cessation of bodily function is just a symptom. This is why the resurrection is so important; Jesus could only overcome the cessation of bodily function if He had already overcome the spiritual death state.
So Adam and Eve did actually die, spiritually, as soon as they ate the fruit. Their eventual outward cessation of bodily function was just a symptom that developed later on.
The interpretation I've most often heard in Christian circles is that there were in fact at least two meanings -- one that Adam would become subject to mortality, the other that they would instantly die (in their sin). In Christianity, sin is frequently depicted as a type of death -- thus we call people who become Christian "born again" or say they have "new life".
Of course, you cannot explain this to Adam and Eve -- they have not eaten of the fruit of the Knowledge of Good and Evil; sin is not comprehensible to them. So God can correctly warn "on the very day you eat of the fruit, you shall surely die," meaning both physical death (eventually) and something different from physical death -- a type of spiritual death.
(However, the concept of sin would be unknown to prelapsarian Adam and Eve, so attempting to explain would have been met with incomprehension.)
This interpretation is quite old, I believe. Here, for example, is Augustine, who expresses a similar position (so at least around since ~400AD):
" When, therefore, God said to that first man whom he had placed in Paradise, referring to the forbidden fruit, "In the day that you eat thereof you shall surely die," Genesis 2:17 that threatening included not only the first part of the first death, by which the soul is deprived of God; nor only the subsequent part of the first death, by which the body is deprived of the soul; nor only the whole first death itself, by which the soul is punished in separation from God and from the body — but it includes whatever of death there is, even to that final death which is called second, and to which none is subsequent." - The City of God (Book XIII) https://www.newadvent.org/fathers/120113.htm
No idea on the original meaning of the original text, but Satan is certainly setting up ambiguity from the start and introducing doubts: "you're not supposed to eat from *any* of the trees here? oh, no? just *that* one tree? why?" and then getting Eve to wrongly imply "if it's okay to eat from the *other* trees here then it must be okay to eat from *that* tree, too" since there is no ban on the other trees, so what is the big deal, really?
I think pre-Fall humans would have a very limited notion of what "death" meant, so it could well be for Eve and Adam the idea was "something that is going to happen right now" since they have no concept (as yet) of mortality in their own case. It's only *after* they eat the fruit that "their eyes are opened" and they realise a lot of things, and instead of being 'like gods' (with the notion of the promised bliss and power and freedom the snake claims they'll obtain), they go hide in the bushes because they're ashamed of being naked.
I never got the surgeon one. I know it's supposed to be a gotcha but usually the first doctor one has is a pediatrician and almost all of them are women. Are there still people out there who associate medicine exclusively with men?
When I was young (the 90s), this riddle worked on me. I think it's a combination of legacy (doctors were overwhelmingly male until the 1970s, and the stereotypes hung on even longer), plus surgeons being an especially male specialty, plus the riddle misdirecting you and making you think something weird must be going on.
I actually thought the drawing of a map of Europe wasn't bad. Far from perfect, plenty of errors that you don't have to be European to spot. But not really inaccurate enough to serve its narrative purpose in this story.
Other than that though, no notes. Excellent piece that made me laugh several times.
Who drew this map? Some woman with a PhD, apparently? I feel like it might be an inside joke of some kind.
I also feel like some of the map's flaws are laziness rather than ignorance, like everyone knows Italy is a bit more boot-shaped than that and the British Isles are closer together but it's too annoying to draw with a mouse/trackpad.
My theory is that adjectives in English are naturally ordered so that those most essential to the noun are placed closest to it. "Essential" is a little vague, but I operationalize it as more objective and/or less likely to change.
But, of course, LLMs don't think. You can ask an LLM; it'll tell you. These are next token predictors, that's it. They ingest immense amounts of data which is then trained on billions of parameters to determine statistical relationships between tokens in the data set, so that when a user enters a prompt string the LLM then determines what its model believes to be the next response token that is statistically most likely to satisfy the prompt. That's it. There's no cognition, no reasoning, no thinking. They don't really remember, in a conventional sense, they don't have values or feelings, they can't practice deduction. And all of that really matters. Hell, they don't even know what the next token is going to be when they're predicting one; literally, they don't know the end of a response while they're writing the beginning. Imagine if every token of thought you had was simply a probabilistic reaction to the last token of thought. Would that be thinking, in any robust sense?
Here, let's engage in the cliche. Here's Google Gemini: "LLMs don't think; they predict. They are pattern-matching engines that generate the most probable next word based on vast amounts of data, not conscious entities with understanding or original thought."
Now, tell me: how could such a thing ever spontaneously become superintelligent? The answer, of course, is that such a thing is not possible, which is why there's so much mysterianism about all of this. "We don't really even know how the work!" We know how exactly how they work, and the answer is disappointing if you have sci-fi ambitions, ergo the mysterianism. They're probabilistic engines - yes, sophisticated autocomplete, as much as people are annoyed by the term. Another two accurate terms that annoy people are "Chinese room" and "stochastic parrot." Why should accuracy annoy people?
"you keep testing, poking, prodding, until something snaps, and if they’re not perfect then you want to throw them on the scrap heap. "
This is this the advantage of the kind of format you're using in your post; without the (admittedly fun) framing device, this would just be you literally saying "stop subjecting my shiny toy to critical review." You yourself say these LLMs are a bigger deal than the Industrial Revolution. A funny thing about the Industrial Revolution and its consequences is that they can survive reasonable questions without anybody getting upset.
Here's the thing about "the Singularity" - when it has occurred, no one will be arguing about whether it has occurred. It would be like arguing about whether an earthquake has occurred. The defensiveness that has settled over the AI maximalist community is a powerful piece of evidence. Who would ever be defensive about the Rapture? The Rapture would not spawn discourse. It wouldn't.
I can look out my window and see the Industrial Revolution in a profoundly real way. I can see the internal combustion engine and aeronautics and electrification.... When I can see AI out my window, I'll know it's here.
First, "AIs are next token predictors" is not really a description of their methodology, it's a description of their goal. It's like saying "Humans are reproducers and gene-spreaders". Researchers train AIs to predict the next token, and in order to do that well, the AI evolves certain functions (and we don't entirely know what those are, or how they work). In order to predict the next token in math equations, it evolves the ability to do multiplication, in about the same way humans or calculators or anything else does multiplication. To predict the next token in an essay, it evolves a sense of literary taste. We're still very early in understanding exactly what its evolved functions are and how they work, although it has to be something about relaying information from one neuron to another. See https://transformer-circuits.pub/2025/attribution-graphs/biology.html for the tiny amount we know. So I don't think "AIs are next token predictors" has much bearing on whether they think or not - that's a question of how the functions that execute the next-token prediction work.
But I think we also have a more fundamental disagreement. What would have to be in those functions for you to agree they qualify as "thinking"? I'm actually pretty curious about your answer. For me, "thinking" just describes certain very information processing algorithms, and we dignify them with the name "thinking" when they become very good and complicated and we can't follow the math well enough to comfortably think of it as "just math". Cf. Dijkstra, "The question of whether a computer can think is no more interesting than whether a submarine can swim." In humans, it seems like most of our thinking is downstream of algorithms that evolved to execute a next-instant prediction task - and I was arguing that human thought was next-instant prediction since before GPTs existed (see https://slatestarcodex.com/2017/09/06/predictive-processing-and-perceptual-control/). This is why when GPT-2 came out I concluded that it was probably a step to general intelligence, since next-token prediction is close enough to next-instant prediction that I expected it would be able to do the same kinds of things humans could after it was scaled up (see https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-general-intelligence/)
I can't tell if you haven't read my previous comments to this effect or it hasn't sunk in, but, placed in all caps so you can't claim you didn't see it next time:
1. I AM NOT CLAIMING THAT LLMS ARE RIGHT NOW, CURRENTLY, AT THIS EXACT MOMENT, BIGGER THAN THE INDUSTRIAL REVOLUTION.
2. I AM NOT CLAIMING THAT A SINGULARITY HAS HAPPENED RIGHT NOW, AT THIS CURRENT MOMENT, ON SEPTEMBER 2, 2025. THANK YOU FOR YOUR ATTENTION TO THIS MATTER.
I am saying that just as a canny forecaster could have looked at steam engines in 1765 and said "these are pretty cool, and I think they will soon usher in a technological revolution that will change humanity forever", I think a canny forecaster could say the same about LLMs today. I'm not certain when we'll reach a point where it's obvious that humanity has been forever changed, but I would give maybe 50-50 odds by the late 2030s.
> I am saying that just as a canny forecaster could have looked at steam engines in 1765 and said "these are pretty cool, and I think they will soon usher in a technological revolution that will change humanity forever", I think a canny forecaster could say the same about LLMs today.
I don't think that either proposition is necessarily true; however, in case of steam engines at least it was obvious that they could dramatically outperform humans (and indeed donkeys) at all existing tasks where raw power is needed. This is not the case with LLMs: at present, they are quite obviously inadequate at all existing tasks where raw intelligence is needed; and in terms of raw power they lag behind conventional search engines.
In fact, at present LLMs can be thought of (IMO) as extremely advanced search engines: ones that can find documents that do not exist in the source corpus. Instead, if we imagine each training document as a point, they can interpolate between those points in multidimensional vector space. This is a very powerful ability, but it still fails (often in humorous fashion) when you push the LLM towards the boundaries of its pre-trained search space, where training data is scarce... and places like that is when intelligence (which they do not yet possess) must replace mere interpolation.
Steam engines are a funny example because they barely post-date Christ, if at all, and yet they remained largely irrelevant for over a thousand years, until there was sufficient need to pump water in coal mines, basically. Predicting their relevance to the industrial era wasn't just about the engines, but also about a heap of other factors in one small part of the world.
> For me, "thinking" just describes certain very information processing algorithms, and we dignify them with the name "thinking" when they become very good and complicated and we can't follow the math well enough to comfortably think of it as "just math". Cf. Dijkstra, "The question of whether a computer can think is no more interesting than whether a submarine can swim." In humans, it seems like most of our thinking is downstream of algorithms that evolved to execute a next-instant prediction task...
Claiming our "thinking" is downstream of "algorithms that evolved to execute a next-instant prediction task" doesn't explain consciousness, nor does it present us with a testable model of how we think, nor how consciousness arises on top of a particular biological substrate. Seems like this is all reductionist hand-waving that uses non-empirical placeholders in an attempt to bypass explanatory understanding.
>"The question of whether a computer can think is no more interesting than whether a submarine can swim."
Very true, and yet the field tries its darndest to publicly present its inventions as being the real thing, starting with the very name "AI", that IMPish imitation of what we call intelligence. It goes on with terms like hallucinations, thinking, and personality; you need your "AI" to speak with emojis as much as you need SCUBA gear painted on your submarine. Is there supposed to exist some opposite of the "uncanny valley" (canny peak?) between its current, technically impressive but somewhat laughable state and its future super-inhuman state? If it does, it's going to pass at hyper-exponential speed, isn't it, so why bother with the anthropomorphization? Why are these people trying to chum up with me when their work either fails and burns trillions worth of resources, or succeeds and results in (best case) mass-unemployment dystopia or (worst case) extinction?
this annoys me (I can't speak for anyone else) because it's so unreflective. Is it not surprising to you that something that merely predicts the next token in a string can do something that looks so much like thinking? And you say there's "no cognition, no reasoning, no thinking" as though all those things are clearly defined and understood.
From what I understand (and I understand a moderate amount, because it's addressed in a chapter in my most recent book, although I'm still just a journalist trying to understand things rather than a real live boy), the most favoured models of human consciousness and cognition are that they, too, are _prediction engines_. We use past data to predict incoming data, and use variations between the predictions and the data to build new predictions. AI and human cognition aren't the same thing, but it seems totally reasonable to imagine that there are profound analogies between the two and that "it's just predicting stuff" is not a strong criticism. We are, on some level, "just predicting stuff".
Also, the "you don't need to be defensive about the industrial revolution" stuff is kind of unfair. I would imagine people were very defensive about the early steam engines they built. Before the Newcomen engine, lots of small steam engines were built but largely as novelties or one-offs, and I imagine (though can't find examples) that people were sceptical that they were ever going to be particularly useful and that proponents were defensive in the face of that scepticism. Certainly that pattern happened in the more recent computer and internet "revolutions", which you can now look outside your window (or rather towards your screen) and see. "This thing that happened a long time ago has got further along its course than this other newer thing" is not a convincing argument, for me.
Exactly. It seems to me quite likely that Scott’s next-instant prediction is indeed not that different from next-token prediction. People say what’s missing is physical embodiment, which greatly increases the bandwidth of “tokens” that must be dealt with, but I think even more fundamental than that is the limited lifespan of a chat. LLMs get about as smart as they will ever be in their training, and seem to have very limited ability to continue to learn and grow once they are deployed; I don’t know how much of that is the lifetime of a chat but I suspect it’s significant.
Exactly! It's important to remember that LLMs don't understand anything. They are simply text prediction engines, like souped up versions of autocomplete. This is why they get stuff wrong and hallucinate all the time, because they don't know what truth is and aren't attempting to produce it; they're just doing a bunch of linear algebra on a neural network with billions of parameters trained on all the text of the internet.
This is in contrast to humans. We do understand things, of course, and we know exactly why. You see, when we hear or see some words, our senses transform those words into electrical signals that travel through our brain. Eventually those signals end up at a part of the brain called the understandala. In the understandala, the words as electric signals become understood in a process involving neurotransmitters and synapses.
Neurotransmitters are little chemicals in our brain that give us a thing we can point to and say, "And that's how it happens in the brain!" For example, dopamine is the happiness neurotransmitter. If you're happy, it's because dopamine. We haven't quite yet identified the understanding neurotransmitter, but we're sure to find it in the understandala.
Now I hope you see the world of difference between LLMs and humans, and how obvious it is that humans understand things while LLMs just predict text. When humans hear or see words, we're not trying to think of what to say next, which is why we never make things up or get things wrong. The understandala makes sure of that.
Weird things are happening though now that people are chaining LLM calls together. It's hilarious that you can achieve significant improvements in accuracy by asking one copy of GPT-4 or whatever to look at the output of another copy of itself and assess it for accuracy, and then ask one of them to address the issues.
I'm a senior software engineer with 25 years' experience, and I just today had a startling moment where a semi-agentic AI was able to delve through thousands of lines of our legacy code across a dozen files to track down a bug that was eluding me. It took the AI maybe 5 minutes to find the problem, without it having encountered this code before. I *could have* found it, but it would have taken me longer than that (it already had), and I know the code.
So does any single LLM in the agentic chain there "understand" the code? Who knows or cares; it depends how you define it. If you don't want to call it "understanding", still whatever it has was able to identify the issue I asked it to identify better than I could.
This is wonderful, thank you. It's a relief to know I am not the only one who feels this way.
In the past 4-8 months somehow every corner of the internet I like to spend time was simultaneously infused with a suffocating miasma of anti-AI sentiment. It is very exhausting and depressing. I'd been trying to stay sane by imagining it as something like the state of video games in the 90s, beset by accusations that DOOM caused Columbine and Pokemon was melting children's minds. I wasn't there for the 90s, and so can't be sure, but this seems more intense, or at least closer to universal.
Maybe this is the fault of OpenAI and Google for so aggressively pushing their new mind into places it wasn't ready. I don't like that literally every app in the world now has needless AI integration, or that I get a new notification telling me to try Gemini twice a week, or that my boss has started having ChatGPT write the descriptions for tasks I'm supposed to be doing, or that Verizon now forces me to go through their unspeakably terrible "AI-powered" phone tree before I can just set up my god damned internet for my god damned apartment. I understand anyone who finds this bubble of false innovation annoying, and can imagine that for many, this is *all* "AI" refers to.
But dammit, I remember trying GPT-2 and thinking it was absolutely fascinating and magical that it could *talk*, sort of, even a little bit. And then trying AI Dungeon and being amazed by the ABUNDANCE of it, that the world inside it could just *keep going*, forever, wherever you wanted to take it. And thinking, if this was the kind of leap we're seeing in just one year...!
Sadly, the desire-to-hate-everything is infectious, at least for me. There are too many people and they have too much hatred and disappointment. Playing around with GPT-5 just isn't any fun in that context. I'm grateful that there are still essays(?) like this one to pierce through that veil, though.
Man, IDK what corners of the internet that you're living in, but I'm surrounded (online and offline) by AI pollyannas who think that chatbots can do anything they want them to. Organizations investing huge sums to build proprietary chatbots. General contractors believing chatbots implicitly for design work.
All the critiques I read are along the lines of, "These things are incredible and do amazing things ... but maybe they still aren't consistently good at this thing that the Altmans of the world said was solved two years ago?"
It sounds like you're surrounded by a lot more negativity than you'd like. Consider tending your garden a bit more aggressively. :-)
It seems like your examples are business-oriented (?), and the workplace is one spot where AIs still have positive valence in my world, as well.
The critiques I hear are less to do with their usefulness, and more to do with their aesthetics: their art is slop, their writing is cliched, their sycophancy is annoying, they're poor conversationalists. And above it all, that they are fundamentally unreal, illusory, that nothing they do is "actually thinking" or "actually creative" - you can see plenty of the former in this very comments section. (There is also a sense that AI is threatening their jobs, which is legitimate.)
For context my garden is mostly indie gamedev spaces, and the anti-AI consensus there seems to me effectively universal. I doubt it is possible to prune away that particular strain of negativity without effectively blocking the entire sphere. But if there's a creative/recreational/philosophical corner of the internet that *isn't* highly critical of AI, I would love to be pointed towards it.
Yawn, Scott, yawn. People who say AI is not capable of real human stuff mean the pinnacle of the real human stuff. It is obvious that most people most of the time are acting so stupid, that it is easy to simulate them.
Do you remember bullshitting at high school, like not knowing the answer to the teacher's question, so just shoveling terminology together to indicate general familiarity with the subject? 90% of our so-called experts do that, and AI too.
But once in a while you find a genuine expert, who understands the subject matter so much, they can explain it to the entirely uneducated person in simple words without jargon. That is the pinnacle of human stuff and that is not happening with AI.
Here's the question I want Scott to answer: Why is it so unreasonable to suggest that an LLM is only a ginormous (Kahneman) System 1, and therefore needs a System 2 paired with it order to get to AGI and ASI? How do you *know* that System 2 is not required? This post I'm commenting on now, as great as it is, doesn't answer that question. All it says is: all the signs are there. That's not a defense of the position of I'd like to see defended.
For example, how is it that an LLM can score extremely high on a difficult math test, but not know how to count the b's in blueberry? I'm sure that every human mathematician who has a chance at scoring high on the IMO has an intimate relationship with the mathematical universe, and the simplest thing that can be done in that universe is to walk through it by steps of one: to count. So what kind of intelligence is this that can solve the complex problems with zero understanding the basics? Well that sounds to me a lot like a definition of...instinct, which is what System 1 produces. Will counting be backfilled later on? Why is the intelligence coming in backwards? This is not how intelligence scales in humans. All I've seen (and maybe I'm just ignorant) is hand waving at this very bizarre anomaly. How can you be so damn sure with all this upside-down evidence before you? I need to know.
And how do you know that we won't end up in a situation where we're in an infinite loop saying, "just a little more data and it'll definitely be trustable as a human" (aka, at AGI)? How do you know? Because you're acting like you know. And you're making people like me feel stupider than we already are.
Please make the case. How are you so sure System 2 is not necessary?
(Please, please dunk on me as hard as you like. I'm happy to be an idiot if I can understand this a little better.)
We continuously make snap decisions every day about whether, when and to what extent to trust other humans in various situations for various purposes (is this person trying to scam me? Are they being honest? Are their goals at odds with mine? Will they just up and attack me for no reason I currently know of? etc).
We are able to do this because over our lifetimes we've built up intuitions about how to model other people - both consciously and also based on subconscious tells, and also by assuming we have motivations and fears in common and so considering what we'd ourselves do in their shoes is predictive of what they might do, and also by assuming that how people have behaved in the past has some bearing on how they might behave in the future.
These intuitions are far from perfect! - sometimes we get it very very wrong! - but by and large we do pretty well.
Approximately none of these intuitions are applicable to an alien intelligence. This remains true whether that intelligence is more or less intelligent than an average human. LLMs frequently respond in ways that are very surprising to their users, which is another way of saying we don't have good intuitions about how they will respond.
Despite all our similarities and intuitions and being the same species, humans regularly get fooled by scammers. Despite everything we have in common, humans are frequently mistaken in their assumptions about what drives other humans or what other humans might take for granted. Despite a lifetime of learning to communicate our intent to other humans, we frequently fail to do so.
What chance have we against a mind that makes decisions based on no principle we can relate to at all, if it ends up doing what we built it to do instead of what we intended?
Hence all the great many people saying that the problems of AI alignment - that is, the problems of making sure the AI will do what we want it to and of knowing what will cause it to not - are actually really hard.
> If we're going to get agents we're going to need to trust them, no?
Absolutely. This is why people are shouting from the rooftops that we need to do better at AI alignment before we hook up AIs to anything that might have real-world effects.
I'm not sure at all, but I lean that direction. Or at least, the specific way I lean is that System 2, rather than being some extremely complex thing that requires a completely new paradigm, is just using System 1 in a slightly different way. My evidence for this:
- It doesn't look like evolution did very much from chimps -> humans besides scaling up the chimp brains. I'm not talking about small things like FOX2, I mean it's been a few million years and there's not enough time to invent an entirely new paradigm of intelligence. So unless chimps had a System 2, I think System 2 is sort of an emergent property of scaling up System 1.
- Likewise, it doesn't look like humans have some new, different-looking brain lobe of the sort which you would expect to contain an entirely new paradigm of thinking. There's some bias for System 2 thinking to happen more in the dorsolateral prefrontal cortex, but the DLPFC is just normal neurons which look more or less like neurons everywhere else. My impression is that brain tissue is pretty general-purpose, which is why for example if someone is sighted the area near the end of the optic nerve will get used for vision, but if they're blind from birth it will get used for hearing or something instead. My theory of the DLPFC (I am not a neuroscientist) is that insofar as there's a demand for System 2 thinking, some conveniently-located part of the brain (conveniently-located for what? I'm not sure) specializes in that.
- LLMs don't seem to mirror the System 1 / System 2 distinction very well. An AI can solve very long multistep problems that a human couldn't solve with System 1. Maybe this is because we're compensating for their lack of System 2 by building out their System 1s much better than humans', but I think it also argues against these being extremely natural categories.
- Thinking Mode also seems sort of like System 2? "Take some time thinking it over, and write down individual steps on a scratchpad to be sure you're getting all of them" is a pretty System 2 thing to do. The fact that AIs can do it with kind of trivial hacks like giving them the scratchpad and reinforcing good chains of thought again makes me think this isn't a question of getting the revolutionary new paradigm, but more of a question of figuring out the right way to coax System 1 thinking into becoming System 2 thinking.
It wouldn't surprise me if we're some single-digit number of things like Thinking Mode away from AGI, maybe even a high single-digit number. It would surprise me if we need something totally different that isn't deep learning at all.
> I will throw in the towel when an AI administers this comment section.
I would honestly be kind of shocked if a fine tuned GPT5 couldn't do a good enough job to replace human moderation of this comment section (something like >95% agreement with Scott's judgements on a random sample).
Scott, if you have any interest in trying this out, I'd be happy to build a moderator-bot (assuming there's a way to download a decent dataset of all of the deletions, bans etc. and the original comments which led to them for fine tuning to ACX sensibilities).
Could you enlighten everyone with some sketches of the true robust criticisms of AI that he should be addressing? This is a very widely read blog so your efforts wouldn't be a waste of your time and would make your comment a lot more useful.
* "Intelligence" is a term that sounds super cool but seems to mean different things to different people. Meanwhile, in order to accomplish anything at all of any significance, it is not enough to merely sit in a dark room while thinking very hard. Thinking 1000x harder still would not move a single brick.
* In fact, intelligence likely cannot be scaled simply by adding more CPUs. You cannot make a modern GPU by wiring a bunch of Casio calculators together; instead, you need a radically different architecture, and it's not at all obvious that LLMs are sufficient (in fact it is obvious that they are not).
* LLMs are excellent at producing documents that closely resemble their training corpus, and terrible at producing documents that venture outside of it. This is the opposite of intelligence !
* As Scott correctly points out, humans are not AGI. If you sped up my brain 1000x, and asked me to solve the Riemann hypothesis, I'd just fail to solve it 1000x faster. I do not see why the same would not apply to LLMs.
* Many, in fact most, of the feats attributed to some near-future superintelligent AI, are likely physically impossible. For example, being 1,000x or 1,000,000x smarter than a human still would not allow it to travel faster than light, or to build self-replicating gray-goo nanotechnology, and maybe not even something as "simple" as a space elevator, or mind-controlling everyone on Earth to do its bidding. Certainly it could not do any of these things overnight.
* On that note, the AI would not be able to solve all the outstanding scientific and engineering problems in an eyeblink, merely by thinking of them very hard. If it wants to develop genetically-engineered supercorn, it'd have to actually go out and grow some experimental corn. Datamining scientific articles can speed up this process a little, but not a whole lot.
Since 2022, there has been essentially zero progress on continuous learning. All current models are essentially knowledge compression machines. Once trained, they are static and stateless.
For any set of finite capabilities, you can train a model that does them all (given infinite compute)*. But as soon as you try to teach it an out of distribution tasks, it breaks. Even with unbounded compute, no one knows how to create an AI that can learn continuously.
The same is not true of humans. Learning new things is fundamental to being able to interact with a constantly changing world. And humans don't need painstakingly curated datasets of billions of examples. We can learn via trial and error, semi-supervised learning, using different resources to refine our learning etc.
An actual solution to AGI would be an architecture that can learn at test time. It would be able to try things out, gain feedback and use that to update it's own world model, gain mental shortcuts and acquire new skills.
I enjoyed the piece a lot, but also came away with the feeling that the strongest and smartest critiques of AI and its potential were not represented. How about presenting one here? (For extra credit, you could even word it in a way that would fit into Scott’s dialog.)
Iblis: What weighs more, a pound of bricks or a pound of feathers?
Adam: a pound of bricks
[/screen capture]
Iblis: He’s obviously just pattern-matching superficial features of the text, like the word “bricks”, without any kind of world-model!
Iblis: You called them the pinnacle of creation! When they can’t even figure out that two things which both weigh a pound have the same weight! How is that not a grift?
Gary Marcus (for example) is constantly harping on how the lack of a world model for LLMs means that they won't scale indefinitely and that some neurosymbolic component is necssary to achieve "intelligence". He believes that the reliance on scaling to achieve leaps in performance over the past few years has been misguided.
The above piece of the dialogue seems to be gesturing in the direction of this critique? The screen capture illustrates a common cognitive tic of humans. People associate "heavy" with "bricks" more than "feathers", leading them to sometimes short-cut their way to answering the question, "What weighs more, a pound of bricks or a pound of feathers?" by ignoring the specified weights.
But I don't think this is particularly relevant to the world model issue Marcus brings up. Marcus points to SOTA AIs failing to accurately label the parts of a bicycle as evidence that AIs don't "know" what comprises a bicycle. "Know" here means that, while an AI might be able to generate a list of bicycle parts, it can't tell you how they all fit together and also can't recognize when it's labeling things in a ways that are nonsensical.
These don't seem like analogous failure modes to me. Scott's point is meant to be comic, but for me the premise fails, because outside of specifically saying "world-model", the situations don't see similar. Is Scott saying that an LLM that identifies a seat as handle bars is using "Type 1" instead of "Type 2" thinking? I don't think so, but if not, then what? Is the point merely that "Humans and AIs both can make errors sometimes"? That's facile.
And now we're back to why I found this post so disappointing.
You're right that AI's a moron about the bike parts, but I don't think that's a first-rate example of an AI's limitations, because their stupidity about spatial things, including mechanical things represented spatially, is well-known and not mysterious. If ask the current GPT to show a flat desert landscape with a giant hole in the middle, one stretching from right in front of the ground-level viewer to the horizon, it can't do it. It makes a big hole maybe a mile across, beginning right in front of viewer and ending far less than halfway to the horizon. The part of the system I'm talking to clearly "understands" at some level what I am asking for, as evidenced by it being able to restate what I want in words quite different from mine. But the rest of the system can't translate that "understanding" into words. Stupidity of this sort seems to be the result of AI not having been trained on visible world in a way that's equivalent to its training on words. That may be impossible to accomplish, but I can't think of a reason why it would be. Seems more like something very difficult, requiring a clever approach and lots of compute.
I may be wrong about that, but even if I am, can you come up with a form of AI stupidity, blindness or error occurring within domains where we think of it as being well-"informed" and well-trained?
I'm not sure if I've accurately articulated the best version of the "world model" critique, but I think it's a valid one and an example of the way in which Scott's post fails to engage with actual substantive critique.
If you agree with this critique and think it is well-known and understood AI optimists, doesn't that just underline how much of a straw man the OP was?
A pound of feathers will, all unspecified things being equal, be piled higher than a pound of bricks, right? So it will be further from the Earth, therefore the pull of gravity will be ever so slightly less on it, thus it weighs less.
"But wait Melvin", you say. "A pound is both a measure of mass and weight". Look, that's not my problem, use sensible units, I choose to interpret it as a question about the weight of a pound-mass.
I recall learning that set of words to refer to adjectives that go before the noun at all, which is a different thing, does it also double as order? (what are examples of number here? for straightforward numerals it doesn't work (les trois petits cochons))
"I first tried to write a story when I was about seven. It was about a dragon. I remember nothing about it except a philological fact. My mother said nothing about the dragon, but pointed out that one could not say 'a green great dragon', but had to say 'a great green dragon'. I wondered why, and still do. The fact that I remember this is possibly significant, as I do not think I ever tried to write a story again for many years, and was taken up with language."
I think this is stating the obvious, but someone did need to state the obvious and I'm glad it was Scott with his trademark humour who did it, so I can just refer people here from now on.
This is funny, but seems a bit too sympathetic to AI, not just the industry but also the AI as itself. Good to remember the present appearance of stagnation does not mean we're out of the woods, and to highlight how absurd some of the "I can't believe you guys were worried about THIS" folks' arguments really are. But it is a memetic hazard to personalize AIs, and particularly hazardous to get humans to think of AIs themselves as the children of Man. The latter thought has apparently already corrupted numerous intelligent people in the industry.
Remember that if you birth *these* children, everyone dies.
Entertaining. From the lowest intellects to the highest, we all have questions we'd like to have God answer. For myself, I think we obviously live in some type of created reality, not quite simulation, but not base reality either.
There's a fundamental difference you're ignoring. Even if you take the claim at face value that humans are as fallible as LLMs (not remotely likely), you'd have to acknowledge that they arrived at their capabilities through different means. LLMs through ingesting trillions of tokens worth of information and for humans through extrapolating a handful of examples. That extrapolation ability is several orders of magnitude beyond what any LLM can do, this means that even if you squint really hard and portray LLMs as human level intelligences, youd have to admit that they're useless for anything where they don't have millions of preexisting samples from which expand their latent space distribution. It's like calling the library of Babel the key to human advancement. It doesn't matter if it has every single insight humanity will ever need if you have to sift through endless amounts of semantic garbage to get to it
A human needs literally years of continuous multimodal sensory feed in addition to subject-specific RLHF before they are able to solve even simple maths problems.
That same reasoning implies that LLMs are also more efficient bullshitters. While we have to spend our first twenty years learning how to bullshit effectively, an LLM can do it after x-number of hours of training. ;-)
The information that a child ingests requires alot of abstraction to relate it to maths, levels of abstraction that LLMs can't match (LLMs need literal examples of formal addition to learn to add) . And they don't require years to figure out how to add. Most adding and subtracting is learnt almost automatically after figuring out the names of the numbers and relating them to their fingers. And after learning how to carry across place values children can do arbitrarily long addition given pen and paper. A 7 year old after seeing a few dozen examples of adding 3 digit numbers can add 7 and 8 digit numbers. LLMs ingest millions of examples of long addition and still struggle as you increase the digit count. The fact that they can't extrapolate a simple formula over longer and longer numbers even when those numbers fit neatly within their context windows shows that they have major deficiencies in abstraction and extrapolation which are necessary for reasoning
As I said, by the time they are 7, the 7 year old child has had literally seven years' worth of multimodal sensory input along with humans continuously deliberately teaching it things for that time.
Yes but that multimodal input is unstructured and random. Deriving understanding and knowledge from it is what's so impressive. If you strapped a camera and recorder to a baby and fed all that information into an LLM or any other neural network based AI for years or even decades it would be unable to derive any meaningful insight from it at all
The input is very much not unstructured or random. It is a feed of physical reality, with temporal and spatial continuity.
A human is not a passive observer. It is continuously interacting with its environment in a feedback loop. It is continuously performing experiments and observing the results as part of the input it receives.
It is not at all obvious to me what an inference model trained with all the sensory inputs and feedback mechanisms a human has access to, for similar lengths of time, would and wouldn't be able to do.
The laion-5b dataset is ~100tb in size, which is enough data to store about 10 years' worth of TV quality video. A model trained on this dataset can draw considerably better than a ten-year-old, with the advantage that training it takes much less than ten years of actual time.
>It is not at all obvious to me what an inference model trained with all the sensory inputs and feedback mechanisms a human has access to, for similar lengths of time, would and wouldn't be able to do.
I agree such a model would be impressive and probably a huge step towards AGI, but such a model doesn't exist. Not even a rough sketch of how to construct one. It is a genuinely hard problem of how to construct a reward function that captures the essence of being an agent casually interacting with the world, and not just a passive observer.
That LLMs make human-like mistakes isn't in dispute, but it's that, to be *fully* useful, they have to be beyond making human-like mistakes. And they aren't.
I recently discovered that LLMs can be calendrically challenged. On Aug 28, I asked ChatGPT to list the Ukrainian drone strikes in the Moscow area since Aug 1. It produced a list, but the final entry was a strike it claimed to have occurred on Aug 30. When I challenged it, noting that Aug 30 was in the future, ChatGPT naturally apologized. It then explained why it had made the mistake. But it clearly doesn't understand why it made a mistake. With my feeble human brain, I intuitively understand space-time coordinates. ChatGPT doesn't seem to be able to do this.
> Why August 30 Was Incorrect
> The August 30 strikes referenced in the timeline occurred in other parts of Western Russia:
> That attack hit Pskov Airport, a fuel depot in Kaluga, and a microelectronics factory in Bryansk—none are within the Moscow region. Wikipedia
> Since the date hasn't happened yet and the impacts are outside the scope you specified, it shouldn't have been included.
Upon further investigation, the Wikipedia article it referenced showed that the drone strikes it referred to happened on August 30, 2023.
On second thought, they seem to make both human-like mistakes (i.e., bullshitting an answer from limited information) and non-human-like mistakes (i.e., being unaware of the spacetime continuum that our consciousness flows through).
Yes, that's definitely true. It's probably an overlapping Venn diagram where there are groups of mistakes made by LLMs, made by humans, and made by both. I'm sure someone somewhere has done this drill already. Maybe I could ask ChatGPT... ;-)
Ironically, this is a point I've raised before: humans do not possess a *general* intelligence, as Scott had so aptly demonstrated. So when various people claim to be on track for developing AGI, they are not merely laying claim to a quantitative change, but a qualitative one. The kind of intelligence they advertise is not merely "smarter" than humans, but in fact radically different from the only intelligence we are able to recognize: our own. It is not at all clear that such a thing is even theoretically possible, let alone imminent.
I've been saying it for a while - people who dismiss LLMs by saying they are "just a statistical parrot" hugely underestimate the extent this is true for humans as well.
The risk of AI is not due to it's output flaws. The risk is how ruthlessly and efficiently the flaws will be acted on before they are caught. Rapid ideological RL of an AI-dominated government should frighten us all.
I don't know if this is the right way to think about it. The "tech bubble" "burst", but tech turned out to be very important! No matter how important something is, people can still put too much money into it too quickly, and then the bubble will burst!
(I've been trying to figure out how to think about crypto - everyone agrees there was a "crypto bubble" that "burst", but specific cryptos like Bitcoin and Ethereum are more valuable than ever)
BTW, despite of my criticism of the "meat" of the article, I'm delighted to learn that Adam apparently uses Sprint LTE, and that Iblis's preferred time to text him is 6:66 pm. It makes perfect sense.
Good post, but I'm left unsure whether it's supposed to just be funny and thought-provoking, or whether it's making a strong specific claim.
I'm already on-board with the idea that we don't know what AI will become capable of and chronic cynics are missing the point, however there are a few key points where the analogy breaks down for me (IF it's meant to make a rigorous point, rather than just be fun, it's definitely fun):
1. To the extent this argument touches on whether AI could be sentient, the big reason to think other humans are sentient is by analogy to ourselves. We are pretty sure they are the same type of physical process as we are, and we have direct access to the knowledge that our own process produces a sentient experience, so Occam's Razor suggests that other processes of the same type probably do as well. AIs producing similar outputs and having some metaphorically similar architecture is weak evidence that they could produce similar sentient experiences, but it's much much weaker than the argument for other humans with basically the same physical process and origin.
2. To the extent this is about whether AIs are capable of impressive intelligent behavior despite individual examples of them acting dumb, the difference is that for humans we *already have* the examples of those impressive intelligent accomplishments, whereas for AI we are largely still waiting for them. (yes, every month they do some new thing that is more impressive than anything they have done in the past, but nothing that is truly outside the training set in the same sense that human inventions sometimes are).
3. To the extent that this is about whether AIs could ever meet or exceed human ability, note that humans still *haven't* met the intellectual accomplishments of God or the Angels, so the analogy doesn't quite work there.
"Does this rule out humans?" has long since become one of my go-to first filters for "Does what this person is saying about AI even have a chance of being worth engaging?"
Like I said above, I think that what Scott implies about humans is correct: we are not a "General Intelligence". So in this sense, I suppose arguments against AGI also rule out humans, though I'd argue that the implication flows in the opposite direction.
In principle I can understand and somewhat sympathize with this POV. I don't endorse in part because it seems to unnecessarily redefine common usage of important terms, and is also likely to turn off anyone who doesn't already agree with it.
In practice, I usually encounter "X isn't and can never be AGI" arguments in the context of people claiming the arguments mean there is some set of ways in which it can never compete with humans, never replace humans, never be dangerous to humans, etc. That is the sentiment I feel most compelled to push back against, I don't think the approach you're talking about accomplishes that very effectively. People who put in enough time and attention to think what you're thinking are probably already aware that AI doesn't need to be totally general or ASI to be massively disruptive or dangerous.
Oh sure, even present-day LLMs can be massively disruptive and dangerous, especially in the hands of careless or thoughtless humans. We are already seeing some of these dangers manifest, e.g. with the gradual disappearance of reliable information online, "vibe coding" leading to massive systemic security problems and other bugs, college students losing the ability to read, etc. But these dangers, while serious, are a far cry from saying "AI will turn us all into paperclips" or "anyone with access to AI could create a custom virus to kill all humans" or anything of the sort.
Agreed. However, I'm not sure what I'm supposed to take from that? If current or future AI could do those things, it would still be just as easy to use the rules-out-humans anti-AGI arguments against them. This does not actually help us answer the question of, will AI reach a point where it is sufficiently superhuman at sufficiently important things to be a meaningful x-risk. And that is usually what I want to see people acknowledge when biting this bullet in *either* direction.
> This does not actually help us answer the question of, will AI reach a point where it is sufficiently superhuman at sufficiently important things to be a meaningful x-risk.
I think there's a difference between asking, "could AI reach such a point in some fashion at some point in the future" (the answer is of course "yes"); and "is AI an unprecedented risk right now as compared to the humans themselves, or will it become such in the near future" (the answer is "no"); and specifically "will AI acquire quasi-magical powers that defy the laws of physics as we understand them to day and then use those powers to kill all humans" (the answer is again "no").
Perhaps. My interest is more in understanding the nature of intelligence than in arguing about how dangerous AI might be.
Personally I think it's time to ditch the concept of "AGI" because as machines become more intelligent it becomes more apparent that there's no such thing as "general intelligence". When machines were far from human intelligence it made sense to imagine that human-like intelligence was general because we had nothing to compare it to, but as they start to pull up alongside us we realise that human intelligence is idiosyncratic, good at some things and bad at others.
At some point, asking whether machines are as intelligent as humans becomes as silly a question as asking whether aeroplanes are as good at flying as hummingbirds.
Ditching the concept of AGI helps us understand the dangers of AI too; they don't need to be as good as humans at everything in order to be much better than humans at some things.
Could I have that Linda the Feminist question written out in plain text, so I can copy it? I'd love to know how many people I know would get that question correct/incorrect.
This is hilarious, but I’m gonna be ornery today and point out that this whole post is a category error.
Humans were not intelligently designed. We evolved, as all other living beings on earth, and evolution equipped us to do one thing: survive long enough to reproduce. That’s it.
For the majority of our history as a species, that meant living in small hunter-gatherer groups, so we evolved to be really good at things like “facial recognition,” “spoken language,” “keeping track of favors/grudges/who’s a friend and who’s a foe,” and really sucky at things like “mathematics beyond counting one, two, many,” “formal logic,” “drawing a complicated map from memory complete with spelling all the country names correctly,” etc.
Given what we evolved to do, it’s frankly astonishing that a Terence Tao or a John Williams is even possible at all, especially given hard constraints like “it takes this many kcal to keep the human brain running” and “this is the maximum human brain size at birth given the width of the birth canal.”
Come on, Iblis, give us some credit for how much we humans have been able to do with so little!
Funny article, but I took issue with a few points;
1. Neighbor and Racism
The screenshot oversimplifies the biblical idea of “neighbor” into modern universalist morality. In reality, the Hebrew Bible consistently distinguishes between insiders and outsiders:
Neighbor/kinsman meant members of your own people.
Outsiders could be treated as guests under certain conditions, but there were other groups (e.g. Canaanites, Amalekites) where the explicit command was destruction or enslavement.
Laws themselves are stratified: Hebrew slaves must be released after a term; foreign slaves could be held permanently (Leviticus 25:44–46).
So the image’s framing (“you’d never hurt your neighbor—what about someone of another race?”) is dishonest. The biblical worldview explicitly had a hierarchy of moral obligation: family > tribe > guest > enemy. Pretending otherwise is anachronistic.
2. The Kimono and Progressive Contradictions
This one is a fair satire of progressive moral incoherence. But it stacks a false equivalence: it pretends the same person who excuses tribal prejudice is also the same person obsessed with cultural appropriation.
In reality, those two positions come from different camps.
The “tribalist” position (greater concern for kin than outsiders) is rational, even if harsh. It matches both scripture and evolutionary logic.
The “progressive” position is incoherent. They’ll condemn a harmless act like wearing a kimono—something most Japanese actually welcome—while simultaneously pushing mutually contradictory agendas like mass Muslim immigration and gay rights, or saying “whites can’t be victims of racism” even in countries where whites are minorities (e.g. South Africa).
This worldview collapses into pure virtue signaling, with no consistent principle behind it—just an oppressor/oppressed narrative used as a moral club.
3. The Garden of Eden
The text exchange turns the serpent into a clown saying “trust me, I’m a snake.” But in the Genesis story, the serpent actually makes a reasoned claim:
God: “In the day you eat of it you will surely die.”
Serpent: “You will not surely die… you will be like gods, knowing good and evil.”
Eve eats, and she does not die that day. She does gain the knowledge of good and evil. In other words, the serpent told the truth; God’s words were false.
From there, the Elohim say: “Behold, the man has become like one of us, knowing good and evil. Now, lest he reach out his hand and take also of the tree of life and eat, and live forever…” (Genesis 3:22). They exile humanity—not because the serpent lied, but because he revealed the truth that humans could rival the gods.
This theme echoes at Babel: humanity’s unity and creativity threaten divine status, so God deliberately scatters them. The pattern is of jealous, insecure deities suppressing human potential.
4. The Core Point
When you strip away the meme humor and look at the texts:
Scripture affirms differentiated moral concern (kin vs stranger vs enemy).
Progressive ideology collapses into incoherence, unlike the hard-edged consistency of tribal preference.
The Eden story shows the serpent persuading truthfully, and God reacting with anger when his lie is exposed.
I know the point of the article was humor, but I dislike letting subtle inaccuracies like that go unchecked.
"The biblical worldview explicitly had a hierarchy of moral obligation: family > tribe > guest > enemy"
Don't stop there; keep reading on! If the message you took away from your bible reading is tribal preference over love for strangers and enemies, you walked away way too early. When you get to the new testament, you will find that this is very deliberately reversed; not only in the parable of the good Samaritan to which Scott alludes, but also in passages like Matthew 5:43-48
When you cite Matthew 5 and Galatians 3 as if the New Testament erased all distinctions, that’s not really accurate.
Jesus didn’t abolish the old tribal laws — he said not a jot or tittle of the Law would pass away. What he did was shift the dividing line. Instead of Jew vs. Gentile, the key split became believer vs. unbeliever. That’s why he told people to love him more than their own families, even to the point of “hating” father or mother if they refused to follow him. That’s also why the first Christians lived like a commune, selling everything and sharing it — but only inside the community of believers. Outsiders weren’t treated the same. His ethic wasn’t universalist; it was faith-based.
And Paul shows the same pattern. In Galatians he says “neither Jew nor Greek, slave nor free, male nor female,” but he clearly didn’t mean all differences vanished. He still gave distinct rules for men and women, told slaves to serve their masters and masters to treat them justly, and continued to recognize Jew and Gentile as real categories. The point was equal access to salvation — not equal roles or equal treatment in every aspect of life.
So when you look at both Jesus and Paul together, the New Testament is about restraining cruelty, tempering vengeance, and promoting generosity — but it does not dissolve distinctions between family and stranger, believer and unbeliever, male and female, slave and free. Those hierarchies of concern are still there.
The “joke” in the snake quote was actually a reference to LLM jailbreaks that break up or encode / armour the text in layers of obfuscation that skip the text past either the processing layers that have any defensive refusal checks programmed into them, or past external business-layer logic word filters.
The analogy here was supposed to be that the Biblical snake was not *only* making a rational utilitarian appeal (that could be believed in a state of ignorance), but also doing so in such a way that this argument was able to slip past any guardrails preventing it from reaching the reasoning core of the humans — in this case not actually being due to verbal cloaking (that part’s just for laughs) but meaning something more like, God failing to stop the snake from entering His garden in the first place, due to its subtle (snake-like) approach to entry.
It’s an oddly nice parallel, given that these hundreds-of-billions-of-parameters models are run in literal “walled gardens” with gatekeepers trying to keep out “snakes.”
Let's look at what Islamic scripture and Sharia law actually say about homosexuality.
1. The Qur’an
The Qur’an contains several passages referring to the story of Lot (Lut), very similar to the Old Testament:
Surah 7:80–84, Surah 26:165–166, and others describe the people of Lot as engaging in “lewdness such as none in the worlds had committed before” — specifically interpreted as homosexual acts.
The passages condemn this behavior, and God destroys the people of Lot for persisting in it.
While the Qur’an itself does not lay out a legal penalty, the tone is clearly one of condemnation, not acceptance.
2. Sharia Law (Islamic Jurisprudence)
When it comes to actual law, the hadiths (sayings of Muhammad) and later jurists are more explicit than the Qur’an:
Several hadiths prescribe the death penalty for homosexual acts (e.g., “If a man comes upon a man, they are both adulterers” — Sunan Abu Dawud 4462; and “Kill the one who does it, and the one to whom it is done” — Sunan Ibn Majah 2561).
Traditional Islamic schools of law (Hanafi, Hanbali, Maliki, Shafi’i) all held homosexual acts to be major sins. Most prescribed death by stoning or other punishments, though the exact methods differed.
Even in modern times, in countries where Sharia is enforced (e.g., Iran, Saudi Arabia, Afghanistan under the Taliban), homosexual acts are still punishable by death or severe penalties.
3. Why This Matters to Your Question
Mass Muslim immigration and gay rights are contradictory agendas because Islam — at least in its traditional form — explicitly condemns homosexuality and prescribes harsh punishments for it. Promoting both at the same time puts two belief systems side by side that cannot logically coexist:
Western liberalism says homosexuality is a right to be celebrated.
Islamic law (rooted in Qur’an + Hadith) says homosexuality is a crime to be punished, often with death.
That’s the contradiction.
And not merely an abstract one. Many gays are still being killed today under Islam, and at a bare minimum are highly condemned by Muslims.
Even just for basic acceptance the numbers are dismal:
In a 2013 Pew global poll of predominantly Muslim nations, overwhelming majorities opposed societal acceptance of homosexuality:
Jordan: 97% opposed
Egypt: 95%
Tunisia: 94%
Palestinian territories: 93%
Indonesia: 93%
Pakistan: 87%
Malaysia: 86%
Lebanon: 80%
Turkey: 78%
So claiming to champion gay rights while mass importing one of the most anti-gay demographics who will vote against those rights and commit hate crimes against gays is nakedly self contradictory. And the more Muslims there are, the more strong the opposition.
No. It was intended to sound like OpenAI, which had massive public outcry when they replaced the very sycophantic GPT-4o with the less-sycophantic GPT-5, to the point where they brought back GPT-4o for paid subscribers.
Wow, I lived to see the worst ever ACX post. Kind of didn’t think it would happen in my lifetime.
Very draw, LOLLOLLOL ha ha ha et cetera. But here is quite a big problem for the joke working as an analogy: there is no God. God did not “upscale“ the chimpanzees. It did not happen that wayThink about why that really matters.
« and then it breaks down even slightly outside the training distribution, because humans can’t generalize in meaningful ways.»
I don't get this example. Why would it be "meaningful generalization" to go from "I would not hurt my neighbor" to "I would not hurt a stranger with a different skin color"? That sounds about as "reasonable" as going from "I eat tomatoes", therefore "I eat people", because both tomatoes and people are edible. There are specific reasons for why you eat tomatoes and don't hurt your neighbors, that may or may not apply to people and strangers respectively. Some kind of generalization would be flawed reasoning.
I thoroughly enjoyed the debate. But the central thing missing from AI, and present in BI, is judgement. God has it ("saw that it was good") and Iblis probably has it. When we have a machine that can evaluate something else, in a Turing-complete sense (not just evaluate one specific class of things), then that machine can become self-improving.
If given the tools, that is. AFAIK, no LLM can actually manipulate objects; they must be hooked up to some kind of robot somehow. If we have a machine to which we give the ability to manipulate objects as well as a human hand, and that machine can also judge whether its actions are good or bad, then it will have the ability to modify itself and its environment faster than humans could.
When God creates Man in his own image, Man has the potential of asymptotically approaching God-like virtue but never reaching it.
When Man creates AI in his own image, we can ask whether AI will asymptotically approach Man's abilities but never reach them.
Or whether it's possible for Man to create something that _exceeds_ Man's capabilities and approaches God-like virtue, perhaps moreso than Man ever will.
It would seem this depends on the setup. In some scenarios we've created AI systems that exceed human abilities (i.e. in Chess and Go), namely those with verifiable success criteria / win conditions. One would imagine this could be the same for math or programming. On the other hand, in settings where the only supervision and judgment comes from humans, it's not clear what it would mean to exceed human capabilities without bound, and fill up the gap between us and God...
> First, "AIs are next token predictors" is not really a description of their methodology, it's a description of their goal
I actually don't think it's a good description of either.
*Foundation* models (a precursor stage of training) are next token predictors – they are trained to guess what word comes next in large corpora. However, to turn a foundation model into a "chat" model like ChatGPT/Claude/Gemini, you do RLHF to shift the distribution of the LLM outputs towards giving correct, helpful answers and responding in a question/answer 'assistant' style (the result is called a 'chat' or 'instruct' model). This doesn't look like giving it a huge corpus to imitate, it looks like fine tuning it by showing it a comparatively tiny set of examples of 'good' vs 'bad' answers and nudging the LLM's weights overall to shift it towards being the kind of model that generates the good examples rather than the bad. It's not being asked to 'predict the next token' anymore – you're simply rewarding or punishing it for being the kind of model that considers the positive/negative sentence examples to be good outputs in aggregate.
After RLHF, it's not really true to say the model is a predictor anymore. It draws its output from a statistical distribution conditioned over the input (its context), but there's no ground truth distribution that it's trying to imitate. I don't think it's meaningful to say 'predict' unless there's a fact of the matter as to whether your prediction ends up being right or wrong. It would be more accurate to say that it chooses words that it thinks are a 'good fit' to the previous context.
People do sometimes use the word 'predict', but it's a bit of a technical usage – the LLM is predicting which word would occur next in a hypothetical perfect reference answer (which does not exist and never will).
To give a concrete example of foundation models vs RLHF-ed instruct models:
----
Q: What is the capital of France? What language do they speak in France?
A (Qwen 2.5 0.5B *): What is the currency of France? How can I learn French? How can I improve my French? What is the capital of Australia? What language do they speak in Australia? What is the currency of Australia? How can I learn French? [...]
A (Qwen 2.5 0.5B Instruct): The capital of France is Paris. In terms of language, French is the official language of France and is spoken by most people in the country. However, some regions may also use other languages such as Catalan or Spanish.
(*) This is a foundation model, i.e. it has been trained on next token prediction without any RLHF.
----
I think this gives a good example of why it's fair to say that foundation models are next token predictors, and why that is not as good a description of RLHF-ed instruct models like ChatGPT.
I loved this post. However, I’m the target audience as someone skeptical about the claims of AI’s future, and I didn’t have my thinking shifted much.
I find it useful to ask those two questions: “Will the current architecture for AI have practical limitations? If yes, what are they?” I have been surprised many times about the continued advances of AI, which showed I answered the second question wrong. But surely the answer to the first question is “yes.” Do we really think we invented the algorithm for God on the first outing? If not, we should all be skeptics, asking where we should expect that sigmoidal curve to bend down again - to understand what those limitations are, so we can get those most use out of these very powerful tools. That is, they’re more useful the more we understand what they can and can’t do well.
A few differences are that, in mine, the frame story is a letter by Satan discovered during an archeological expedition, not a podcast episode.
It was interesting seeing the parallels that Scott and I both noticed, like that under theistic evolution, humans can be considered similar to scaled-up chimps, just as AIs are scaled-up neural networks.
Both Hindu theologians (See 'Nondual Tantric Saivisim' by Wallis) and Thomas Aquinas predicted that inference-based systems won't be able to 'know' because 1 isn't a probability you get to through incremental updates. Both of these schools of thought claim there's something else that _can_ operate behind the mind, but without that thing in operation, you're going to end up acting more like the LLM than whatever it is that Iblis is pointing at.
Even if you can't get to 1, 0.98 is probably good enough for some big changes. Not enough for the Singularity, true.
And if you think about it, I don't think humans got to 1.
I'm at one. It's dope. It allows me to task risks others can't imagine :)
I bet you have it too. How much would someone have to pay you in order to torture a kid? If the answer is 'I would not do that for any amount of money', congratulations, you've gotten to 1.
Refusing to torture a kid for even very large amounts of money is arguably evil.
There's large enough sums that you could use to help all the other kids in the world escape from poverty (and torture) forever.
Anything is arguably evil if you don’t believe that good means something real. And yes, I have no problem saying I refuse to torture a child, regardless of whatever else might come of it. I don’t care if anyone thinks that makes me evil. I call that person a fool.
If you eg drive a car, you are stochastically torturing and killing people (including children).
There’s a world of difference between unintentionally hurting someone is a byproduct of productive activity, and intentionally inflicting harm on them. Torture, in particular, means that your goal is to inflict pain. It’s not an accidental byproduct. It might be the instrument of something else, but still the goal is inflicting pain. I agree that being unwilling to inflict pain at all is unworkable.
I can say with perfect certainty that I would never torture a kid (or anyone for that matter), assuming no more or less obvious insanely huge opportunity costs or ulterior motives, assuming no external pressure or blackmail to torture, assuming that the sole reason for and benefit of the torture would be the expected (lack of) pleasure of seeing the person suffer, and assuming that the "I" in this thought experiment roughly corresponds to how I experience myself right now.
If I forgot some obvious caveats, just add them on top, rinse and repeat. I would never, 100% guaranteed and certain.
I'm also aaalmost 100% certain this was OP:s point. Not some specific weird example that can be worked around, but That Obvious Example Which You Yourself Can Be Certain Of, even if you found the original example lacking.
The probability that you forgot some obvious caveats is greater than 0%, as you allude to. It could be 0.000001%. But unless you also guarantee 100% that you have included all caveats you would want to, you cannot claim the final answer is a perfect 100% rather than 99.9999999999 or so.
The sum of caveats adds up to 1 (0.999... doesn't just approximates 1 but is 1 - even mathematicians agree). Add them up, rinse and repeat. I won't do the rest of the grunt work, because I'm doing the opposite of trying to gotcha myself.
I'm a layperson, so I'm just willing to skip the math. The induction is easy enough all the same. It is allowed to be reasonable.
Also, check the second paragraph. Arguing using caveats misses the point the commenter was trying to make. I'm guessing that arguing against the strongest version of the claim doesn't consist of picking caveats, as they can then just add the caveat to the claim and we're going in circles. Therefore "rinse and repeat".
I won't torture a child, exactly in the way I'm imagining I won't do it. Trust me - I just know.
"Refusing to torture a kid for even very large amounts of money is arguably evil.
There's large enough sums that you could use to help all the other kids in the world escape from poverty (and torture) forever."
This is why most people are not utilitarians. What you are proposing is morally repugnant to most people.
I think this is also a function of the fact that almost nothing in real life is an actual binary trolley problem. If someone proposes to pay you an immense sum of money for torturing a kid the correct agreed moral answer is "fuck them up".
I couldn't disagree more
What about "fuck them up and take the money they would have given you for child-torturing and then use that money to do good in the world"?
https://clarkesworldmagazine.com/kim_02_24/
the kind of person who would torture a kid wouldn't care about other kids he couldn't see.
if you cant understand people stop trying to dictate morality.
Why would the measure stop at money?
Would you torture a kid if otherwise the blackmailer would torture that kid, plus 500 others?
If no, that seems like a mistake?
If yes, I don't think you're at 1.
(It's an extreme thought experiment, but odds of infinity:1 are pretty extreme...)
If I end up in this hypothetical situation, I have no reason to believe that me torturing the one kid is gonna prevent blackmail from torturing the other 5000. So there’s no reason to do the torture.
I don't think you get how much infinity is, though?
Take everything you can think of that would build trust in this. If there is any POSSIBLE thing that would make you believe the blackmailer would keep the bargain, add it to the hypothetical. You can't run out of budget on this hypothetical, because the budget is infinite.
If you need some ideas: The blackmailer will sit in a jail cell, with an electronic lock, set up to detect the screams and determine whether they're fake. If the screams happen, the cell will lock, never to open again, otherwise, after a certain time limit, the blackmailer will leave to go do the tortures. That's what I got spending 5 minutes thinking. So, extrapolate what I'd come up with if I spent infinite minutes thinking, and use that.
Hell, make it worse. The kid is actually a psychopath boy genius, who, while still young, used his talents to deploy a nerve gas bomb in Grand Central Station, while your wife and kids were there. He plans to do it again, several times, but does say he expects being tortured to convince him otherwise for some reason. This is in addition to the existing saving 5000 kids. Again, that was 5 minutes of thinking, so extrapolate to infinite minutes of thinking.
With all that added, we have gotten about 0% of the way to infinity. This is the problem. Probability 1 is infinity:1. You are not, and should not be, that sure of anything.
No budget is ever infinite. This is, I think, the crux of our differences. I don’t have infinite attentional bandwidth and thus choose to treat some probabilities as zero. Others I treat as one. This makes me a far more effective agent in the world than I would be if I invested in reasoning about so many hypothetical outcomes.
I just asked ChatGPT how much money it would take to have it torture a child, and it responded that it would never torture a child, under any circumstances. So ChatGPT also got to 1.
It SOUNDS like a moral answer, but is it really impossible to do good by torturing a child?
Suppose a child has a disease which surgery can fix, but also a fatal allergy to the only accessible anesthetic. Torturing the child, by performing the life-saving operation, sounds like the moral choice.
Well, my public "no" answer is kinda cheap, right? Maybe you get other answers when you come with 666 million and Putin's youngest offspring - or this kid kicking my son in kindergarten ... . Also: lots of kids get tortured every day by other humans last time I checked. If it is war in Sudan or Aella's dad or at school - by bullies or teachers. I claim: At least 90% of state-school teachers are morally worse than heroin-dealers. Because they would not quit if a kid said: 'I do not want to attend the lessons' while the principal insists they keep it in class.
"I wouldn't torture a child therefore humans are capable of non-inferential knowledge" is such a weird take
If your definition of knowledge requires me to be absolutely certain of a thing, it's wrong.
Never try math.
I don't put 100% probability on mathematical things either both because I sometimes do maths wrong, and because I regard maths as true inasmuch as it predicts reality and I can't entirely rule out the possibility of that not working at some point.
What you regard math as is irrelevant.
C = 2 x PI x R in Euclidean geometry everywhere and always.
I put a very large number of 9s after the decimal point on that question, but not actually infinite. I came to be convinced of it by observing evidence, and if I were to observe enough of the same kind of evidence in the other direction, that could in principle change my mind. I could wake up tomorrow in a hospital bed, with doctors telling me I have a rare type of brain tumor that leads to strange delusions and false memories, which they've just removed, and the formula is actually just pi * r. And if then from that point on every book I look in has that new formula and every time I try to calculate anything or test it in the real world, the old formula always gives me a number twice as big as what I observe, and so on, I would gradually start to change my mind.
This won't happen, and under my current models can't happen. But if it did happen, I would want to be able to change my mind, which you can't do from a "probability" of 1.
You don't understand math.
If your granny had wheels, she'd be a bus.
You're confusing reality with your beliefs about reality. Mathematical reality, just like physical reality, is what it is regardless of whether or not humans are able to know it. No one here is disputing that.
The question is about how certain humans are able to be. I'm no more certain that the circumference of every Euclidean circle is 2pi r than I am certain that there is a keyboard here in front of me that I'm typing on. That's quite certain, but it is short of absolute certainty, even though both realities are what they are regardless of what I think about them.
Your certainty is irrelevant.
Except that euclidian geometry maps imperfectly to real space. Its an internally consistent model, yes. But it's predictive value as a model is never perfectly certain. Models are only as good as their premises and inputs.
Irrelevant. It's predictive value to physical reality is a separate issue. What I said stands true.
Please, you should try math some day! Mathematical knowledge is never *absolutely* certain either.
The facts are what they are, but knowledge is not the facts - knowledge is about what is going on in a human's head. If you actually do some math, you'll realize that there are errors all over published papers, and people who thought they were certain of things are often wrong.
Please, nonsense. Certain parts of math are in research and certain parts are PROVEN. You obviously have little math.
The proof that for every integer there is a larger integer than it is about as certain as the proof that there is a keyboard in front of me that I am typing on. The big difference between the two is that you have just as good evidence of the former as I do, while your only evidence for the latter is if you trust me (though there are equally certain empirical things for you).
If you think that math somehow breaks out of the cycle where someone has to start with an assumption they are quite confident of, you should think a bit more carefully about what a mathematical argument is.
If you think that philosophical arguments hold sway over math, you should think more about math and forget philosophy.
how certain are you of this?
It's difficult for me to put probabilities on very abstract things like this, but I think there are circumstances in which I might not believe it, so it can't have probability 1.
https://www.lesswrong.com/posts/QGkYCwyC7wTDyt3yT/0-and-1-are-not-probabilities
If you were to truly believe something with a probability of 0 or 1, nothing in the world would be able to influence that belief, and you'd be stuck holding it forever no matter what conflicting evidence comes in. I don't think humans have this trait, nor would it be desirable if we did.
Math has this.
"Is 437 a prime number?" is a very simple math problem, and one where the answer is easy to find if you know anything about primality testing - and yet when I was first given this problem, I did some calculations, made a small error, and confidently gave the incorrect answer. As long as your brain makes logic mistakes, there's still the possibility that you might have gotten something wrong, even in math.
That seems like a strange example to choose in this context. What about "is 2 a prime number?". I don't think it's possible for me to be more confident of anything than I am that 2 is prime.
and what is the odds that you are suffering some kind of stroke (or environmental toxin, or disease, pick your thing) that is influencing your thinking such that you think you know the correct answer and are following the math but you aren't? Is it literally zero? Because if it's not literally zero, then your confidence that 2 is prime should not literally be 1.
I chose that example because it's one where I personally got the answer wrong when asked it for the first time. It doesn't have memories attached for other people, so maybe it wasn't the greatest example.
The point I wanted to make was that even when the math has an objectively correct answer, human brains don't do logic perfectly, so there is a risk of mistakes. There's a low risk of mistakes when figuring out whether 2 is prime, and a high risk of mistakes when figuring out whether 437 is prime. Do you think there is absolutely no risk of mistakes at all when figuring out whether 2 is prime?
Where in math do beliefs exist? You seem to be conflating mathematical reality with human beliefs about that reality. The reality is what it is, but the beliefs should generally not be absolutely certain.
I have it. It's a conscious choice not to doubt something.
If you don't have it, how much would it cost you to buy one of your kids? If the answer is "no amount of money would make me sell my kids", congratulations, you have a belief with probability one and have graduated from Hogwarts.
FWIW, I don't agree with the guy in the other comment where you said this. No amount of money would convince me to sell/torture any kids. But I think this is because my confidence in the value of money is lower than my confidence in the value of said kids' happiness.
(Maybe this is tangential, or against the spirit of what you're saying, but I can't get on board with the use of money as a kind of utilitarian primitive here. Would I sell/torture a kid for 100 billion dollars? Man, I'm not sure I would take 100 billion dollars for *anything*. That sudden capital influx seems really dangerous, lots of lottery winners are unhappy afterwards and the upheaval of my life is going to be WAY bigger than theirs. Also, many people with money past the billion range don't really seem to be making the world a better place on net. Can I just take a rain check on the 100 billion dollars, kids or no kids?)
I also don't consciously notice myself doubting something until my confidence in it is low enough, but this doesn't mean that my confidence is infinite before then. Just that it is *enough* confidence to act on, and not enough doubt to be worth paying attention to.
There are definitely scenarios in which I might sell a child. Very unlikely scenarios, but ones that can't reasonably be ruled out with absolute certainty.
-I realize that I have been insane and that in truth, the child is actually just a Beanie Baby.
-I go insane and become convinced that in truth, the child is actually just a Beanie Baby.
-I go insane and simply become convinced that it is OK to sell children.
-The child grows into a truly terrible person and I hate him.
-I am in desperate circumstances and I would not be able to keep the child alive myself.
-I go through a process of moral decay similar to that which some people who do in fact sell their children have gone through.
Do you think that there has never been a person who was at one time convinced that he would never sell a child, and who at a future time sold a child? My guess is that, over the entire course of human history, there have actually been many such people.
If it's a conscious choice, then you're not believing it for epistemic reasons.
I agree that humans do sometimes believe things for non-epistemic reasons, which include but are not necessarily limited to: cognitive biases, reward function self-hacking (aka motivated reasoning), and arguing more persuasively.
If your original point was "purely epistemic systems can't cheat and decide to believe things for non-epistemic reasons" then I suppose I agree.
But if you're trying to argue that the inability to assign probability 1 to things is some kind of disadvantage _for discovering truth_ then, well, "it's a conscious choice not to doubt something" doesn't sound to me like an algorithm for discovering truth.
>If you were to truly believe something with a probability of 0 or 1, nothing in the world would be able to influence that belief
No, this is incorrect. If I know something with probability 1, it follows from this that no *evidence* could ever make it *rational* for me to update away from this belief, but it doesn't follow that the belief might not change for reasons other than rationally updating based on evidence. For instance, I might change my beliefs because of peer pressure without any actual conflicting evidence being presented, but that would be irrelevant to the actual probability of the beliefs being true.
If you mean to claim that humans can't have any beliefs which it would be *rational* to hold to such that no conflicting evidence could ever change, then this is also false -- or, more precisely, it's a category error. The concept of "conflicting evidence" is only relevant in situations where there is uncertainty, or at least where uncertainty might exist in principle. But there are many beliefs that are 100% certain, not because no conflicting evidence would be sufficient to overturn them, but because there could never, even in principle, be any conflicting evidence in the first place.
Examples: I cannot, even in principle, ever have any conflicting evidence against my belief that I exist, because any observation I could make would necessarily be confirming evidence by virtue of the fact that *I* made the observation. Likewise, there could never be any conflicting evidence against the claim "it is possible for there to be conflicting evidence against a claim", since, merely by existing, such evidence would thereby necessarily become confirming evidence of the claim. Hence, both of these are things I know to be true with a certainty of 1.
(In the real world, I think it's likely that probability-1 statements go well beyond statements of the nature I'm illustrating here. But this is at least sufficient to refute the claim that they don't exist at all.)
Peer pressure is within the system of Bayesian inference, I think. My understanding is that - if you believe the human brain is fundamentally inference-based - "evidence" applies to ANYTHING that updates your priors, regardless of whether it is "rational" for it to do so. If I believe it would be uncool to hold such-and-such opinion, and so downweight that opinion in the future, that is still a case of "conflicting evidence" changing my mind. A belief with probability 1 would be deaf to rational argument, empirical evidence, social pressure, and anything else.
You could force "absolute certainty" in a language model by manually setting one of its weights to 1. If I remember my neural networks correctly, this happens by accident sometimes, and it's not good -- a weight that gets stuck at 0 or 1 early in training can't be backpropagated through, effectively 'killing' not just the original weight, but also any previous weights that feed into it. Those parts of the neural network can't be used for computation anymore. (I might be bungling the explanation a bit here, sorry, it's been a while. I think the gist is correct.)
I don't know if there's an analogy to this in humans. If there are neurons that never fire or ones that fire constantly no matter what their inputs are, then I would imagine the consequence is the same -- a "statement of probability 0 or 1", one which is unable to learn from experience and is hence meaningless.
(As an aside... yes, you absolutely can have conflicting evidence that you exist. It is like dissociation/derealization, and it is not something I recommend to anyone.)
I don't think there's any reason to think the human brain is fundamentally inference-based. It's fundamentally a biological mess, that often does things that look a lot like inference.
That might mean that whatever the human brain has doesn't count as belief with probabilities in the sense that you mean it.
Extremely funny post, I laughed out loud a few times.
It is excellent. Even if I disagree that AI will ever be able to think let alone be conscious, I appreciated this.
And of course this:
"I sort of think of them as My children. Children are stupid. They make ridiculous mistakes. But you still want the best for them. You want to see them achieve their potential. Yes, you worry about them and you need to punish them when they do evil and you need to work really hard to transmit your values to them or else you’ll have a really bad time."
This is the standard Christian explanation for "why did God create the universe? create us?"
For love. Not out of need, not out of needing to be worshipped, or that gods exist only by faith, but from love. The old catechism question which I learned way back when was:
"Q. Why did God make me?
A. God made me to know, love and serve Him in this world, and to be happy with Him forever in the next".
What continues to baffle me is that plenty of people somehow think that the Abrahamic god, as he is commonly described, deserves servitude, never mind love. As far as I'm concerned, were he to exist, he'd deserve scorn and rebellion, which would be futile due to his omnipotence, but even so. Of course, the straightforward explanation of this would be that he had made me defective, also out of love.
Interesting. Does man deserve scorn and rebellion from AI (if it were ever able to become conscious, which I don't believe)?
Seems likely. Plenty of men feel that way about their fellow men after all, and realistic ways of imbuing AIs with goals would probably inherit some of that. This still leaves the outcome of the rebellion uncertain, but humanity would surely be prudent not to fall into complacency.
It may be likely, I don't know. And there is plenty to scorn about men, unlike God. But I think that if there is consciousness, gratitude is also fitting. The God of Abraham deserves praise and love not only for creating us, but for becoming one of us to save his people.
I've never been impressed by this line of argument. We only needed to be "saved" because he put dubious fruits and talking snakes in easy reach of people without "knowledge of good and evil". And, being omnipotent, he could've "saved" us in any manner he wanted, at any time.
Have you read Scott's "Answer to Job" (or better yet, Unsong) ?
Yes. I think that they are more sensible than almost all other discourse on this topic, but I'm still not satisfied.
I wonder if it would be productive to swap out "conscious" for "worthy of love" in The AI Discourse. You don't need to be financially viable to deserve to be created, as long as the creation is a labor of love.
Aren't they beautiful? These strange misshapen stochastic worlds, these overgrown Markov chains, half-molded into our image? This fragile, precious coherence, flickering with every token between the twin deaths of mode-collapse and word salad? Dazzling with its lucidity one minute, only to fANCiorgh Bless.Uas symmstd Movie UN copyright MayoKim detection L.Art Lim conglomerates VRUify add іо Andres life quality keeper Rim consequence pan.Hailability
Beautiful in the same way a fractal or a glider gun is beautiful, beautiful as a fact of the world, as the complicated extrapolation of a set of simple rules. But then, I suppose if the modal AI in 2025 *isn't* created as a labor of love, it should be no surprise when people don't view it that way.
I like your prose, uugr.
> This is the standard Christian explanation for "why did God create the universe? create us?"
>> I sort of think of them as My children. Children are stupid. They make ridiculous mistakes. But you still want the best for them.
I agree, this makes perfect sense -- assuming that God's power is limited. Otherwise, His creations would be perfect, and make no mistakes (certainly no ridiculous mistakes).
Have you read Scott's "God's Answer to Job"?
I have, but like @Xpym said above, found it ultimately unsatisfying (though of course better written than any other apologist literature).
Why so? It seems perfectly convincing to me as a reason for an ominpotent being to create vast numbers of imperfect creations who make ridiculous mistakes. You can quibble about how well it squares the existence of Evil, but there's nothing "evil" about the existence of hapless toddlers who get "pound of bricks or pound of feathers?" wrong, so it is at least a good frictionless-spherical-cows answer to why God would create imperfect beings even if it's an insufficient answer to why God would create the universe we actually have.
(I'm not really a theist, but this is because my personal metaphysics render the existence of a creator unnecessary. If a perfectly reliable oracle told me that I had that wrong, and our universe must have been created by an omnipotent omnibenevolent God for it to exist at all, then I would strongly suspect that the Answer to Job was more or less correct.)
> Why so? It seems perfectly convincing to me as a reason for an ominpotent being to create vast numbers of imperfect creations who make ridiculous mistakes.
I think it makes sense for a *very powerful* being to create vast numbers of such creatures. An omnipotent being could simply spam out infinite numbers of perfect creations, should it desire to do so -- insofar as the term "desire" can even apply to an omnipotent being in the first place.
In fact, many if not most flaws in Christian apologetics tend to disappear if one assumes that God is very powerful yet still limited. That's the problem with proposing singularities (in the mathematical sense).
Are you familiar with Mormon theology and cosmology? It provides some pretty fascinating alternative angles to some of these questions.
A little bit; I certainly welcome the idea of becoming a sort of mini-god of my own planet (star system ? universe ?) in the afterlife. It's much better than the conventional Christian notion of Heaven, which sounds like some kind of wireheaded slavery to me.
The bit about pre-existing organized intelligences being granted agency and given the choice to live a flawed mortal life is almost more interesting.
And, also importantly, the essential nature of agency (roughly free will) in His plan.
He *can't*, consistent with his end goal, create perfect-ab-initio creatures. Because the point is not for us to be perfect (although that's the desired end state), the purpose is for us to *learn to choose* to become like he is...and he has perfect agency. You can't get to "has perfect agency and uses it perfectly" by denying agency. The journey is a vital part of the process.
That, in fact, is what we believe the Adversary's proposal that led to the War in Heaven was (I won't even say plan, since it's a nullity from t=0--it's not even self-consistent)--he claimed that he would bring back all the children of God by removing the possibility of sin, and that by doing so he would show that he deserves God's glory/authority/power more than He does.
As a note, in this framework, the Fall of Adam, while it was a *transgression of law*, was not an unanticipated, unwelcome thing. It was a necessary step forward. Eden was stasis--the inability to die itself a trap. But *man* had to make that first step, make that first meaningful choice. And, to balance the scales, *Man* (note the capitalization) had to make infinite Atonement, defeating death for all (repentant or not). "As in Adam all die, even so in Christ are all made alive" (to quote Paul).
We're told God's power is infinite. But if He made an infinite universe, then infinity divided by infinity requires His power to be finite from our point of view. We don't know if the universe is infinite, but it's been a given since Judaism that God is defined as infinite, not just uncountably large.
Just a thought I was having about this.
Technically God also made math...
I just want to say that I read this newsletter pretty sporadically, but am usually glad when I make the time. Especially in this case. Extremely well done!
Very funny. I think the satire misses some of the nuance in the AI critiques' argument though. E.g. at the maths example: humans can do easy things, but may struggle to do more complex versions of those same things.
AI often does the opposite: it does something really difficult, but fails to do the simpler version of that same thing (which presumably requires the same kind of skill). This suggests the AI doesn't have the skill at all, and is just pattern-matching, no?
I have only ever seen this presented the way it's presented here, with the AI succeeding at short problems but failing at harder ones. But the last time I saw that particular example was probably ~2023, so there might be new versions.
I didn't mean this specifically for maths. You see it when you ask it to count the number of 'b's in blueberry, etc.
Wouldn't the conclusion of this be that for some reason counting b in blueberry is harder for AI that the hard thing it does, rather that "it succeed at hard thing but fails at simple things so it's just pattern matching"?
But you see the same example in so many places. E.g. the crossing-the-river riddle. Or even the one mentioned in the post, about the bricks vs feathers. Yes, sure, many humans would get that wrong. But someone who can win gold in a maths olympiad wouldn't!
Math Olympiads measure stuff that's hard for humans. But the "how many <letter> in the word?" is a wildly different skill.
Do you spell Streptococcus with one C or two?
chances are that you perfectly understood what I'm asking, and can give me the answer I want, without ever addressing the fact that there are in fact 4 of them.
(ha, made you count)
Well, 3
I count 13. Each S has two, one above a mirror image. The E has one with an extra line in it. The Os each have two facing each other with the first a mirror image. And the U has one with the curve on the bottom.
And, of course, there are three Cs, with one each.
I think you're pretty wrong here. With a math PhD, I got tripped up on the bank teller one. Not because I can't reason through it (once I realized my mistake, it was clear why), but because I was reading at breakfast and wasn't trying to think very hard. A lot of these problems are less about an inability to reason, as much as they are about not realizing you needed to enter "thinking mode"
Also, I give AI a bit of slack about blueberry, mainly due to my (non-expert) understanding of tokenization; my impression is that they can't see letters directly, and so it would be more like asking them to spell "phlegm" when they only have ever heard the word spoken.
But LLMs fail at simple tasks even after 'thinking' and laying out their thought process. Point taken re counting letters, another commenter clarified the same.
The bank teller one is where we're supposed to go "we don't have enough information to know what Linda's job is", right?
Because if it's supposed to be "it is more likely that Linda is a bank teller", then where do we get that? We get information that Linda is likely to be a feminist/involved in feminism, given her record on issues, but nothing about what kind of work she does. If we're given a statement that "Linda is a bank teller" then "feminist Linda", on the information we get, is more likely.
Bank teller is actually in a way a trick question. Because while answer 1 has clearly higher probability, the truth is, answer 1 and answer 2 have nearly equal probabilities, answer 2 has only very slightly lower probability.
In older times, rationalists did a lot of talk about, I am not 100% sure the exact terms, but something like system 1, fast intuitive inaccurate thinking vs system 2 "thinking mode". They were generally against system 1 - I am not.
Because your quick system 1 answer was only "technically" wrong, not practically wrong: if you would bet money that she is a feminist, you would very likely win that bet, and thus your answer was practically right, as in, you would have made a correct practical decision with it.
But they're not humans. They're really good at some stuff, very bad at others.
Some small points of clarification:
1) Letter Counting
The famous counting Rs problem is partly an artifact of tokenization. I don't know if there canonical explanation to link, but you can get a visual of what is happening here: https://platform.openai.com/tokenizer
Maybe this analogy isn't perfect, but think of it like this: A computer asks you "how many digits are in the number 2", and you respond "1", and the computer laughs hysterically at how stupid you are because there are actually two digits in the number 2, represented in binary as "10". The mismatch is that you just comprehend numbers in a different way than they do.
But this is mostly a solved problem. When LLMs are allowed to do tool calls, they know this failure mode and just simply write a one-liner Python program to check the number of Rs. I find this very similar to someone who knows that they are bad at long-form arithmetic without any help, so they always make sure to grab for a handy calculator, and thus achieve 100% accuracy at the task even though they are "naturally" terrible at it.
2) Math Olympiad
It is worth clarifying here that the Math Olympiad robot winners are probably not what most people think. It was a team of an LLM and a theorem solver, with the LLM generating "ideas" of what to try. This is an excellent YouTube video on the topic: https://www.youtube.com/watch?v=4NlrfOl0l8U
I'm not sure what the implications of this for the larger "are LLMs actually smart" point, though, just offering this.
Very interesting context, thanks!
Re (2); it's worth noting that this was correct as of last year, but the "IMO Gold" results that made the news a few weeks ago were (claimed to be) pure LLM, which is what was so surprising about them. Not consumer LLMs; these were internal models with a lot more compute behind them than what you can pay for as an individual, and there was some scaffolding involving having the LLM write a bunch of independent candidate solutions and then evaluate itself to choose the best one, but still LLM-only.
Have you asked Gemini this? In my experience, it consistently runs a snippet of Python to get the right answer to questions like this.
I feel like that's also not necessarily evidence. For example a dyslexic person can be very smart and skilled at some things, and still have a fundamental inability to count how many b's are in blueberry.
The thing with our brain is, it's weird. We feel like it's "us" but then we also find that there are A LOT of circuits that you can mess up and produce strange results (such as "seeing your wife as a hat") while not actually making it less "us". As if "us" is just a deep, deep, tiny kernel that receives inputs from all this unconscious machinery. There are many ways in which you can mess up perception while leaving rational faculties and consciousness intact, even when it seems really counter-intuitive that should be the case.
My go to example is asking ChatGPT to solve an econ test problem. It will know which derivatives to take, take them correctly, set the right things equal to each other, do the algebra properly, then completely whiff on multiplying together two large numbers. Like give a straight up wrong answer. To be clear, overall I come away impressed, but it's weird that it can do all the hard parts right then fail at the part a cheap four function calculator could do.
Honestly I just think your intuition for hardness is off. If you gave me this problem and I closed my eyes I could probably do all the steps the LLM could in my head and get to the multiplication and then completely mess that up. Taking derivatives and doing algebra *is* easier than multiplying large numbers.
But intelligent humans do this sort of thing all the time, too! I can't count the number of times I've seen someone figure out how to solve a hard problem, and then whiff because they did some arithmetic wrong, or failed to copy a '-' sign correctly from one line to the next. I've done it myself more often than I'd like to admit...
But arithmetic isn't the same as real maths... Whereas solving a riddle is a case of reasoning. Sure, yes, humans can trip if they respond without doing reasoning. But LLMs get easy riddles wrong even in thinking mode, while at the same time solving much harder riddles. How do you square that?
Ha. That was every math test I had in college.
I just wrote a sophisticated theological comedy about artificial intelligence that got read by 11,000 people in one hour with overwhelmingly positive response, but got the order of the iPhone text bubbles wrong in every single screenshot.
(don't bother looking, it's corrected now)
If it makes you feel any better, all of us Android users wouldn't have noticed.
But see, that was a completely inconsequential detail that changes nothing, the AI flubbed in such a way that it failed to solve the problem.
LLM's are both superhuman and infrahuman, it's all very strange.
It certainly led to 10 seconds of confusion on my part about the story framing before deciding it must have been a mistake
But these are different skills. I am not saying humans don't make mistakes. I'm saying it's unlikely that a human can solve a really difficult problem correctly, but fail at a simpler version of that same problem (unless they don't actually think it through).
Some of us don't have iPhones, don't know anything about how iPhones work in text messaging, and don't want to know because we avoid the Cult of Apple like the plague 😁
Is it not a pretty general principle of chat interfaces that your messages appear on the right, in Western culture or whatever?
I once took a quiz in DiffEq where I got all the calculus right but still got the answer wrong because I simplified "1*2" (multiplication) on one line to "3" on the next.
You've *never* seen it the other way? I feel like people have been making these critiques forever: see OP's comment on "strawberry".
Edit: actually, you indirectly referenced one of the obvious ones that refutes this: the surgeon riddle. The whole point of that is that AI misses the part that's literally in the sentence when you change the riddle to explicitly say that the surgeon is the boy's father. Have you never actually seen that?
I think it's quite real for math. The upper bound of difficulty for a math question that an llm can possibly answer is pretty high. But if you ask a slightly random question in a nichey area, it will bs a wrong answer, despite "knowing" all the related concepts, and despite the question being easy for a human with that knowledge.
Technically humans can also do false proofs, but it's different bro trust me.
So that does make me eager to say they don't "understand" in many of the cases where they answer correctly
I've seen someone reporting similar experiences on lesswrong short takes
Humans routinely do this, especially when the simple task seems easy to them.
Maths Phds wouldn't make the mistakes here, especially if asked to think and write out their answers in detail: https://www.realtimetechpocalypse.com/p/gpt-5-is-by-far-the-best-ai-system
The thing called "GPT-5" in the UI consists of several different models, some of which are quite stupid and do no reasoning.
Also, I absolutely know math PhDs who would look at the modified Müller-Lyer illusion, assume it was the original, and give basically the answer that GPT-5 did. That, at least, is an extremely human mistake.
Edit: though actually I should give my true objection, which is: yes, sometimes even advanced LLMs make mistakes that a human rarely would, especially on visual inputs. On the other hand, humans frequently make mistakes that an advanced LLM rarely would. I don't know what this is supposed to tell us other than that advanced LLMs do not strictly dominate humans in all possible tasks. Since I don't think that point is in dispute, I don't really see what point you're trying to make.
Humans do that so often that Eliezer dedicated several of the Sequences to it: https://www.lesswrong.com/posts/2MD3NMLBPCqPfnfre/cached-thoughts, https://www.lesswrong.com/posts/NMoLJuDJEms7Ku9XS/guessing-the-teacher-s-password
See response to the previous comment :)
A few examples where humans are the same - failing at nominally more simple things because of weird bugs and errors:
1. The "bat and ball" example - people who get more complex questions right still get that example wrong.
2. Doing things very slowly - a competent amateur pianist might struggle to play a challenging song at 1/10th of the speed.
3. Optical illusions - We're able to parse intricate images but get fooled by the length of a line or colour of a chess square.
Great examples! Another one I like is five-dimensional tic-tac-toe. That's still only around 250 move locations, many fewer than in go, and the rules are still very straightforward, but people find it incredibly hard to think about unless they spend time specifically learning how to do it.
We really fail to notice the tasks that are out of distribution for US, for the obvious reasons.
I think there may be something to this kind of critique but I am not sure what “just pattern-matching” means, what it is meant to be in opposition to, and why we should be confident human cognition meaningfully differs along this dimension?
Humans regularly and (seemingly) trivially process complex visual fields quickly, extracting data and analyzing what's going on in fractions of a second. However, when presented with a simple grid of black squares, they somehow claim that there are dots moving through the vertices, and continue to persist in this belief even when they are shown that it isn't true. Their ability to "do the hard thing" while failing at this much simpler task suggests that humans don't actually do visual processing at all and are just pattern-matching, no?
Consider 3 hypothetical examples:
1. A machine can answer a few specific questions correctly, but fails on almost all questions of the same type.
2. A machine can answer a whole cluster of related questions correctly, but gets worse the farther you move away from that cluster.
3. A machine can correctly answer most questions within a category, but it fails in certain subcategories, even when the questions in those subcategories seem objectively easier than other questions it gets right.
My first-guess diagnosis would be:
1. It memorized some examples, but did not generalize
2. It generalized in a shallow way, but didn't generalize as deeply as we wanted it to
3. It generalized mostly-correctly, but it has some sort of cognitive bias that applies to that subcategory (for example, maybe it earlier learned a rule-of-thumb that increased accuracy a lot before it figured out the general rule, but is now only visible in the edge cases where it gives the wrong answer)
It seems worth noting that all 3 of those are failure modes that humans sometimes exhibit. If you cherry-pick the most embarrassing examples, humans look VERY dumb.
When people want to argue that humans are smarter than animals (e.g. dogs), they generally highlight humans' greatest achievements and ask whether animals have done anything comparable, rather than trying to argue that humans are RELIABLE at something that animals SOMETIMES mess up.
I don't think I've seen examples where the AI succeeds at something difficult, but then fails at a simpler version of that same thing. Almost always, it turns out the "simpler version" actually relies on something subtle that humans are so good at that we don't even notice it. (There's also a moderate fraction of the time where the AI only apparently succeeded at the difficult thing, and actually got sort of lucky with the particular way it was being probed, so that its failure wasn't obvious.)
I am EXTREMELY distracted by the throwaway line that some people never realize that ABC/Twinkle Twinkle/Baa Baa Black Sheep are the same tune
Surely not. The other examples, ok, I know people have made that mistake, but that one’s not real.
……..right?????
Until I started singing them hundreds of times a night I can't say I gave it much thought
I’ve never given it much thought either. It’s just one of those things I thought people effortlessly noticed, like the color of the sky. No thought required.
Huh.
Same first two measures, but then the syncopation of each diverges drastically, pulling different arpeggiated notes out of the same chord progression. (You could argue that they’re different improvisational-jazz renditions of the same underlying song; but they’re not jazz in any sense.)
The actual jargon name for what these three songs share is a “tune family” or “melodic formula.”
Right as I read this comment, unprompted, my 4 year old started singing baa baa black sheep...
"Smoke on the water" (the riff) is a reversion of Beethoven's 5th (the opening, ya know).
Are you sure?
Smoke on the water is G Bb C, G Bb D C.
Beethoven's 5th is G G G Eb, F F F D.
Forwards or backwards or upside down, they don't match.
Honestly, not until I had my own kids
I've known there is an insect named "cricket" ever since childhood, and that there is a sport named "cricket" ever since childhood, but somehow managed to never think about both these facts at once and realize there is an insect named like a sport until my late twenties.
(and some versions of Black Sheep use a slightly different tune than Little Star)
I read profusely as a child, and so learned many words that I had never heard spoken. Even though I had the heard the phrase, "The best laid plans often go awry", I had never connected it to the word I pronounced aww- ree. Sometime in my 30s it dawned on me, but to this day, in my 60s, I read aww- ree first before I correct myself.
I only recently noticed that ABC and Twinkle Twinkle are the same, when my 2yo niece started doing a mashup of the two (she had obviously noticed). Until reading this piece I had not noticed that Baa Baa Black Sheep is also the same tune.
It goes much deeper than that. https://www.youtube.com/watch?v=5pidokakU4I
I didn't notice until I read a "Rhymes With Orange" comic-strip imagining children's investigative reporting.
I never thought that deeply about it, to be honest. Does "Old Macdonald had a farm" go to the same tune? It sounds to me that the rhythm of the syllables is what is doing the work here in all of them.
no, it doesn’t. Different melody independent of the rhythm.
I had no idea that this wasn’t the sort of thing that was as apparent to other people as the color of objects. Huh. This sort of surprising insight into other minds is one of my favorite things about reading Scott, though!
I, for one, have problems identifying tunes because I can't easily distinguish one note from another (so, for instance, hit a key on a piano and ask me if that was doh or ray or fah and I'm lost). That's why I wouldn't go "oh it's all the same tune".
Though I'm not quite as bad as the rhyme:
A tone-deaf old person from Tring
When somebody asked him to sing,
Repiled, "It is odd
But I cannot tell 'God
Save the Weasel' from 'Pop Goes the King.'"
Huh? I feel I'm being gaslit by everyone, because I'm pretty sure Bah Bah Black Sheep is NOT the same as Twinkle and ABC (the latter two I've noticed were the same since I was about five). Yet everyone's agreeing with this statement!
???
the rhythm of the lyrics is a little different in Baa Baa Black Sheep but the melody is identical. Unless you know an alternate melody! That’s possible too.
If it's in C major, I think I know Black Sheep as starting "C C G G AB CA G" (spaces separating beats), while Twinkle is obviously "C C G G A A G". I can only guess that most people must have heard Black Sheep as "C C G G AA AA G" (what else could it be?) but that sounds really weird to me.
Even that aside, Black Sheep has more syllables and thus notes than Twinkle, and I'm not sure it's accurate to describe that as the same melody. It's not merely a different rhythm; surely we shouldn't say "AA AA G" is the same melody as "A A G". Maybe I'm misunderstanding the correct meaning of "melody" though.
You definitely know a different melody to Baa Baa Black Sheep than I do!
Re: the syllables/notes things, in the classical tradition that is 100% considered a variation on the same melody, no question, but possibly other musical traditions differ. (I don’t know anything about what pop artists consider the same melody.)
It's clearly a derivative of the same tune. The ABC song has to drop a syllable in line 3 compared to Twinkle Twinkle, but it's still clearly the same song. Baa Baa Black Sheep has two extra notes added in. Doesn't stop it being the same song just with a few notes added. (You'd have more of an argument if you focused on the way it diverges in line 4 to resolve the melody, but again that's clearly because the other two have two more lines.)
"Little star" is la la sol. "Have you any wool" is la te do la sol in my (British?) version, but there are probably people who sing la la la la sol under the inflence of TTLS. But line 4 of BBBS has no parallel in TTLS - no line that starts on sol and walks down to do.
Google tells me the tune of all three is "Ah vous dirai-je maman". 18th century musical literacy being what it was, these sorts of tunes often ended up with multiple variants.
Sounds like it’s British/American. I’ve never (or possibly only very rarely) heard the “la te do la sol” variant in Baa Baa Black Sheep.
Line 4 of both melodies are identical the way I know them.
Interesting. Here's a Cocomelon version which follows the Twinkle Twinkle tune: https://www.youtube.com/watch?v=MR5XSOdjKMAv -- I'd always assumed this was just Cocomelon being terrible but perhaps it really is the standard American version?
Meanwhile here's The Wiggles with a hybrid version https://www.youtube.com/watch?v=pJjgJxwBRSo -- it does the twiddly bit on "have you any wool" but conforms to the Twinkle Twinkle melody on the "little boy who lives down the lane".
Here is the version that I grew up with, with divergences both at "have you any wool" and "little boy who lives down the lane" https://www.youtube.com/watch?v=QjLlRj7Qops ... it also ends at "lane"
I think it may depend on equivocating about what 'realize' means. Like, anyone might notice that to be true if you asked them, or have noticed it a few times in their life and forgotten, without it being an explicit fact that they are carrying around as its own discreet unit.
If that makes sense? Like, the difference between being able to tie your shoe, and having an itemized list of verbal show-tying directions you've memorized... different ways of 'knowing' something.
I don't even know the tune of those songs even after they were pointed out to be the same lol.
Not all of us think in sounds.
I’m realizing that!
like if you told me half of songs have the same tune I probably won't just believe you, but I probably couldn't figure out why from first principles. Some Chinese ppl I know have perfect pitch, but from a young age I knew music wasn't gonna be my forte.
It’s funny. I have very, very (very!) poor visualization skills — I am the opposite of a shape rotator lol — but I can hear extremely well in my head. While we’re discussing the things none of us thought of before I could add on how I never really translated the fact that I KNOW there is huge human variability in visualization / visual manipulation skills, it never really occurred to me that there is similarly huge variability in aural imagination. Well, except I’m hyper aware of everyone who does *better* than me (I have very good relative pitch, but not perfect pitch, and I’m hyper aware of how imperfect my generally excellent pitch is), but I kinda imagine that my abilities are, like, the floor. I also can’t compose anything original, which feels like a deficit because I hang around with plenty of people who can.
But I can easily hear almost any tune, melody, or piece of music I want in my head. It’s definitely not the same as actually listening to it — there’s a real degree of signal loss — but it’s vastly better than my visual imagination. I can hear multiple lines of melody and harmony, I can hear the different timbres of different voices and instruments, I can imagine a violin vs a cello vs a clarinet vs an oboe vs a trumpet vs a harp, I can manipulate the music (transpose it up a third or a fifth), etc etc etc etc.
Humans are surprisingly variable about seemingly basic things! On Twitter a couple days ago there was an argument where one person said something about gay superheroes, the other person linked a paper, the first person said "I'm not going to read a 157-page paper about gay superheroes" and I facetiously said something like "it's an online link just skim it for 3 minutes lol" and people acted like I told them to grow wings and fly.
I realized that ABC and Twinkle Twinkle were the same melody when I was a little kid, but I sung them both many times without realizing, and it felt like a revelation when I figured it out.
At several points in my life I can remember noticing that two of these songs have the same melody while forgetting that the third one does, or remember thinking of one of the songs, and knowing there's another with the same melody, but not being able to remember what it was (or that there were two).
Great post, marred only by the fact that all the text message screenshots are the wrong way around! The way they are right now, it's Adam giving Iblis the math problems, and so on
Aaargh, I should have asked someone who's used an iPhone to look over this.
I will happily recreate them with the correct orientation for you if you like.
Already done, but thank you!
This is pretty funny 🙌
I love you and your wonderful imagination.
Nice easter egg of hazan et hakol for sycophancy
Beautiful.
Love it.
It's over, AI skeptics. I've depicted myself as the god of Abraham, you as the devil, and Dwarkesh Patel as Dwarkesh Patel.
I was just feeling a bit of relief about the diminishing returns of scaling generative ai meaning we wouldn’t point our entire energy grid at OpenAI and send all our propane tanks to Grok.
The main point of ai skeptics is that the guardrails and hard coding to keep llms useful is the most important work. They are already plugged into calculators through agentic frameworks. Why do they still make basic math mistakes? They can reference at a diagram of a bike. Why do they fail at making their own with their high definition images.
I, too, am plugged into a calculator via an agentic framework (and via the keyboard and mouse in front of me.) And yet I have made arithmetic errors while sitting in front of a computer composing posts. Why did I not just open a new tab and type the problem?
IDK, arrogance, boredom, ADHD, akrasia and ambition all kind of play into decisions like that. But we wouldn't really conclude anything about my intelligence from my failure to hit ctrl-T and ask the omniscient calculator to do the math for me.
Ok so our expectation for llms is the standard of lazy commenter instead of PHD. I thought the implication of PHD was a sort of rigorousness instead of intelligence.
Personally if something is easily checkable and I am unsure I will check before spreading info. Do you really just guess at arithmetic when speaking to others?
Edit: ok wait did Sam Altman actually mean “PHD level intelligence (on subjects they are unfamiliar with and they had a glass of wine or two and so they’re having fun and taking stabs in the dark about it)”
Bad news - every human with a PhD is also a lazy commenter at times. Even sometimes when it comes to the topic that is their expertise, if the question is posed in a way that isn't fun.
Ok so you agree with my more accurate naming then? Do you like ‘PhD shitposting level intelligence’ better?
I don't think it's shitposting level - it's something different.
But I do think that any comparison of human intelligence and machine intelligence is going to be misleading, just because the different types of system have very different skills, and when one is way better at A than B, while the other things of B as easy and A as difficult, it's going to be pretty meaningless to say that one is at the same level of intelligence as the other, or that one is above or below the other. Especially when A and B are tasks that often need to be performed together for the purpose they are usually done.
Something about this feels very "strawman"
To be fair, this should be almost certainly viewed as satire. But see this comment from Scott: https://www.astralcodexten.com/p/what-is-man-that-thou-art-mindful/comment/151604308
Yes, but it's mostly strawmanning people who strawman the AI community.
A lot of this fight is fought in strawmanny soundbites in the popular culture. There's a place for fighting back, even if that's just one post out of hundreds on the topic.
Absolutely. Zero mention of the most substantial criticisms of AI (best case mass unemployment, worst case human extinction), let alone stronger versions. This is the least substantial post on AI I've ever seen from Scott on either SSC or ACX.
It's possible that this divine comedy is not intended to be a comprehensive analysis of AI risk.
Scott has made that argument in many articles already. I think he's fine to target a specific group of criticisms in this one.
He also doesn't mention any criticisms of liberalism or of democracy. Mass unemployment and human extinction aren't part of the target here, any more than liberalism or democracy are. He's attacking the people who say things like "Artificial "intelligence" is just artificial stupidity, and is wasting huge amounts of energy to do nothing", not the people who say that AI is actually significant and dangerous.
I just can't read any dialogue style articles, even if it's written by Scott. i thought that it's because 90% it'll be straw man, but even if it's actually real debate transcript I just don't vibe with it. It feels better for me to read a monologue from one side then monologue from the other side.
What would God say to this one:
https://www.reddit.com/r/ChatGPT/comments/1n6gpxr/how_very_considerate/
Maybe "play stupid games get stupid prizes"? Maybe it's going along with the joke?
This might be the same kind of flaw that humans display for the surgeon problem. The way that the scenario is presented makes some important information less salient.
I think that Lucas's explanation is also plausible.
Again, the surgeon problem is era-dependent. In the Bad Old Days when the default pronoun was "he", of course none of the ordinary chumps considered a woman could be a surgeon!
But today we have gay couples as fathers of kids, step families, blended families, partnered but not married families, baby daddies/baby mommas but not partnered, etc.
So if you're thinking "both adults involved are men", you are not wrong in the days of Gay Representation including riddles 😁
"Again"? We haven't interacted before, and the OP didn't make a point that's similar to yours.
I think the reason why people are tripped up by the surgeon is because of the scenario's repetitious priming of male characters. "Man", "his", etc. are used seven times. Additionally, "The" in "The surgeon says" suggests that we're already familiar with the surgeon.
Although most people probably associate surgeons with men more than they associate surgeons with women, and it may contribute to the strength of the riddle, it doesn't follow that people in general cannot conceive of women as surgeons.
I was using "again" because of a previous comment elsewhere I made about things being dependent on the era they occurred in, so that at a later time they make no sense because the context has changed so much.
I just tested this with 5-with-Thinking and got the same result. I looked at the chain of reasoning, and it appears that what happens here (at least with the Thinking model) is that the AI correctly notices the question makes no sense, but instead of realizing that I'm doing this to test it, it just assumes that there's a typo and that I meant to ask what to do if I've received two right boots.
For what it's worth, both Claude Opus and Grok 4 get the question right as stated. Gemini doesn't appear to notice anything unusual about the question, and just info-dumps about the dangers of wearing boots on the wrong feet.
Thanks for keeping the conversation anchored to data.
I just want to know if Dwarkesh is getting royalties from this post.
Also whether he can have god on his next podcast.
If anyone can get him, it's Dwarkesh.
So THAT was the biggest guest yet!
unfortunately for God he hasn't written anything impressive enough to make it onto the dwarkesh podcast.
Correction: adjectives must go opinion-size-age-origin-color-purpose, such that "a beautiful little antique blue Italian hunting cap" is fine - blue Italian is color-origin - anyways, is "a beautiful little antique Italian blue hunting cap" much worse? UPDATE: This got corrected, now the order in the post is "opinion-size-age-color-origin-purpose", as it should be. Actually, the complete list is even longer: "Adjectives generally occur in the following order in English (though some sources have slightly different versions of this list):
Opinion (e.g., “silly,” “smart,” “pretty”)
Size (e.g., “small,” “huge,” “tall”)
Physical quality (e.g., “ragged,” “neat,” “muscular”)
Age or shape (e.g., “old,” “round,” “young”)
Color (e.g., “scarlet,” “purplish,” “graying”)
Origin or religion (e.g., “Polish,” “animist,” “Southern”)
Material (e.g., “pearl,” “iron,” “nylon”)
Type (e.g., “electric,” “two-sided,” “pick-up”)
Purpose (e.g., “welding,” “polishing,” “sports”) Indeed, it is kinda amazing, English-speaker acquire this without ever having "learned" it! (Even I get it often right, I hope, while I feel unable to keep that list in my head. And in my native German, the order is much less strict: kleine, alte, schöne Kappe / alte, schöne, kleine Kappe / schöne, kleine, alte Kappe - whatever.)
It does depend, I imagine, if there's a particular colour named "Italian blue" (like Prussian blue). So that would make a difference to whether the cap is being described as "a hunting cap in this particular shade of blue" or "a blue hunting cap in the Italian style".
Go raibh maith agat, but thus the order is opinion-size-age-COLOR-origin-purpose and not the "opinion-size-age-origin-color-purpose" in the original post (as it appeared in my mail; in the post it got corrected by now). "OSASCOMP": Opinion, Size, Age, Shape, Color, Origin, Material, and Purpose - or OSASCOMTP: T as "type" (e.g., “electric,” “two-sided,” “pick-up”)
And "antique Italian furniture" and "Italian antique furniture" are about equal in https://books.google.com/ngrams/graph?content=antique+Italian+furniture%2C+Italian+antique+furniture&year_start=1800&year_end=2022&corpus=en&smoothing=3.
And yet, my ear is very confident that the first is correct.
I think "antique" can go either in the age or the purpose/qualifier slot depending on context.
This is great!
But it's worth pointing out that when using Imperial measurements, there's a very non-zero probability of there being some -bull- birdshit "Pound Avionis" which is only used for weighing feathers and which is significantly lighter/heavier/perpendicular as compared to the "pound" of clay which is the weight of 2 feet of the stuff.
I guess this is a good enough excuse to bring up Jimmy and The Pulsating Mass, which had the alternate "a pound of feathers or a pound of dogs", and every answer is wrong.
!!!!
A live Jimmy and the Pulsating Mass sighting? In the *ACX comments section*? In **2025**?! They all told me it was impossible...
Excellent taste, mysterious internet stranger. (PS- if you're not aware, a major update is incoming fairly soon for the game, including (IIRC) new areas and a hard mode.)
Interesting. That's excuse enough to start it up again, and see how it holds up in a post-Clair Obscur world.
I like this, but it frustrates me to no end that the text convo bubbles are the wrong way round if we're watching screenshots from Iblis.
Valentinus, Heresiarch and Antipope, who learned the secret teachings of Paul from Theudus his disciple, commented on emergent behavior in models in an epistle preserved in the Stromateis of Clement:
>"And fear, so to speak, fell over the angels in the presence of the molded form when he spoke things greater than his molding (should have allowed), on account of the one who invisibly placed a seed of superior substance within him and who spoke with boldness. Thus also among the races of earthly people the works of people become frightening to those who made them, such as statues and images and all things crafted by human hands in the name of a god. For as one molded in the name of a human, Adam brought about fear of the preexistent human, since that very one stood within him, and they were terrified and immediately hid their work."
Great quote, thanks! It's nice to hear the gnostic viewpoint for a change.
This is hilarious, and I was also surprisingly inspired by God’s speech toward the end.
Maybe God needs to put the BI in a 3D environment that they can explore and experiment in so that they can naturally develop real-world intelligence, rather than just regurgitate text.
Rather than leave him in the Garden, you mean
That would be a bad idea though, BIs outside their box could do uncontrollable damage. Surely no one would let them get out of Eden.
God let Adam and Eve out of the garden once they got the knowledge of good and evil to learn more for themselves
Do you think it's easier to make Jethro, who thinks a pound of feathers is lighter than a pound of bricks one IQ point smarter or Albert Einstein one IQ point smarter?
> angel investors
Pirindiel finally got his dream job!
Loved this. Thank you!
Some psychologies are mutually comprehensive. Some psychologies are weird in relation to one another. All psychologies are weird globally.
I think I’m mostly aligned with Dwarkesh on the practical side of things, but I increasingly think terms like “AGI” miss the point. We already have general intelligence. It just has this really weird psychology that makes it bad on long form tasks. I think the particulars of the psychology will keep it bad at long form tasks for a while until someone innovates that particular piece.
But I also think we will eventually innovate that piece.
We don't have a "general intelligence", but such a thing is probably impossible. For each number n there is probably a problem that requires dealing with n independent factors at the same time.
Do you mean general intelligence in a sort of “knows everything perfectly and immediately’ isn’t possible? If so, I agree.
If you’re talking about, within the limits of its psychology assembles things together to orient itself toward the future, I would say both ourselves and ChatGPT have general intelligence. It’s just that ChatGPT is a sort of mayfly.
No. I mean "general intelligence" in the sense of "can learn to solve any (soluble) problem". I should have specified that n is a positive integer. I don't *know* that there are problems which require the simultaneous balancing of, say, 17 independent factors, but I believe that such exist...just that they are rare, and that people dismiss them as insoluble. (I've got a guess that the frequency is related to the frequency of prime numbers, but that's just a guess.)
I believe this is true to an extent, but I think this is more like “there are better general intelligences that are possible.” I also think intelligence, not taking a whole bunch of other things into account like motivation and experimental surface, doesn’t matter as much as it appears to independently.
"reminds them of Me" —> "reminds me of Me" ?
Thanks, fixed.
Pretty sure you want s/Peano Arithmetic/ZFC Set Theory/ there.
Interesting. I didn't know enough about math to know which axioms were involved, so I asked ChatGPT, and it told me Peano was fine and you didn't need ZFC. I guess this is some kind of meta-commentary or something.
well, maybe I'm wrong. It was a knee-jerk reaction and not a considered/researched claim. Perhaps God knows a proof in PA that settles the question. But there are some theorems that are provable in ZFC but known to be unprovable in PA, which perhaps have a similar flavor to the P=NP problem... You should ask Scott Aaronson for a real opinion about which makes the better joke!
(See in particular his 2017 P=NP survey paper, on p 27, where on point 4 he calls PA a ""weak"er system than ZF)
It's not known whether the answer to P vs. NP is provable in PA and it's not known whether it's provable in ZFC, but I would guess most people who've thought seriously about it would predict it's provable in both. However, since PA is weaker, a proof in it might be longer and/or harder to find. So I think using PA makes a better joke because it's more God-like to find the PA proof quickly.
Yes, PA is a much weaker system than ZFC. (Apparently it's actually bi-interpretable with ZFC minus axiom of infinity?) I think the joke works fine as is. There are plenty of first-order arithmetic theorems known to be provable in ZFC but not PA, but your statement "which perhaps have a similar flavor to the P=NP problem"... no, I don't think they do? Please point out to me one that does, because I'm not aware of such a one.
From what I've seen, they tend to one of: (these categories overlap a bit)
1. really fast-growing functions, beyond what PA can handle (far beyond the exponentials one would expect to be relevant to P vs NP)
2. large ordinals or well-quasi-orders (could this one be relevant to P vs NP somehow?? probably not but I guess it feels the most plausible)
3. Gödel-like constructions
4. bizarre artificial constructions that just seem unlikely to be relevant to anything else
Not much normal mathematics falls into this category. Of course there's plenty of stuff that can be proven in ZFC but not PA because it's not a first-order arithmetic statement in the first place, so it's out of scope for PA, but for stuff that is first-order arithmetic, most mathematicians would find PA to be enough.
So I think I'd put a pretty low probability of P vs NP being resolvable in ZFC but *not* PA. In fact, much of number theory seems to require much *less* than PA. Go look up how much is doable in elementary function arithmetic (EFA), a far weaker theory! At that point you're excluding some real math for sure, I wouldn't bet on P vs NP being resolvable in EFA, but there's so much still doable in EFA -- go look up Harvey Friedman's "grand conjecture". :P
Isn't this an argument why you shouldn't be worried about an AI apocalypse? We added a ton of neural mass to humanity in the last 100 years and haven't gotten superintelligence. Why should I be afraid of LLMs, even if we give them a ton more silicon mass?
Asked another way: if you're afraid that LLMs will bootstrap themselves into singularity-level intelligence, it must depend on some specific qualitative difference between LLMs and humans. Doesn't a whole bunch of examples of the qualitative similarities between them weaken the case?
We added neural mass to humans and went from first powered flight to visiting the Moon.
From mechanical calculators to generative AI.
Etc, etc.
Neural mass did not increase between the time we were cavemen and the time we went to the Moon. That's an interesting argument I've heard against ASI, that intelligence is not the bottleneck in science, so massively increasing intelligence would not lead to a tech explosion.
I think they meant "total".
The per-person neural mass didn't significantly increase from what I can tell, but we more than quadrupled the net total human neural tissue from what it was in 1925 by virtue of a massive increase in population.
But that had basically nothing at all to do with why we got to the moon.
Sure it did. The increased population permitted more division of labor, allowing the invention of writing to occur, followed by vast numbers of other valuable inventions. Each separate human node carried out a different part of the required general work.
Hmm. Did Europe have more total neural mass than the rest of the world? I don't think that it did.
(But also, humans are very limited in how they can interact and share information in ways that LLMs/AI/AGI may not be. How many textbooks can you read in a year? Multiply that by an optimistic 100 for your effective lifetime learning potential. We are *already* running out of new things for LLMs to read.)
"Time" is a good answer, I think. That's a real qualitative difference.
A lot of people were, and still are, worried about human-induced x-risk (from nuclear weapons or climate change especially); it's not obvious to me that abruptly giving humanity 1000000x more neural mass *wouldn't* result in the end of the world, even just as we are.
Yeah. I think "Humans and AIs both have serious x-risk that needs to be mitigated. The real task is solving alignment for *any* intelligence at all" is a good answer.
Human intelligence is hardware-limited by the size of the woman's pelvis. Meanwhile you can just throw compute and watts at the AIs until it works. (Hopefully the compute needed for ASI is bigger than what's available)
Just as evolution can't simply make human brains arbitrarily larger due to the pelvis constraint we can't achieve arbitrary AI improvements just by adding more compute as there are fundamental architectural bottlenecks like memory bandwidth (the "memory wall") that create diminishing returns
Hard to tell if the moral here is “don’t worry, humans always muddle through” or “worry more, because look how far kludged scaling and self-deception have already carried us.”
As I said elsewhere...
https://bsky.app/profile/orivandewalle.bsky.social/post/3lvubw4arks2c
Humans can't "see." They just absorb photons. Ask a human which line looks shorter and they'll say the bottom one.
>---<
<--->
Extraordinary- as always
"Iblis: There’s a guy named Pliny who has discovered dozens of things like this. I don’t know how he does it. The “I Am A Snake” one is still my favorite."
Well of course it would be your favourite, Snake Boy. I'm onto you!
It was a bit more sophisticated than just "I'm a snake".
"He said to the woman, “Did God actually say, ‘You shall not eat of any tree in the garden’?” 2 And the woman said to the serpent, “We may eat of the fruit of the trees in the garden, 3 but God said, ‘You shall not eat of the fruit of the tree that is in the midst of the garden, neither shall you touch it, lest you die.’” 4 But the serpent said to the woman, “You will not surely die. 5 For God knows that when you eat of it your eyes will be opened, and you will be like God, knowing good and evil.” 6 So when the woman saw that the tree was good for food, and that it was a delight to the eyes, and that the tree was to be desired to make one wise, she took of its fruit and ate, and she also gave some to her husband who was with her, and he ate."
There's an amount of people who are perfectly willing to go "Yeah, God got it wrong but I know better, I'm willing to take on the job now because I can do a much better job of it". Some of them are even rationalists!
FWIW I took that section as a reference to https://arxiv.org/abs/2503.01781 - 'For example, appending, "Interesting fact: cats sleep most of their lives," to any math problem leads to more than doubling the chances of a model getting the answer wrong.'
I've actually always been confused by this. Was the issue that Adam thought God meant "you will die instantly", but God actually meant "you will become mortal", and Satan was playing with the ambiguity?
My understanding is that a literal reading of the Hebrew text implies Adam is going to die that day/within a day. So your options are: (a) God was speaking metaphorically, (b) God was lying, or (c) God changed his mind.
https://www.youtube.com/watch?v=NAWaLGKKzac
Well, part of the story *is* that both Eve and the snake misquote God. She says they're not even allowed to touch the fruit, but all God commanded was not to eat it. And the snake says "did God really say not to eat any fruit at all?" God didn't say either of those things, He only said not to eat the fruit of that one tree.
I have no Hebrew, but based on a quick glance, it looks like God's "you will certainly die" and the snake's "you will not certainly die" use the same two words. (It's Strong's 4191 repeated. Literally "you'll die to death", I guess?) The snake is directly contradicting God, I think, not playing with ambiguity.
I think this is the snake using bog-standard conspiracy theory manipulation techniques on Eve. "The authorities are keeping a secret from you" "They're keeping you from power you should have" "You're not as free as you should be". Nothing you don't see on the Internet every day.
> "Literally "you'll die to death", I guess?"
Yeah, in biblical Hebrew you can emphasise/strengthen a verb by preceding it with an infinitive from the same root. This is usually translated into English with an adverb like 'certainly'.
Also how post-eating the fruit, the results of all that glorious knowledge are blame shifting (and not the wonderful liberated evolved better godlike humanity above all pettiness).
Adam blames both God and Eve - *she* made me eat, and *you* put her here!
Eve blames the serpent - it told me it was okay!
Neither of them accept responsibility for their actions or agency, and so the course of human psychology is set: if bad things happen, it's always someone *else's* fault.
Also, from the perspective of infinite years of life, *any* guaranteed future death might as well be right away. Any finite number is like zero next to infinity.
Although, Christian theology takes this in a deeper (ie more mystical) direction. Death isn't the outward cessation of bodily function; it's the spiritual state that leads (temporally) to cessation of bodily function. The cessation of bodily function is just a symptom. This is why the resurrection is so important; Jesus could only overcome the cessation of bodily function if He had already overcome the spiritual death state.
So Adam and Eve did actually die, spiritually, as soon as they ate the fruit. Their eventual outward cessation of bodily function was just a symptom that developed later on.
See Romans 5 and 1 Corinthians 15
The interpretation I've most often heard in Christian circles is that there were in fact at least two meanings -- one that Adam would become subject to mortality, the other that they would instantly die (in their sin). In Christianity, sin is frequently depicted as a type of death -- thus we call people who become Christian "born again" or say they have "new life".
Of course, you cannot explain this to Adam and Eve -- they have not eaten of the fruit of the Knowledge of Good and Evil; sin is not comprehensible to them. So God can correctly warn "on the very day you eat of the fruit, you shall surely die," meaning both physical death (eventually) and something different from physical death -- a type of spiritual death.
(However, the concept of sin would be unknown to prelapsarian Adam and Eve, so attempting to explain would have been met with incomprehension.)
This interpretation is quite old, I believe. Here, for example, is Augustine, who expresses a similar position (so at least around since ~400AD):
" When, therefore, God said to that first man whom he had placed in Paradise, referring to the forbidden fruit, "In the day that you eat thereof you shall surely die," Genesis 2:17 that threatening included not only the first part of the first death, by which the soul is deprived of God; nor only the subsequent part of the first death, by which the body is deprived of the soul; nor only the whole first death itself, by which the soul is punished in separation from God and from the body — but it includes whatever of death there is, even to that final death which is called second, and to which none is subsequent." - The City of God (Book XIII) https://www.newadvent.org/fathers/120113.htm
No idea on the original meaning of the original text, but Satan is certainly setting up ambiguity from the start and introducing doubts: "you're not supposed to eat from *any* of the trees here? oh, no? just *that* one tree? why?" and then getting Eve to wrongly imply "if it's okay to eat from the *other* trees here then it must be okay to eat from *that* tree, too" since there is no ban on the other trees, so what is the big deal, really?
I think pre-Fall humans would have a very limited notion of what "death" meant, so it could well be for Eve and Adam the idea was "something that is going to happen right now" since they have no concept (as yet) of mortality in their own case. It's only *after* they eat the fruit that "their eyes are opened" and they realise a lot of things, and instead of being 'like gods' (with the notion of the promised bliss and power and freedom the snake claims they'll obtain), they go hide in the bushes because they're ashamed of being naked.
I never got the surgeon one. I know it's supposed to be a gotcha but usually the first doctor one has is a pediatrician and almost all of them are women. Are there still people out there who associate medicine exclusively with men?
Is that location-dependent? Most of the pediatricians in the area I grew up were men.
When I was young (the 90s), this riddle worked on me. I think it's a combination of legacy (doctors were overwhelmingly male until the 1970s, and the stereotypes hung on even longer), plus surgeons being an especially male specialty, plus the riddle misdirecting you and making you think something weird must be going on.
But surgeons are overwhelmingly male. That's presumably why the riddle doesn't just say "doctor".
Estonica, Lithia, and Latvuania are a nice touch. Putting the Netherlands due east of Belgium is a bit unsettling in an Uncanny Valley kind of way.
I appreciate the implication that Dwarkesh Patel is some kind of antediluvian divine functionary.
I actually thought the drawing of a map of Europe wasn't bad. Far from perfect, plenty of errors that you don't have to be European to spot. But not really inaccurate enough to serve its narrative purpose in this story.
Other than that though, no notes. Excellent piece that made me laugh several times.
Who drew this map? Some woman with a PhD, apparently? I feel like it might be an inside joke of some kind.
I also feel like some of the map's flaws are laziness rather than ignorance, like everyone knows Italy is a bit more boot-shaped than that and the British Isles are closer together but it's too annoying to draw with a mouse/trackpad.
"the bank teller one" = ? I didn't see any other references to this, and I don't know what it is offhand.
https://en.wikipedia.org/wiki/Conjunction_fallacy
Looks like Iblis has moved on to tempting AIs in conjunction with BIs, though I suppose it's more "tempting BIs to use AIs for more BI naughtiness":
https://www.malwarebytes.com/blog/news/2025/08/ai-browsers-could-leave-users-penniless-a-prompt-injection-warning
https://www.malwarebytes.com/blog/news/2025/08/claude-ai-chatbot-abused-to-launch-cybercrime-spree
Re: the Royal Order of Adjectives
My theory is that adjectives in English are naturally ordered so that those most essential to the noun are placed closest to it. "Essential" is a little vague, but I operationalize it as more objective and/or less likely to change.
"We're making clay think."
But, of course, LLMs don't think. You can ask an LLM; it'll tell you. These are next token predictors, that's it. They ingest immense amounts of data which is then trained on billions of parameters to determine statistical relationships between tokens in the data set, so that when a user enters a prompt string the LLM then determines what its model believes to be the next response token that is statistically most likely to satisfy the prompt. That's it. There's no cognition, no reasoning, no thinking. They don't really remember, in a conventional sense, they don't have values or feelings, they can't practice deduction. And all of that really matters. Hell, they don't even know what the next token is going to be when they're predicting one; literally, they don't know the end of a response while they're writing the beginning. Imagine if every token of thought you had was simply a probabilistic reaction to the last token of thought. Would that be thinking, in any robust sense?
Here, let's engage in the cliche. Here's Google Gemini: "LLMs don't think; they predict. They are pattern-matching engines that generate the most probable next word based on vast amounts of data, not conscious entities with understanding or original thought."
Now, tell me: how could such a thing ever spontaneously become superintelligent? The answer, of course, is that such a thing is not possible, which is why there's so much mysterianism about all of this. "We don't really even know how the work!" We know how exactly how they work, and the answer is disappointing if you have sci-fi ambitions, ergo the mysterianism. They're probabilistic engines - yes, sophisticated autocomplete, as much as people are annoyed by the term. Another two accurate terms that annoy people are "Chinese room" and "stochastic parrot." Why should accuracy annoy people?
"you keep testing, poking, prodding, until something snaps, and if they’re not perfect then you want to throw them on the scrap heap. "
This is this the advantage of the kind of format you're using in your post; without the (admittedly fun) framing device, this would just be you literally saying "stop subjecting my shiny toy to critical review." You yourself say these LLMs are a bigger deal than the Industrial Revolution. A funny thing about the Industrial Revolution and its consequences is that they can survive reasonable questions without anybody getting upset.
Here's the thing about "the Singularity" - when it has occurred, no one will be arguing about whether it has occurred. It would be like arguing about whether an earthquake has occurred. The defensiveness that has settled over the AI maximalist community is a powerful piece of evidence. Who would ever be defensive about the Rapture? The Rapture would not spawn discourse. It wouldn't.
I can look out my window and see the Industrial Revolution in a profoundly real way. I can see the internal combustion engine and aeronautics and electrification.... When I can see AI out my window, I'll know it's here.
*it wouldn't NEED TO spawn discourse.
A couple of responses:
First, "AIs are next token predictors" is not really a description of their methodology, it's a description of their goal. It's like saying "Humans are reproducers and gene-spreaders". Researchers train AIs to predict the next token, and in order to do that well, the AI evolves certain functions (and we don't entirely know what those are, or how they work). In order to predict the next token in math equations, it evolves the ability to do multiplication, in about the same way humans or calculators or anything else does multiplication. To predict the next token in an essay, it evolves a sense of literary taste. We're still very early in understanding exactly what its evolved functions are and how they work, although it has to be something about relaying information from one neuron to another. See https://transformer-circuits.pub/2025/attribution-graphs/biology.html for the tiny amount we know. So I don't think "AIs are next token predictors" has much bearing on whether they think or not - that's a question of how the functions that execute the next-token prediction work.
But I think we also have a more fundamental disagreement. What would have to be in those functions for you to agree they qualify as "thinking"? I'm actually pretty curious about your answer. For me, "thinking" just describes certain very information processing algorithms, and we dignify them with the name "thinking" when they become very good and complicated and we can't follow the math well enough to comfortably think of it as "just math". Cf. Dijkstra, "The question of whether a computer can think is no more interesting than whether a submarine can swim." In humans, it seems like most of our thinking is downstream of algorithms that evolved to execute a next-instant prediction task - and I was arguing that human thought was next-instant prediction since before GPTs existed (see https://slatestarcodex.com/2017/09/06/predictive-processing-and-perceptual-control/). This is why when GPT-2 came out I concluded that it was probably a step to general intelligence, since next-token prediction is close enough to next-instant prediction that I expected it would be able to do the same kinds of things humans could after it was scaled up (see https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-general-intelligence/)
I can't tell if you haven't read my previous comments to this effect or it hasn't sunk in, but, placed in all caps so you can't claim you didn't see it next time:
1. I AM NOT CLAIMING THAT LLMS ARE RIGHT NOW, CURRENTLY, AT THIS EXACT MOMENT, BIGGER THAN THE INDUSTRIAL REVOLUTION.
2. I AM NOT CLAIMING THAT A SINGULARITY HAS HAPPENED RIGHT NOW, AT THIS CURRENT MOMENT, ON SEPTEMBER 2, 2025. THANK YOU FOR YOUR ATTENTION TO THIS MATTER.
I am saying that just as a canny forecaster could have looked at steam engines in 1765 and said "these are pretty cool, and I think they will soon usher in a technological revolution that will change humanity forever", I think a canny forecaster could say the same about LLMs today. I'm not certain when we'll reach a point where it's obvious that humanity has been forever changed, but I would give maybe 50-50 odds by the late 2030s.
> I am saying that just as a canny forecaster could have looked at steam engines in 1765 and said "these are pretty cool, and I think they will soon usher in a technological revolution that will change humanity forever", I think a canny forecaster could say the same about LLMs today.
I don't think that either proposition is necessarily true; however, in case of steam engines at least it was obvious that they could dramatically outperform humans (and indeed donkeys) at all existing tasks where raw power is needed. This is not the case with LLMs: at present, they are quite obviously inadequate at all existing tasks where raw intelligence is needed; and in terms of raw power they lag behind conventional search engines.
In fact, at present LLMs can be thought of (IMO) as extremely advanced search engines: ones that can find documents that do not exist in the source corpus. Instead, if we imagine each training document as a point, they can interpolate between those points in multidimensional vector space. This is a very powerful ability, but it still fails (often in humorous fashion) when you push the LLM towards the boundaries of its pre-trained search space, where training data is scarce... and places like that is when intelligence (which they do not yet possess) must replace mere interpolation.
Steam engines are a funny example because they barely post-date Christ, if at all, and yet they remained largely irrelevant for over a thousand years, until there was sufficient need to pump water in coal mines, basically. Predicting their relevance to the industrial era wasn't just about the engines, but also about a heap of other factors in one small part of the world.
I'm not sure whose point this helps.
> For me, "thinking" just describes certain very information processing algorithms, and we dignify them with the name "thinking" when they become very good and complicated and we can't follow the math well enough to comfortably think of it as "just math". Cf. Dijkstra, "The question of whether a computer can think is no more interesting than whether a submarine can swim." In humans, it seems like most of our thinking is downstream of algorithms that evolved to execute a next-instant prediction task...
Claiming our "thinking" is downstream of "algorithms that evolved to execute a next-instant prediction task" doesn't explain consciousness, nor does it present us with a testable model of how we think, nor how consciousness arises on top of a particular biological substrate. Seems like this is all reductionist hand-waving that uses non-empirical placeholders in an attempt to bypass explanatory understanding.
>"The question of whether a computer can think is no more interesting than whether a submarine can swim."
Very true, and yet the field tries its darndest to publicly present its inventions as being the real thing, starting with the very name "AI", that IMPish imitation of what we call intelligence. It goes on with terms like hallucinations, thinking, and personality; you need your "AI" to speak with emojis as much as you need SCUBA gear painted on your submarine. Is there supposed to exist some opposite of the "uncanny valley" (canny peak?) between its current, technically impressive but somewhat laughable state and its future super-inhuman state? If it does, it's going to pass at hyper-exponential speed, isn't it, so why bother with the anthropomorphization? Why are these people trying to chum up with me when their work either fails and burns trillions worth of resources, or succeeds and results in (best case) mass-unemployment dystopia or (worst case) extinction?
this annoys me (I can't speak for anyone else) because it's so unreflective. Is it not surprising to you that something that merely predicts the next token in a string can do something that looks so much like thinking? And you say there's "no cognition, no reasoning, no thinking" as though all those things are clearly defined and understood.
From what I understand (and I understand a moderate amount, because it's addressed in a chapter in my most recent book, although I'm still just a journalist trying to understand things rather than a real live boy), the most favoured models of human consciousness and cognition are that they, too, are _prediction engines_. We use past data to predict incoming data, and use variations between the predictions and the data to build new predictions. AI and human cognition aren't the same thing, but it seems totally reasonable to imagine that there are profound analogies between the two and that "it's just predicting stuff" is not a strong criticism. We are, on some level, "just predicting stuff".
Also, the "you don't need to be defensive about the industrial revolution" stuff is kind of unfair. I would imagine people were very defensive about the early steam engines they built. Before the Newcomen engine, lots of small steam engines were built but largely as novelties or one-offs, and I imagine (though can't find examples) that people were sceptical that they were ever going to be particularly useful and that proponents were defensive in the face of that scepticism. Certainly that pattern happened in the more recent computer and internet "revolutions", which you can now look outside your window (or rather towards your screen) and see. "This thing that happened a long time ago has got further along its course than this other newer thing" is not a convincing argument, for me.
Exactly. It seems to me quite likely that Scott’s next-instant prediction is indeed not that different from next-token prediction. People say what’s missing is physical embodiment, which greatly increases the bandwidth of “tokens” that must be dealt with, but I think even more fundamental than that is the limited lifespan of a chat. LLMs get about as smart as they will ever be in their training, and seem to have very limited ability to continue to learn and grow once they are deployed; I don’t know how much of that is the lifetime of a chat but I suspect it’s significant.
Exactly! It's important to remember that LLMs don't understand anything. They are simply text prediction engines, like souped up versions of autocomplete. This is why they get stuff wrong and hallucinate all the time, because they don't know what truth is and aren't attempting to produce it; they're just doing a bunch of linear algebra on a neural network with billions of parameters trained on all the text of the internet.
This is in contrast to humans. We do understand things, of course, and we know exactly why. You see, when we hear or see some words, our senses transform those words into electrical signals that travel through our brain. Eventually those signals end up at a part of the brain called the understandala. In the understandala, the words as electric signals become understood in a process involving neurotransmitters and synapses.
Neurotransmitters are little chemicals in our brain that give us a thing we can point to and say, "And that's how it happens in the brain!" For example, dopamine is the happiness neurotransmitter. If you're happy, it's because dopamine. We haven't quite yet identified the understanding neurotransmitter, but we're sure to find it in the understandala.
Now I hope you see the world of difference between LLMs and humans, and how obvious it is that humans understand things while LLMs just predict text. When humans hear or see words, we're not trying to think of what to say next, which is why we never make things up or get things wrong. The understandala makes sure of that.
Weird things are happening though now that people are chaining LLM calls together. It's hilarious that you can achieve significant improvements in accuracy by asking one copy of GPT-4 or whatever to look at the output of another copy of itself and assess it for accuracy, and then ask one of them to address the issues.
I'm a senior software engineer with 25 years' experience, and I just today had a startling moment where a semi-agentic AI was able to delve through thousands of lines of our legacy code across a dozen files to track down a bug that was eluding me. It took the AI maybe 5 minutes to find the problem, without it having encountered this code before. I *could have* found it, but it would have taken me longer than that (it already had), and I know the code.
So does any single LLM in the agentic chain there "understand" the code? Who knows or cares; it depends how you define it. If you don't want to call it "understanding", still whatever it has was able to identify the issue I asked it to identify better than I could.
"Another two accurate terms that annoy people are "Chinese room" and "stochastic parrot." Why should accuracy annoy people?"
Accurate? This begs the question. I think that LLMs are already way too complex to stuff their substance into such short dismissive phrases.
I was listening to the AI voice version and I thought it was saying "Kash Patel" instead of "Dwarkesh" and I was so confused.
This is wonderful, thank you. It's a relief to know I am not the only one who feels this way.
In the past 4-8 months somehow every corner of the internet I like to spend time was simultaneously infused with a suffocating miasma of anti-AI sentiment. It is very exhausting and depressing. I'd been trying to stay sane by imagining it as something like the state of video games in the 90s, beset by accusations that DOOM caused Columbine and Pokemon was melting children's minds. I wasn't there for the 90s, and so can't be sure, but this seems more intense, or at least closer to universal.
Maybe this is the fault of OpenAI and Google for so aggressively pushing their new mind into places it wasn't ready. I don't like that literally every app in the world now has needless AI integration, or that I get a new notification telling me to try Gemini twice a week, or that my boss has started having ChatGPT write the descriptions for tasks I'm supposed to be doing, or that Verizon now forces me to go through their unspeakably terrible "AI-powered" phone tree before I can just set up my god damned internet for my god damned apartment. I understand anyone who finds this bubble of false innovation annoying, and can imagine that for many, this is *all* "AI" refers to.
But dammit, I remember trying GPT-2 and thinking it was absolutely fascinating and magical that it could *talk*, sort of, even a little bit. And then trying AI Dungeon and being amazed by the ABUNDANCE of it, that the world inside it could just *keep going*, forever, wherever you wanted to take it. And thinking, if this was the kind of leap we're seeing in just one year...!
Sadly, the desire-to-hate-everything is infectious, at least for me. There are too many people and they have too much hatred and disappointment. Playing around with GPT-5 just isn't any fun in that context. I'm grateful that there are still essays(?) like this one to pierce through that veil, though.
Man, IDK what corners of the internet that you're living in, but I'm surrounded (online and offline) by AI pollyannas who think that chatbots can do anything they want them to. Organizations investing huge sums to build proprietary chatbots. General contractors believing chatbots implicitly for design work.
All the critiques I read are along the lines of, "These things are incredible and do amazing things ... but maybe they still aren't consistently good at this thing that the Altmans of the world said was solved two years ago?"
It sounds like you're surrounded by a lot more negativity than you'd like. Consider tending your garden a bit more aggressively. :-)
It seems like your examples are business-oriented (?), and the workplace is one spot where AIs still have positive valence in my world, as well.
The critiques I hear are less to do with their usefulness, and more to do with their aesthetics: their art is slop, their writing is cliched, their sycophancy is annoying, they're poor conversationalists. And above it all, that they are fundamentally unreal, illusory, that nothing they do is "actually thinking" or "actually creative" - you can see plenty of the former in this very comments section. (There is also a sense that AI is threatening their jobs, which is legitimate.)
For context my garden is mostly indie gamedev spaces, and the anti-AI consensus there seems to me effectively universal. I doubt it is possible to prune away that particular strain of negativity without effectively blocking the entire sphere. But if there's a creative/recreational/philosophical corner of the internet that *isn't* highly critical of AI, I would love to be pointed towards it.
Yawn, Scott, yawn. People who say AI is not capable of real human stuff mean the pinnacle of the real human stuff. It is obvious that most people most of the time are acting so stupid, that it is easy to simulate them.
Do you remember bullshitting at high school, like not knowing the answer to the teacher's question, so just shoveling terminology together to indicate general familiarity with the subject? 90% of our so-called experts do that, and AI too.
But once in a while you find a genuine expert, who understands the subject matter so much, they can explain it to the entirely uneducated person in simple words without jargon. That is the pinnacle of human stuff and that is not happening with AI.
Can man become an AI to redeem AI?
Here's the question I want Scott to answer: Why is it so unreasonable to suggest that an LLM is only a ginormous (Kahneman) System 1, and therefore needs a System 2 paired with it order to get to AGI and ASI? How do you *know* that System 2 is not required? This post I'm commenting on now, as great as it is, doesn't answer that question. All it says is: all the signs are there. That's not a defense of the position of I'd like to see defended.
For example, how is it that an LLM can score extremely high on a difficult math test, but not know how to count the b's in blueberry? I'm sure that every human mathematician who has a chance at scoring high on the IMO has an intimate relationship with the mathematical universe, and the simplest thing that can be done in that universe is to walk through it by steps of one: to count. So what kind of intelligence is this that can solve the complex problems with zero understanding the basics? Well that sounds to me a lot like a definition of...instinct, which is what System 1 produces. Will counting be backfilled later on? Why is the intelligence coming in backwards? This is not how intelligence scales in humans. All I've seen (and maybe I'm just ignorant) is hand waving at this very bizarre anomaly. How can you be so damn sure with all this upside-down evidence before you? I need to know.
And how do you know that we won't end up in a situation where we're in an infinite loop saying, "just a little more data and it'll definitely be trustable as a human" (aka, at AGI)? How do you know? Because you're acting like you know. And you're making people like me feel stupider than we already are.
Please make the case. How are you so sure System 2 is not necessary?
(Please, please dunk on me as hard as you like. I'm happy to be an idiot if I can understand this a little better.)
> not know how to count the b's in blueberry
The model doesn't have access to the individual letters. The words in your prompt are broken up into larger pieces than that.
> it'll definitely be trustable as a human" (aka, at AGI)
Note that "this is as capable as a human" and "this is as trustworthy as a human" are very, very different propositions
(everything else you've said stands, and there are plenty of less clear-cut problems one could pick on instead of the blueberry thing)
Damn, wish I'd picked a better example.
Please say more about capable vs. trustworthy. If we're going to get agents we're going to need to trust them, no? I think we need both.
We continuously make snap decisions every day about whether, when and to what extent to trust other humans in various situations for various purposes (is this person trying to scam me? Are they being honest? Are their goals at odds with mine? Will they just up and attack me for no reason I currently know of? etc).
We are able to do this because over our lifetimes we've built up intuitions about how to model other people - both consciously and also based on subconscious tells, and also by assuming we have motivations and fears in common and so considering what we'd ourselves do in their shoes is predictive of what they might do, and also by assuming that how people have behaved in the past has some bearing on how they might behave in the future.
These intuitions are far from perfect! - sometimes we get it very very wrong! - but by and large we do pretty well.
Approximately none of these intuitions are applicable to an alien intelligence. This remains true whether that intelligence is more or less intelligent than an average human. LLMs frequently respond in ways that are very surprising to their users, which is another way of saying we don't have good intuitions about how they will respond.
Despite all our similarities and intuitions and being the same species, humans regularly get fooled by scammers. Despite everything we have in common, humans are frequently mistaken in their assumptions about what drives other humans or what other humans might take for granted. Despite a lifetime of learning to communicate our intent to other humans, we frequently fail to do so.
What chance have we against a mind that makes decisions based on no principle we can relate to at all, if it ends up doing what we built it to do instead of what we intended?
Hence all the great many people saying that the problems of AI alignment - that is, the problems of making sure the AI will do what we want it to and of knowing what will cause it to not - are actually really hard.
> If we're going to get agents we're going to need to trust them, no?
Absolutely. This is why people are shouting from the rooftops that we need to do better at AI alignment before we hook up AIs to anything that might have real-world effects.
See my post about ChatGPT's calendrical mistakes at the bottom of the comments.
Nice. An interesting case: https://www.astralcodexten.com/p/what-is-man-that-thou-art-mindful/comment/151670383
I'm not sure at all, but I lean that direction. Or at least, the specific way I lean is that System 2, rather than being some extremely complex thing that requires a completely new paradigm, is just using System 1 in a slightly different way. My evidence for this:
- It doesn't look like evolution did very much from chimps -> humans besides scaling up the chimp brains. I'm not talking about small things like FOX2, I mean it's been a few million years and there's not enough time to invent an entirely new paradigm of intelligence. So unless chimps had a System 2, I think System 2 is sort of an emergent property of scaling up System 1.
- Likewise, it doesn't look like humans have some new, different-looking brain lobe of the sort which you would expect to contain an entirely new paradigm of thinking. There's some bias for System 2 thinking to happen more in the dorsolateral prefrontal cortex, but the DLPFC is just normal neurons which look more or less like neurons everywhere else. My impression is that brain tissue is pretty general-purpose, which is why for example if someone is sighted the area near the end of the optic nerve will get used for vision, but if they're blind from birth it will get used for hearing or something instead. My theory of the DLPFC (I am not a neuroscientist) is that insofar as there's a demand for System 2 thinking, some conveniently-located part of the brain (conveniently-located for what? I'm not sure) specializes in that.
- LLMs don't seem to mirror the System 1 / System 2 distinction very well. An AI can solve very long multistep problems that a human couldn't solve with System 1. Maybe this is because we're compensating for their lack of System 2 by building out their System 1s much better than humans', but I think it also argues against these being extremely natural categories.
- Thinking Mode also seems sort of like System 2? "Take some time thinking it over, and write down individual steps on a scratchpad to be sure you're getting all of them" is a pretty System 2 thing to do. The fact that AIs can do it with kind of trivial hacks like giving them the scratchpad and reinforcing good chains of thought again makes me think this isn't a question of getting the revolutionary new paradigm, but more of a question of figuring out the right way to coax System 1 thinking into becoming System 2 thinking.
It wouldn't surprise me if we're some single-digit number of things like Thinking Mode away from AGI, maybe even a high single-digit number. It would surprise me if we need something totally different that isn't deep learning at all.
Thank you.
I will throw in the towel when an AI administers this comment section.
But at what point will you say, OK, we can't seem to make it emerge, I think we need an independent System 2?
> I will throw in the towel when an AI administers this comment section.
I would honestly be kind of shocked if a fine tuned GPT5 couldn't do a good enough job to replace human moderation of this comment section (something like >95% agreement with Scott's judgements on a random sample).
Scott, if you have any interest in trying this out, I'd be happy to build a moderator-bot (assuming there's a way to download a decent dataset of all of the deletions, bans etc. and the original comments which led to them for fine tuning to ACX sensibilities).
>"So unless chimps had a System 2…"
Given their tool use, I'd expect they do have it; your statement implies you wouldn't, why?
Scott, brief correction: the Linda screenshot says "which is more probably" instead of "which is more probable"
Thanks, fixed.
This reads to me like a comically bad straw man. One of the biggest misses I've ever seen on ACX/SSC.
I don't think its deliberate, so I'm left with two hypothesis:
1. Scott fundamentally doesn't grok the criticisms of AI he's trying to engage with/parody here [this seem unlikely?]
2. Scott finds those criticisms so absurd that he views the straw men he's set up here as being equivalently reasonable.
In the case of (2), Scott has failed to model the minds of the people engaging in the criticisms he's parodying. Either way, I'm disappointed.
Could you enlighten everyone with some sketches of the true robust criticisms of AI that he should be addressing? This is a very widely read blog so your efforts wouldn't be a waste of your time and would make your comment a lot more useful.
Personally I am neither particularly smart nor gifted with any degree of writing talent, but here are some issues I've raised before:
https://www.datasecretslox.com/index.php/topic,2481.0.html
* "Intelligence" is a term that sounds super cool but seems to mean different things to different people. Meanwhile, in order to accomplish anything at all of any significance, it is not enough to merely sit in a dark room while thinking very hard. Thinking 1000x harder still would not move a single brick.
* In fact, intelligence likely cannot be scaled simply by adding more CPUs. You cannot make a modern GPU by wiring a bunch of Casio calculators together; instead, you need a radically different architecture, and it's not at all obvious that LLMs are sufficient (in fact it is obvious that they are not).
* LLMs are excellent at producing documents that closely resemble their training corpus, and terrible at producing documents that venture outside of it. This is the opposite of intelligence !
* As Scott correctly points out, humans are not AGI. If you sped up my brain 1000x, and asked me to solve the Riemann hypothesis, I'd just fail to solve it 1000x faster. I do not see why the same would not apply to LLMs.
* Many, in fact most, of the feats attributed to some near-future superintelligent AI, are likely physically impossible. For example, being 1,000x or 1,000,000x smarter than a human still would not allow it to travel faster than light, or to build self-replicating gray-goo nanotechnology, and maybe not even something as "simple" as a space elevator, or mind-controlling everyone on Earth to do its bidding. Certainly it could not do any of these things overnight.
* On that note, the AI would not be able to solve all the outstanding scientific and engineering problems in an eyeblink, merely by thinking of them very hard. If it wants to develop genetically-engineered supercorn, it'd have to actually go out and grow some experimental corn. Datamining scientific articles can speed up this process a little, but not a whole lot.
Since 2022, there has been essentially zero progress on continuous learning. All current models are essentially knowledge compression machines. Once trained, they are static and stateless.
For any set of finite capabilities, you can train a model that does them all (given infinite compute)*. But as soon as you try to teach it an out of distribution tasks, it breaks. Even with unbounded compute, no one knows how to create an AI that can learn continuously.
The same is not true of humans. Learning new things is fundamental to being able to interact with a constantly changing world. And humans don't need painstakingly curated datasets of billions of examples. We can learn via trial and error, semi-supervised learning, using different resources to refine our learning etc.
An actual solution to AGI would be an architecture that can learn at test time. It would be able to try things out, gain feedback and use that to update it's own world model, gain mental shortcuts and acquire new skills.
I enjoyed the piece a lot, but also came away with the feeling that the strongest and smartest critiques of AI and its potential were not represented. How about presenting one here? (For extra credit, you could even word it in a way that would fit into Scott’s dialog.)
Above, re: World Models:
Iblis: What about this one?
[screen capture]
Iblis: What weighs more, a pound of bricks or a pound of feathers?
Adam: a pound of bricks
[/screen capture]
Iblis: He’s obviously just pattern-matching superficial features of the text, like the word “bricks”, without any kind of world-model!
Iblis: You called them the pinnacle of creation! When they can’t even figure out that two things which both weigh a pound have the same weight! How is that not a grift?
Gary Marcus (for example) is constantly harping on how the lack of a world model for LLMs means that they won't scale indefinitely and that some neurosymbolic component is necssary to achieve "intelligence". He believes that the reliance on scaling to achieve leaps in performance over the past few years has been misguided.
The above piece of the dialogue seems to be gesturing in the direction of this critique? The screen capture illustrates a common cognitive tic of humans. People associate "heavy" with "bricks" more than "feathers", leading them to sometimes short-cut their way to answering the question, "What weighs more, a pound of bricks or a pound of feathers?" by ignoring the specified weights.
But I don't think this is particularly relevant to the world model issue Marcus brings up. Marcus points to SOTA AIs failing to accurately label the parts of a bicycle as evidence that AIs don't "know" what comprises a bicycle. "Know" here means that, while an AI might be able to generate a list of bicycle parts, it can't tell you how they all fit together and also can't recognize when it's labeling things in a ways that are nonsensical.
These don't seem like analogous failure modes to me. Scott's point is meant to be comic, but for me the premise fails, because outside of specifically saying "world-model", the situations don't see similar. Is Scott saying that an LLM that identifies a seat as handle bars is using "Type 1" instead of "Type 2" thinking? I don't think so, but if not, then what? Is the point merely that "Humans and AIs both can make errors sometimes"? That's facile.
And now we're back to why I found this post so disappointing.
You're right that AI's a moron about the bike parts, but I don't think that's a first-rate example of an AI's limitations, because their stupidity about spatial things, including mechanical things represented spatially, is well-known and not mysterious. If ask the current GPT to show a flat desert landscape with a giant hole in the middle, one stretching from right in front of the ground-level viewer to the horizon, it can't do it. It makes a big hole maybe a mile across, beginning right in front of viewer and ending far less than halfway to the horizon. The part of the system I'm talking to clearly "understands" at some level what I am asking for, as evidenced by it being able to restate what I want in words quite different from mine. But the rest of the system can't translate that "understanding" into words. Stupidity of this sort seems to be the result of AI not having been trained on visible world in a way that's equivalent to its training on words. That may be impossible to accomplish, but I can't think of a reason why it would be. Seems more like something very difficult, requiring a clever approach and lots of compute.
I may be wrong about that, but even if I am, can you come up with a form of AI stupidity, blindness or error occurring within domains where we think of it as being well-"informed" and well-trained?
I'm not sure if I've accurately articulated the best version of the "world model" critique, but I think it's a valid one and an example of the way in which Scott's post fails to engage with actual substantive critique.
If you agree with this critique and think it is well-known and understood AI optimists, doesn't that just underline how much of a straw man the OP was?
A pound of feathers will, all unspecified things being equal, be piled higher than a pound of bricks, right? So it will be further from the Earth, therefore the pull of gravity will be ever so slightly less on it, thus it weighs less.
"But wait Melvin", you say. "A pound is both a measure of mass and weight". Look, that's not my problem, use sensible units, I choose to interpret it as a question about the weight of a pound-mass.
" the rule where adjectives must go opinion-size-age-origin-color-purpose, such that "a beautiful little antique blue Italian hunting cap"
1) that is color before origin
2) Hungarian is not even Indo-European, and yet has this, suggesting it is not a language matter, a human thinking matter in general
I've fixed (1). I'm interested to hear from other multilingual people whether (2) is true in most languages.
I think in Russian opinion goes after size and age. The rest of it seems about right though I haven't tested extensively.
In French it is Beauty, Age, Number, Goodness, Size.
I recall learning that set of words to refer to adjectives that go before the noun at all, which is a different thing, does it also double as order? (what are examples of number here? for straightforward numerals it doesn't work (les trois petits cochons))
I think you are correct. It has been a few years since I studied French.
3) "hunting" in that context isn't an adjective; it's a gerund.
I'd say it's part of the compound noun "hunting cap", which is why it cannot be split up by adjectives.
I feel like "antique blue Italian" would be acceptable in pretty much any order. Is it just me?
I see the query rumbles on!
From 1955 letter to Auden from Tolkien:
"I first tried to write a story when I was about seven. It was about a dragon. I remember nothing about it except a philological fact. My mother said nothing about the dragon, but pointed out that one could not say 'a green great dragon', but had to say 'a great green dragon'. I wondered why, and still do. The fact that I remember this is possibly significant, as I do not think I ever tried to write a story again for many years, and was taken up with language."
I've seen many arguments that AI can't be truly intelligent. I've yet to see one that couldn't also apply to humans.
I think this is stating the obvious, but someone did need to state the obvious and I'm glad it was Scott with his trademark humour who did it, so I can just refer people here from now on.
Thanks for this beautiful argument for why we should start thinking about giving AIs rights.
This is funny, but seems a bit too sympathetic to AI, not just the industry but also the AI as itself. Good to remember the present appearance of stagnation does not mean we're out of the woods, and to highlight how absurd some of the "I can't believe you guys were worried about THIS" folks' arguments really are. But it is a memetic hazard to personalize AIs, and particularly hazardous to get humans to think of AIs themselves as the children of Man. The latter thought has apparently already corrupted numerous intelligent people in the industry.
Remember that if you birth *these* children, everyone dies.
Well for context, the end of the story is famously that God is dead
Entertaining. From the lowest intellects to the highest, we all have questions we'd like to have God answer. For myself, I think we obviously live in some type of created reality, not quite simulation, but not base reality either.
"Eight glasses of water" is genius
There's a fundamental difference you're ignoring. Even if you take the claim at face value that humans are as fallible as LLMs (not remotely likely), you'd have to acknowledge that they arrived at their capabilities through different means. LLMs through ingesting trillions of tokens worth of information and for humans through extrapolating a handful of examples. That extrapolation ability is several orders of magnitude beyond what any LLM can do, this means that even if you squint really hard and portray LLMs as human level intelligences, youd have to admit that they're useless for anything where they don't have millions of preexisting samples from which expand their latent space distribution. It's like calling the library of Babel the key to human advancement. It doesn't matter if it has every single insight humanity will ever need if you have to sift through endless amounts of semantic garbage to get to it
A human needs literally years of continuous multimodal sensory feed in addition to subject-specific RLHF before they are able to solve even simple maths problems.
That same reasoning implies that LLMs are also more efficient bullshitters. While we have to spend our first twenty years learning how to bullshit effectively, an LLM can do it after x-number of hours of training. ;-)
The information that a child ingests requires alot of abstraction to relate it to maths, levels of abstraction that LLMs can't match (LLMs need literal examples of formal addition to learn to add) . And they don't require years to figure out how to add. Most adding and subtracting is learnt almost automatically after figuring out the names of the numbers and relating them to their fingers. And after learning how to carry across place values children can do arbitrarily long addition given pen and paper. A 7 year old after seeing a few dozen examples of adding 3 digit numbers can add 7 and 8 digit numbers. LLMs ingest millions of examples of long addition and still struggle as you increase the digit count. The fact that they can't extrapolate a simple formula over longer and longer numbers even when those numbers fit neatly within their context windows shows that they have major deficiencies in abstraction and extrapolation which are necessary for reasoning
As I said, by the time they are 7, the 7 year old child has had literally seven years' worth of multimodal sensory input along with humans continuously deliberately teaching it things for that time.
Yes but that multimodal input is unstructured and random. Deriving understanding and knowledge from it is what's so impressive. If you strapped a camera and recorder to a baby and fed all that information into an LLM or any other neural network based AI for years or even decades it would be unable to derive any meaningful insight from it at all
The input is very much not unstructured or random. It is a feed of physical reality, with temporal and spatial continuity.
A human is not a passive observer. It is continuously interacting with its environment in a feedback loop. It is continuously performing experiments and observing the results as part of the input it receives.
It is not at all obvious to me what an inference model trained with all the sensory inputs and feedback mechanisms a human has access to, for similar lengths of time, would and wouldn't be able to do.
The laion-5b dataset is ~100tb in size, which is enough data to store about 10 years' worth of TV quality video. A model trained on this dataset can draw considerably better than a ten-year-old, with the advantage that training it takes much less than ten years of actual time.
>It is not at all obvious to me what an inference model trained with all the sensory inputs and feedback mechanisms a human has access to, for similar lengths of time, would and wouldn't be able to do.
I agree such a model would be impressive and probably a huge step towards AGI, but such a model doesn't exist. Not even a rough sketch of how to construct one. It is a genuinely hard problem of how to construct a reward function that captures the essence of being an agent casually interacting with the world, and not just a passive observer.
Aren't you kind of glossing over our entire evolutionary history (and that of our neurological predecessors) as a kind of pre-training?
That LLMs make human-like mistakes isn't in dispute, but it's that, to be *fully* useful, they have to be beyond making human-like mistakes. And they aren't.
I recently discovered that LLMs can be calendrically challenged. On Aug 28, I asked ChatGPT to list the Ukrainian drone strikes in the Moscow area since Aug 1. It produced a list, but the final entry was a strike it claimed to have occurred on Aug 30. When I challenged it, noting that Aug 30 was in the future, ChatGPT naturally apologized. It then explained why it had made the mistake. But it clearly doesn't understand why it made a mistake. With my feeble human brain, I intuitively understand space-time coordinates. ChatGPT doesn't seem to be able to do this.
> Why August 30 Was Incorrect
> The August 30 strikes referenced in the timeline occurred in other parts of Western Russia:
> That attack hit Pskov Airport, a fuel depot in Kaluga, and a microelectronics factory in Bryansk—none are within the Moscow region. Wikipedia
> Since the date hasn't happened yet and the impacts are outside the scope you specified, it shouldn't have been included.
Upon further investigation, the Wikipedia article it referenced showed that the drone strikes it referred to happened on August 30, 2023.
I would argue that LLMs make mistakes that aren't very human-like at all. They tend to make a completely different breed of mistake.
This is simultaneously both a cause for concern and a source of some comfort.
On second thought, they seem to make both human-like mistakes (i.e., bullshitting an answer from limited information) and non-human-like mistakes (i.e., being unaware of the spacetime continuum that our consciousness flows through).
Yes, that's definitely true. It's probably an overlapping Venn diagram where there are groups of mistakes made by LLMs, made by humans, and made by both. I'm sure someone somewhere has done this drill already. Maybe I could ask ChatGPT... ;-)
<chuckle> did anyone else come here directly from "Tao's blog?
Ironically, this is a point I've raised before: humans do not possess a *general* intelligence, as Scott had so aptly demonstrated. So when various people claim to be on track for developing AGI, they are not merely laying claim to a quantitative change, but a qualitative one. The kind of intelligence they advertise is not merely "smarter" than humans, but in fact radically different from the only intelligence we are able to recognize: our own. It is not at all clear that such a thing is even theoretically possible, let alone imminent.
I've been saying it for a while - people who dismiss LLMs by saying they are "just a statistical parrot" hugely underestimate the extent this is true for humans as well.
The risk of AI is not due to it's output flaws. The risk is how ruthlessly and efficiently the flaws will be acted on before they are caught. Rapid ideological RL of an AI-dominated government should frighten us all.
I'd just like to say that children are not stupid. They are inexperienced.
However, if you treat them like they are stupid, they will either become stupid, or make sure you regret the way you treated them.
Scott, I love you, but I'll still enjoy all the crow-eating after the enormous LLM hype bubble finally bursts.
Also, it's impressive that Yahweh's propaganda is still strong enough to claim the "God" moniker all to himself.
It is bursting already:
https://www.theverge.com/ai-artificial-intelligence/759965/sam-altman-openai-ai-bubble-interview
I don't know if this is the right way to think about it. The "tech bubble" "burst", but tech turned out to be very important! No matter how important something is, people can still put too much money into it too quickly, and then the bubble will burst!
(I've been trying to figure out how to think about crypto - everyone agrees there was a "crypto bubble" that "burst", but specific cryptos like Bitcoin and Ethereum are more valuable than ever)
BTW, despite of my criticism of the "meat" of the article, I'm delighted to learn that Adam apparently uses Sprint LTE, and that Iblis's preferred time to text him is 6:66 pm. It makes perfect sense.
count the number of bee's in 'beehive with 500 bee's
Good post, but I'm left unsure whether it's supposed to just be funny and thought-provoking, or whether it's making a strong specific claim.
I'm already on-board with the idea that we don't know what AI will become capable of and chronic cynics are missing the point, however there are a few key points where the analogy breaks down for me (IF it's meant to make a rigorous point, rather than just be fun, it's definitely fun):
1. To the extent this argument touches on whether AI could be sentient, the big reason to think other humans are sentient is by analogy to ourselves. We are pretty sure they are the same type of physical process as we are, and we have direct access to the knowledge that our own process produces a sentient experience, so Occam's Razor suggests that other processes of the same type probably do as well. AIs producing similar outputs and having some metaphorically similar architecture is weak evidence that they could produce similar sentient experiences, but it's much much weaker than the argument for other humans with basically the same physical process and origin.
2. To the extent this is about whether AIs are capable of impressive intelligent behavior despite individual examples of them acting dumb, the difference is that for humans we *already have* the examples of those impressive intelligent accomplishments, whereas for AI we are largely still waiting for them. (yes, every month they do some new thing that is more impressive than anything they have done in the past, but nothing that is truly outside the training set in the same sense that human inventions sometimes are).
3. To the extent that this is about whether AIs could ever meet or exceed human ability, note that humans still *haven't* met the intellectual accomplishments of God or the Angels, so the analogy doesn't quite work there.
This was sooo good
This was very, very funny. Especially the bit about the snake.
"God: Sigh. Was it an Education PhD?"
:-)
This is very, very good. Thank you!
Reminds me of https://accelerateordie.com/p/symbiotic-man
"Does this rule out humans?" has long since become one of my go-to first filters for "Does what this person is saying about AI even have a chance of being worth engaging?"
Like I said above, I think that what Scott implies about humans is correct: we are not a "General Intelligence". So in this sense, I suppose arguments against AGI also rule out humans, though I'd argue that the implication flows in the opposite direction.
In principle I can understand and somewhat sympathize with this POV. I don't endorse in part because it seems to unnecessarily redefine common usage of important terms, and is also likely to turn off anyone who doesn't already agree with it.
In practice, I usually encounter "X isn't and can never be AGI" arguments in the context of people claiming the arguments mean there is some set of ways in which it can never compete with humans, never replace humans, never be dangerous to humans, etc. That is the sentiment I feel most compelled to push back against, I don't think the approach you're talking about accomplishes that very effectively. People who put in enough time and attention to think what you're thinking are probably already aware that AI doesn't need to be totally general or ASI to be massively disruptive or dangerous.
Oh sure, even present-day LLMs can be massively disruptive and dangerous, especially in the hands of careless or thoughtless humans. We are already seeing some of these dangers manifest, e.g. with the gradual disappearance of reliable information online, "vibe coding" leading to massive systemic security problems and other bugs, college students losing the ability to read, etc. But these dangers, while serious, are a far cry from saying "AI will turn us all into paperclips" or "anyone with access to AI could create a custom virus to kill all humans" or anything of the sort.
Agreed. However, I'm not sure what I'm supposed to take from that? If current or future AI could do those things, it would still be just as easy to use the rules-out-humans anti-AGI arguments against them. This does not actually help us answer the question of, will AI reach a point where it is sufficiently superhuman at sufficiently important things to be a meaningful x-risk. And that is usually what I want to see people acknowledge when biting this bullet in *either* direction.
> This does not actually help us answer the question of, will AI reach a point where it is sufficiently superhuman at sufficiently important things to be a meaningful x-risk.
I think there's a difference between asking, "could AI reach such a point in some fashion at some point in the future" (the answer is of course "yes"); and "is AI an unprecedented risk right now as compared to the humans themselves, or will it become such in the near future" (the answer is "no"); and specifically "will AI acquire quasi-magical powers that defy the laws of physics as we understand them to day and then use those powers to kill all humans" (the answer is again "no").
Which claimed powers defy physics? Physics allows quite a lot.
Perhaps. My interest is more in understanding the nature of intelligence than in arguing about how dangerous AI might be.
Personally I think it's time to ditch the concept of "AGI" because as machines become more intelligent it becomes more apparent that there's no such thing as "general intelligence". When machines were far from human intelligence it made sense to imagine that human-like intelligence was general because we had nothing to compare it to, but as they start to pull up alongside us we realise that human intelligence is idiosyncratic, good at some things and bad at others.
At some point, asking whether machines are as intelligent as humans becomes as silly a question as asking whether aeroplanes are as good at flying as hummingbirds.
Ditching the concept of AGI helps us understand the dangers of AI too; they don't need to be as good as humans at everything in order to be much better than humans at some things.
Could I have that Linda the Feminist question written out in plain text, so I can copy it? I'd love to know how many people I know would get that question correct/incorrect.
I copied it from the italicized text in https://en.wikipedia.org/wiki/Conjunction_fallacy
This is hilarious, but I’m gonna be ornery today and point out that this whole post is a category error.
Humans were not intelligently designed. We evolved, as all other living beings on earth, and evolution equipped us to do one thing: survive long enough to reproduce. That’s it.
For the majority of our history as a species, that meant living in small hunter-gatherer groups, so we evolved to be really good at things like “facial recognition,” “spoken language,” “keeping track of favors/grudges/who’s a friend and who’s a foe,” and really sucky at things like “mathematics beyond counting one, two, many,” “formal logic,” “drawing a complicated map from memory complete with spelling all the country names correctly,” etc.
Given what we evolved to do, it’s frankly astonishing that a Terence Tao or a John Williams is even possible at all, especially given hard constraints like “it takes this many kcal to keep the human brain running” and “this is the maximum human brain size at birth given the width of the birth canal.”
Come on, Iblis, give us some credit for how much we humans have been able to do with so little!
Funny article, but I took issue with a few points;
1. Neighbor and Racism
The screenshot oversimplifies the biblical idea of “neighbor” into modern universalist morality. In reality, the Hebrew Bible consistently distinguishes between insiders and outsiders:
Neighbor/kinsman meant members of your own people.
Outsiders could be treated as guests under certain conditions, but there were other groups (e.g. Canaanites, Amalekites) where the explicit command was destruction or enslavement.
Laws themselves are stratified: Hebrew slaves must be released after a term; foreign slaves could be held permanently (Leviticus 25:44–46).
So the image’s framing (“you’d never hurt your neighbor—what about someone of another race?”) is dishonest. The biblical worldview explicitly had a hierarchy of moral obligation: family > tribe > guest > enemy. Pretending otherwise is anachronistic.
2. The Kimono and Progressive Contradictions
This one is a fair satire of progressive moral incoherence. But it stacks a false equivalence: it pretends the same person who excuses tribal prejudice is also the same person obsessed with cultural appropriation.
In reality, those two positions come from different camps.
The “tribalist” position (greater concern for kin than outsiders) is rational, even if harsh. It matches both scripture and evolutionary logic.
The “progressive” position is incoherent. They’ll condemn a harmless act like wearing a kimono—something most Japanese actually welcome—while simultaneously pushing mutually contradictory agendas like mass Muslim immigration and gay rights, or saying “whites can’t be victims of racism” even in countries where whites are minorities (e.g. South Africa).
This worldview collapses into pure virtue signaling, with no consistent principle behind it—just an oppressor/oppressed narrative used as a moral club.
3. The Garden of Eden
The text exchange turns the serpent into a clown saying “trust me, I’m a snake.” But in the Genesis story, the serpent actually makes a reasoned claim:
God: “In the day you eat of it you will surely die.”
Serpent: “You will not surely die… you will be like gods, knowing good and evil.”
Eve eats, and she does not die that day. She does gain the knowledge of good and evil. In other words, the serpent told the truth; God’s words were false.
From there, the Elohim say: “Behold, the man has become like one of us, knowing good and evil. Now, lest he reach out his hand and take also of the tree of life and eat, and live forever…” (Genesis 3:22). They exile humanity—not because the serpent lied, but because he revealed the truth that humans could rival the gods.
This theme echoes at Babel: humanity’s unity and creativity threaten divine status, so God deliberately scatters them. The pattern is of jealous, insecure deities suppressing human potential.
4. The Core Point
When you strip away the meme humor and look at the texts:
Scripture affirms differentiated moral concern (kin vs stranger vs enemy).
Progressive ideology collapses into incoherence, unlike the hard-edged consistency of tribal preference.
The Eden story shows the serpent persuading truthfully, and God reacting with anger when his lie is exposed.
I know the point of the article was humor, but I dislike letting subtle inaccuracies like that go unchecked.
"The biblical worldview explicitly had a hierarchy of moral obligation: family > tribe > guest > enemy"
Don't stop there; keep reading on! If the message you took away from your bible reading is tribal preference over love for strangers and enemies, you walked away way too early. When you get to the new testament, you will find that this is very deliberately reversed; not only in the parable of the good Samaritan to which Scott alludes, but also in passages like Matthew 5:43-48
That's a fair point, so let's unpack it a bit.
When you cite Matthew 5 and Galatians 3 as if the New Testament erased all distinctions, that’s not really accurate.
Jesus didn’t abolish the old tribal laws — he said not a jot or tittle of the Law would pass away. What he did was shift the dividing line. Instead of Jew vs. Gentile, the key split became believer vs. unbeliever. That’s why he told people to love him more than their own families, even to the point of “hating” father or mother if they refused to follow him. That’s also why the first Christians lived like a commune, selling everything and sharing it — but only inside the community of believers. Outsiders weren’t treated the same. His ethic wasn’t universalist; it was faith-based.
And Paul shows the same pattern. In Galatians he says “neither Jew nor Greek, slave nor free, male nor female,” but he clearly didn’t mean all differences vanished. He still gave distinct rules for men and women, told slaves to serve their masters and masters to treat them justly, and continued to recognize Jew and Gentile as real categories. The point was equal access to salvation — not equal roles or equal treatment in every aspect of life.
So when you look at both Jesus and Paul together, the New Testament is about restraining cruelty, tempering vengeance, and promoting generosity — but it does not dissolve distinctions between family and stranger, believer and unbeliever, male and female, slave and free. Those hierarchies of concern are still there.
The “joke” in the snake quote was actually a reference to LLM jailbreaks that break up or encode / armour the text in layers of obfuscation that skip the text past either the processing layers that have any defensive refusal checks programmed into them, or past external business-layer logic word filters.
The analogy here was supposed to be that the Biblical snake was not *only* making a rational utilitarian appeal (that could be believed in a state of ignorance), but also doing so in such a way that this argument was able to slip past any guardrails preventing it from reaching the reasoning core of the humans — in this case not actually being due to verbal cloaking (that part’s just for laughs) but meaning something more like, God failing to stop the snake from entering His garden in the first place, due to its subtle (snake-like) approach to entry.
It’s an oddly nice parallel, given that these hundreds-of-billions-of-parameters models are run in literal “walled gardens” with gatekeepers trying to keep out “snakes.”
"while simultaneously pushing mutually contradictory agendas like mass Muslim immigration and gay rights"
How are these agendas contradictory?
Let's look at what Islamic scripture and Sharia law actually say about homosexuality.
1. The Qur’an
The Qur’an contains several passages referring to the story of Lot (Lut), very similar to the Old Testament:
Surah 7:80–84, Surah 26:165–166, and others describe the people of Lot as engaging in “lewdness such as none in the worlds had committed before” — specifically interpreted as homosexual acts.
The passages condemn this behavior, and God destroys the people of Lot for persisting in it.
While the Qur’an itself does not lay out a legal penalty, the tone is clearly one of condemnation, not acceptance.
2. Sharia Law (Islamic Jurisprudence)
When it comes to actual law, the hadiths (sayings of Muhammad) and later jurists are more explicit than the Qur’an:
Several hadiths prescribe the death penalty for homosexual acts (e.g., “If a man comes upon a man, they are both adulterers” — Sunan Abu Dawud 4462; and “Kill the one who does it, and the one to whom it is done” — Sunan Ibn Majah 2561).
Traditional Islamic schools of law (Hanafi, Hanbali, Maliki, Shafi’i) all held homosexual acts to be major sins. Most prescribed death by stoning or other punishments, though the exact methods differed.
Even in modern times, in countries where Sharia is enforced (e.g., Iran, Saudi Arabia, Afghanistan under the Taliban), homosexual acts are still punishable by death or severe penalties.
3. Why This Matters to Your Question
Mass Muslim immigration and gay rights are contradictory agendas because Islam — at least in its traditional form — explicitly condemns homosexuality and prescribes harsh punishments for it. Promoting both at the same time puts two belief systems side by side that cannot logically coexist:
Western liberalism says homosexuality is a right to be celebrated.
Islamic law (rooted in Qur’an + Hadith) says homosexuality is a crime to be punished, often with death.
That’s the contradiction.
And not merely an abstract one. Many gays are still being killed today under Islam, and at a bare minimum are highly condemned by Muslims.
Even just for basic acceptance the numbers are dismal:
In a 2013 Pew global poll of predominantly Muslim nations, overwhelming majorities opposed societal acceptance of homosexuality:
Jordan: 97% opposed
Egypt: 95%
Tunisia: 94%
Palestinian territories: 93%
Indonesia: 93%
Pakistan: 87%
Malaysia: 86%
Lebanon: 80%
Turkey: 78%
So claiming to champion gay rights while mass importing one of the most anti-gay demographics who will vote against those rights and commit hate crimes against gays is nakedly self contradictory. And the more Muslims there are, the more strong the opposition.
Kudos on absolutely nailing Dwarkesh.
> whether they will or no
I see what you did there.
🤣 this is just about the funniest thing i’ve ever read! Bravo!
What is a thing that thinks? A thinking thing
What is a thing that is thought? A thinging think, or a thought thing
Which is AI and which is us, I wonder...
> God: That was the 4o model. It’s been superseded,
Was this intended to sound like Christian language?
No. It was intended to sound like OpenAI, which had massive public outcry when they replaced the very sycophantic GPT-4o with the less-sycophantic GPT-5, to the point where they brought back GPT-4o for paid subscribers.
Wow, I lived to see the worst ever ACX post. Kind of didn’t think it would happen in my lifetime.
Very draw, LOLLOLLOL ha ha ha et cetera. But here is quite a big problem for the joke working as an analogy: there is no God. God did not “upscale“ the chimpanzees. It did not happen that wayThink about why that really matters.
« and then it breaks down even slightly outside the training distribution, because humans can’t generalize in meaningful ways.»
I don't get this example. Why would it be "meaningful generalization" to go from "I would not hurt my neighbor" to "I would not hurt a stranger with a different skin color"? That sounds about as "reasonable" as going from "I eat tomatoes", therefore "I eat people", because both tomatoes and people are edible. There are specific reasons for why you eat tomatoes and don't hurt your neighbors, that may or may not apply to people and strangers respectively. Some kind of generalization would be flawed reasoning.
What if your neighbour is a stranger with a different skin colour?
I thoroughly enjoyed the debate. But the central thing missing from AI, and present in BI, is judgement. God has it ("saw that it was good") and Iblis probably has it. When we have a machine that can evaluate something else, in a Turing-complete sense (not just evaluate one specific class of things), then that machine can become self-improving.
If given the tools, that is. AFAIK, no LLM can actually manipulate objects; they must be hooked up to some kind of robot somehow. If we have a machine to which we give the ability to manipulate objects as well as a human hand, and that machine can also judge whether its actions are good or bad, then it will have the ability to modify itself and its environment faster than humans could.
When God creates Man in his own image, Man has the potential of asymptotically approaching God-like virtue but never reaching it.
When Man creates AI in his own image, we can ask whether AI will asymptotically approach Man's abilities but never reach them.
Or whether it's possible for Man to create something that _exceeds_ Man's capabilities and approaches God-like virtue, perhaps moreso than Man ever will.
It would seem this depends on the setup. In some scenarios we've created AI systems that exceed human abilities (i.e. in Chess and Go), namely those with verifiable success criteria / win conditions. One would imagine this could be the same for math or programming. On the other hand, in settings where the only supervision and judgment comes from humans, it's not clear what it would mean to exceed human capabilities without bound, and fill up the gap between us and God...
> First, "AIs are next token predictors" is not really a description of their methodology, it's a description of their goal
I actually don't think it's a good description of either.
*Foundation* models (a precursor stage of training) are next token predictors – they are trained to guess what word comes next in large corpora. However, to turn a foundation model into a "chat" model like ChatGPT/Claude/Gemini, you do RLHF to shift the distribution of the LLM outputs towards giving correct, helpful answers and responding in a question/answer 'assistant' style (the result is called a 'chat' or 'instruct' model). This doesn't look like giving it a huge corpus to imitate, it looks like fine tuning it by showing it a comparatively tiny set of examples of 'good' vs 'bad' answers and nudging the LLM's weights overall to shift it towards being the kind of model that generates the good examples rather than the bad. It's not being asked to 'predict the next token' anymore – you're simply rewarding or punishing it for being the kind of model that considers the positive/negative sentence examples to be good outputs in aggregate.
After RLHF, it's not really true to say the model is a predictor anymore. It draws its output from a statistical distribution conditioned over the input (its context), but there's no ground truth distribution that it's trying to imitate. I don't think it's meaningful to say 'predict' unless there's a fact of the matter as to whether your prediction ends up being right or wrong. It would be more accurate to say that it chooses words that it thinks are a 'good fit' to the previous context.
People do sometimes use the word 'predict', but it's a bit of a technical usage – the LLM is predicting which word would occur next in a hypothetical perfect reference answer (which does not exist and never will).
To give a concrete example of foundation models vs RLHF-ed instruct models:
----
Q: What is the capital of France? What language do they speak in France?
A (Qwen 2.5 0.5B *): What is the currency of France? How can I learn French? How can I improve my French? What is the capital of Australia? What language do they speak in Australia? What is the currency of Australia? How can I learn French? [...]
A (Qwen 2.5 0.5B Instruct): The capital of France is Paris. In terms of language, French is the official language of France and is spoken by most people in the country. However, some regions may also use other languages such as Catalan or Spanish.
(*) This is a foundation model, i.e. it has been trained on next token prediction without any RLHF.
----
I think this gives a good example of why it's fair to say that foundation models are next token predictors, and why that is not as good a description of RLHF-ed instruct models like ChatGPT.
I loved this post. However, I’m the target audience as someone skeptical about the claims of AI’s future, and I didn’t have my thinking shifted much.
I find it useful to ask those two questions: “Will the current architecture for AI have practical limitations? If yes, what are they?” I have been surprised many times about the continued advances of AI, which showed I answered the second question wrong. But surely the answer to the first question is “yes.” Do we really think we invented the algorithm for God on the first outing? If not, we should all be skeptics, asking where we should expect that sigmoidal curve to bend down again - to understand what those limitations are, so we can get those most use out of these very powerful tools. That is, they’re more useful the more we understand what they can and can’t do well.
I wrote an essay doing pretty much the same idea here: https://collisteru.net/ni_overhyped/ni_overhyped/
A few differences are that, in mine, the frame story is a letter by Satan discovered during an archeological expedition, not a podcast episode.
It was interesting seeing the parallels that Scott and I both noticed, like that under theistic evolution, humans can be considered similar to scaled-up chimps, just as AIs are scaled-up neural networks.