IQ is not a real thing. An IQ test is a vanilla measure of cognitive function that has been misunderstood to be measuring some fundamental underlying characteristic called "IQ" which does not exist. Cogntive function is important but you don't need an IQ test to measure it, and being slightly better at rotating shapes than the median does not make you the master race.
Re: cryptocurrency it's worth remembering that any success it has had in poor countries is a direct result of Know Your Customer laws and other restrictions on the banking system by the western world. In a world where anyone could simply avail themselves of a numbered account with an email address the appeal of cryptocurrency would have been limited to some ideological purists.
That's not to defend the role that crypto plays just to observe that it's really a question of regulation more than technology.
> Cryptocurrency has become an important part of poor countries’ financial infrastructure, so much so that I think it should objectively be considered a huge tech success story.
I'm sorry to keep beating this horse, but I don't really think this is true. I this crypto proponents *claim* this is true, because it matches their ideal theoretical usecase ("what if we didn't have any sort of reliable central bank system and had to rely on decentralized finance for everything"), while also conveniently being ideologically appealing ("look we're helping the global poor") and happening very far away so it's hard for anyone in a first world country to really verify. But I haven't seen any non-crypto people actually talk about crypto succeeding in third world countries, which makes this look exactly how it'd look if it were empty stories being sold by a bunch of determined grifters who'd thrown a bunch of money at the problem and had the inevitable occasional-thing-that-looked-like-a-success-story or broad trend that could be made to look like a success by careful selection of metrics.
Once we remember that these forces exist in strength around crypto and adjust for them, I don't think the case that it's been helpful in the third world holds water.
I am so glad you wrote this response. The Banana post was very bad and yet was so positively received.
It’s easy to forget nowadays how good of a scientist Kahnemann (and Tversky!) really were. If you look at the Many Labs replication project results, you see not only that their research stands out by replicating, but often replicating with stronger effect sizes than the originals. Remember they used to do evaluations for the Israeli army at a time when they were constantly engaged in wars against seemingly horrible odds - there was a lot of skin in the game ...
And precisely because the effect sizes are so large, you can often replicate them in classroom contexts. For example, Anchoring, one of my favorites, which you can see play out in basically any sort of negotiation.
I'm going to admit that the fact that this person's name is Banana makes me significantly more likely to read their writing. Is that some sort of cognitive flaw?
The idea that people are "mindless sheep“ has been around forever, like you say, this concept of enlightenment and awaking from a sort of lifelong fugue state is heavily used in cults. I find it interesting that lately the trend is to use video game logic to describe it (non-player characters), as if player characters have more agency, when really they're still following a prescribed path set forth by the devs or at least conforming to the rules of the game. I'm also curious how actual sheep compare to more or less scripted NPCs. I certainly feel more empathy for sheep.
I think most of the time when we make choices and things turn out well for us, we don't tend to care about how conscious that choice was. Driving on autopilot from work to home every day tends to be fine, our brain being capable of quietly making the multitude of tiny choices from point A to point B. But if said thoughtless operation results in something horrible (an accident, or just missing a new detour) we easily blame ourselves for our unthinking motions. But it's human. Constant questioning and awareness of why you do everything you do sounds more like an anxiety disorder than wakefulness, doesn't it?
I think the core issue is that many psychologists focus their attention on studying phenomena like biases or whatever, which are "secondary effects", artifacts of a constellation of mechanisms that approximate rationality, when really it would be better to study "primary" effects, such as rational choice, because the effect sizes are going to be much bigger, making the results more robust (less sample size needed for bigger effects) and more applicable (more uses available for bigger effects).
That said, small-scale stuff like "people would rather eat an apple than touch a hot stove" is probably too obvious. But I think this can be solved by looking at bigger-scale stuff that would be harder to observe in everyday life, for instance performing big factor analyses, mapping out measurement methods and developmental trajectories, collecting wisdom, and so on.
> Cryptocurrency has become an important part of poor countries’ financial infrastructure.
Can someone explain why they can't just use dollars? Is there no online service that allows people from poor countries to open accounts? Sounds pretty trivial to make.
(Yes, that would make it centralized, but most people own crypto through centralized platforms anyway.)
I think automaticity has to be evolutionarily selected for.
In our ancestral environment, it wouldn't make sense for our brains to weight every piece of sense data equally. Our eyes have to jump straight to the tiger. Our ears have to be attuned to things like screams.
Thinking about things is expensive. It paralyzes you. Sure, often it's the best thing to do—particularly as the threat of instant death recedes from society. But I'm learning not to look down on people who don't plan everything out. Honestly, for ANY mistake humans commonly make, there's probably a big evolutionary pressure holding them at that point.
Funnily enough, for the religious word priming, I got Christina for the second one and thought it was out of place. Though maybe that was because it wasn't far enough down to get the pattern yet.
For quite some time now, I’ve been looking for a survey paper outlining which areas of psychology have replicated/failed to replicate. (The best I have found is estimates of replication percentages in cognitive psychology and social psychology.) Has anyone seen such a paper?
Another factor you're not mentioning is that many biases are at least partially adaptive in practice. The religion world scramble for example: In most natural situations, you'll just be discussing (in whatever media) something, and it's advantageous to understand what the other person is talking/wants to talk about. Unless someone deliberately wants to trick you, they won't suddenly switch topics, and there is also no advantage to them (and little disadvantage to you except a mild annoyance). It's related to Bayes as well, we've trained the prior "topic is important" very hard for all our life, and unless we train these kind of word scrambles every day we will not get rid of it.
I don't think automaticity is a good label for the ideas Literal Banana talks about. Situationism is the problem: https://en.wikipedia.org/wiki/Situationism_(psychology) There was a "heroic" age of social psychology from the 1950s to the 1970s when famous studies like the Asch conformity experiments, the Pygmalion effect, the Stanford prison experiment, the Milgram experiment, etc. seemed to show that environmental circumstances have a huge effect on behavior. Those studies were generally fake or greatly overinterpreted, but social psychology grew into a big field and has been stuck in the situationist paradigm ever since. When you combine this false view of human behavior at the core of social psychology with methodological standards where running an experiment on a few dozen undergraduates and then fiddling with the data until you get p<0.05 constitutes scientific proof, there is nothing surprising about the replication crisis.
Looking at the Wikipedia article on Conjunction Fallacy, the prize example seems terrible (at least to this literal-minded idiot):
"Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.
Which is more probable?
1.Linda is a bank teller.
2. Linda is a bank teller and is active in the feminist movement."
The 'correct' answer is "Ha ha, you fools, you all picked No. 2 but you're wrong!" Why? Because there's nothing there to say Linda is a feminist as well as a bank teller. Yeah, but there's also nothing there to say she's a bank teller. So the real proper answer is "Neither is more probable" or "I can't tell, there is insufficient information".
If you're asking me to estimate if Linda is a bank teller *or* a feminist bank teller, without giving me any information as to what her job really is, then yeah I'm taking the description of Linda's character that you *did* give me and extrapolating from that that she is likely to be a feminist.
I'm not saying the thing *isn't* a fallacy, just that it's a dreadful example to use and ordinary people are not idiot machines for picking No. 2 over No. 1. You're asking them to make a judgement, you're not giving them the full information, so naturally they will make a decision based on the information you did give them, which is that Linda is the age range and background to be a feminist. We still have no idea if she's a bank teller, surf instructor, or Cordon Bleu chef, but you're the one proposing that she's a bank teller.
Imagine reading something that was said to have been posted online about JESUS NEVER EXISTED RELIGION IS HORSESHIT CHRISTIANITY IS NUTS THERE ARE NO SUCH THINGS AS GODS OR DEVILS BELIEVERS ARE SHEEPLE SCIENCE IS REAL BELIEF IS NOT.
"Is it probable that this person is an atheist and a teenager?" Well gosh, George, how could I possibly tell from what you've given me there? Maybe it was Pope Francis posted that on the Vatican website. I wouldn't want to be Conjuncting any Fallacies, now would I? 😀
I'm having a hard time figuring out where the major points of disagreement are with the Carcinisarion post. I read the banana as saying that the existence of cognitive biases led to a whole body of research that advanced the notion that human beings are always and everywhere slave to an innumerable number of unconscious biases, so much so that this version of automaticity become indiscernible from woo.
Maybe I'm being too charitable, but I read the banana as saying that it's this woo model of automaticity that isn't real, not that the underlying cognitive biases.
As for the alternative framing, I think the phenomenological approach as a lot going for it. For example, here's my phenomenological explanation for the first word scramble game: I, like many Americans, started taking standardized tests at a young age. Sucess, or lack of success, on those tests played a deterministic role in how the first half of my life played out. So, when presented with a question in this form, I immediately treat it as some kind of exercise in pattern-recognition, because that's how most test-taking strategies work. In a sense, I've been primed, but I think it's more accurate to say that solving puzzles has been somewhat central to my existence; therefore it's only natural that I approach these sorts of tasks in this manner.
Actually, I'd be curious to see what happens when the same word scramble is presented to people without the same history of standardized test-taking or for whom such tests where never of great importance. Perhaps their answers would be less conforming to the pattern.
I'm still ruminating about your last point, how we'd all probably have thought slavery was fine if we were born in the 1700's in the South. Yup, no denying it. I've thought about that a lot myself. There's this hard-to-resist fantasy that I, in my old-timey dress and lice-infested hair (which I would not be distressed by because everyone else had head lice too) would jump up and shout "Fuck this shit! We are treating these Africans like they are animals, with no rights and no feelings. Wake up, you filthy 18th century fools! They have all the same rights we do." But I know I would not have. At most I would have had some private wonderings about what it was like to be them, and whether my husband was a bit too hard on them.
There's a book I admire called How to Think, by Alan Jacobs (a better title would have been How to Be Fairminded) that discusses this issue at length. Jacobs' view is that while we experience our thinking about important issues as a private weighing of this and that, that's an illusion. Thinking is actually a collaborative, group activity, and one by-product of that is that people in the same culture tend to see many of the same things as obvious truths. He's pretty good on the subject of trying to free yourself from your bond with these socially acquired postulates, and on the related subject of working with your mind's instinctive hatred and devaluation of the outgroup. But I think he'd agree that our ability to do those things has limits.
I often feel that there's this buried subtext in Rationalism that it will enable off of us to shuck off the trance of belief in the truisms of our era, and think with perfect freedom and clarity. I think that Rationalism certainly helps, but there are limits. Jacobs talks about the challenge of thinking more freely and clearly in quite a different way from Rationalists, and I find his approach very helpful. He comes at it more from the direction of working with one's anger and self-righteousness -- you know, the feelings cluster that is stirred by an encounter with a member of the outgroup. He talks about ways to quell rage. One that works really well for me is to do an introspective hunt for ways I have made the same error the infuriating outgroup member has done. I can always find one -- it may be in an error about a different topic, it may be a smaller version of the the error (or it may not!), but there's always one. The point of doing that isn't to hate yourself or let go of your objections to the outgroup members outgroupy idea. It's to break the illusion that they are a whole different, and lower, order of person. And also I can usually remember how I arrived at my version of the error, so that helps me understand how the outgrouper arrived at his.
For instance very early in this thread somebody jumped in aggressively with something like "Nope, nope, what you said about IQ being meaningful is false, IQ is a worthless concept." And it was irritating and fact Scott ended up banning the guy after his rage to content ratio got so high there was no content. But I've said some version of what that guy said, and I did it for the exact reason Scott suggested to the vehement anti-IQ poster: The topic of IQ makes me feel guilty and uncomfortable. Being smart is an important part of my identity and always has been and that seems sort of ugly. I didn't become smart by effort, I was just born that way. If I knew someone who had been born wealthy and thought of her wealth as an important part of her identify, I would disapprove of that. And yet being smart is a part of a person that can't be removed the way wealth can. It's more of a real part. I handle the topic of IQ in a more honest way these days, but there certainly was an era when I would yammer on about how IQ was not as meaningful as people think.
I suspect the boom era of priming studies popularized by Malcolm Gladwell had to do with academics trying to make big bucks off marketers by assuring the MBAs that marketing / advertising wasn't an art, it was a science. You didn't have to be creative and work hard to come up with effective advertising, you could just prime consumers into buying your product with scientific tricks.
My guess is that in reality you can prime some of the customers some of the time into buying your stuff, but that lucrative priming techniques cycle in and out of effectiveness.
That was, in effect, more or less, Warren Buffett's explanation for why he bought 20% of the Coca-Cola corporation: it's really hard to create a brand as good as Coke's, so of course he wants to own a big chunk of a super-brand created by a century of smart, expensive, and inordinately effective investment.
The conjunction fallacy is not a fallacy. to think it is, is to commit the fallacy of thinking that natural language statements reduce to logical propositions or, to put it another way, that judging the probability of the truth of tthe natural language statements about Linda is exactly equivalent to deciding which columns to tick on a spreadsheet about Linda or where to place her on a Venn diagram.
The point is Paul Grice's theory of conversational implicatures (and incidentally this is a free-standing theory, not an ad hoc way round the Linda problem) which states inter alia that conversational statements come with the implication that they are maximally informative: to say "Linda is a bank clerk" is to say "Linda is a bank clerk *and that is the most interesting thing I have to say about her*. A moment's thought shows this to be true: if you met an old mutual acquaintance, who told you she was a b.c., and subsequently found the acquaintance also knew all about the activism, you would ask why he had deceived you.
Here's another way natural language is not equivalent to logical propositions: I don't know if this works in american english, but in British English a rude way to question a claim of status, e.g. "I am the Professor of philosophical logic at Harvard" is to say "Yeah, and I am the Queen of Sheba." To think this means I would tick the "Queen of Sheba" box when filling in a spreadsheet is to make a fairly obvious mistake. So what we have in logical terms is A implies not-A and also not-B, which is impossible in logic but in practice easily understood.
This is not difficult. I would be a bit concerned to learn that the fallacy is still commited when the claim is amended to "Linda is a bank teller whether or not she is active in the feminist movement", so am disappointed to read in wikipedia that "If the first option is changed to obey conversational relevance, i.e., Linda is a bank teller whether or not she is active in the feminist movement the effect is decreased, but the majority (57%) of the respondents still commit the conjunction error." This shows that 57% of people are profoundly stupid, but also doesn't destroy my point because the new wording takes us out of the realm of conversation - it is not something anyone would ever say - and nobody is expecting the rules of conversation to be followed.
Nice post, thanks. Nitpicking one of your examples:
There is a pretty good case that loss aversion doesn't exist in any meaningful sense: it models declining marginal utility, and to the extent that it 'overreacts', that's because it is accounting for ergodicity. It's perfectly rational to be ultra-cautious in a non-ergodic system (like, say, life) where the linear average probability is different to the ensemble probability.
I keep coming across other central examples of biases that fall apart in similar ways (sunk costs, availability bias, confirmation bias, planning fallacy) and I'm not even looking!
So I think a good synthesis here is to be aware that apparently exciting cognitive 'biases' turn out to be rational behaviour under closer inspection, and especially outside the confines of highly contrived lab environments. I got totally suckered by this myself, so I'm not pointing fingers. But I'm not sure if e.g. the rationalists ever walked back all the hype around this stuff.
> I can’t help wondering if there’s some understanding of of “automaticity” or “being less automatic” that could have helped 1700-me question my belief - or wondering what equivalent automatic beliefs I should be questioning today.
My starting point for answering this for myself is Holden's systemization + thin utilitarianism + sentientism (https://www.cold-takes.com/future-proof-ethics/), which I think you've likely read, although I'm not sure if you've already discussed it on ACX / slatestarscratchpad etc.
While I suppose "automaticity" is real, I think you can also doubt whether psychologists have shone any light on it. I imagine magicians (really, illusionists) have known some of these for eons.
> Although PURGATORY is a slightly (?) more common word than PREROGATORY
You're misstating this. For example, the Corpus of Colloquial American, at 1.001 billion words, includes 1948 instances of "purgatory" (a 0.000002% rate!) and 0 (zero) instances of "prerogatory". "Prerogatory" has no entry in Merriam-Webster or in Wiktionary. It is not a valid solution to the question posed for the obvious reason that it is not an existing English word. "Purgatory" is not just several thousand times more common than "prerogatory", it is infinitely more common.
When you ask people to do what can't be done, you're going to get a lot more nonsense than usual.
(My results on the words of interest: GLEAN / GOD / ???????? ("purgatory doesn't have two Os, or an E") / ???????? ("______ STAGE?"). At the moment of composing this comment, I've realized that that last one is PEARLY GATES, but STAGE didn't help me get there.
You didn't mention what #4 is supposed to be; I got SPRITE and don't see another option, but it certainly isn't on theme...)
Interestingly (and completely irrelevant for the discussion at hand, sorry) gravity does imply a maximum possible height for a skyscraper on Earth. Which is kind of trivial when you figure it out but still came as a mildly interesting surprise to me. Calculations this maximum height is left as an exercise for the reader.
"in 1700, everyone thought slavery was fine, even though now in the 2000s everyone hates it."
This is tangential and possibly forbidden by culture-war rules (I am British, so don't fully understand the US rules) but I would question that, absent 1700 polling data, sermons, letters to the press saying it was all fine. etc. I read what I thought was a well-sourced claim which infuriatingly I cannot now track down that in the 1550s when Sir john Hawkins put the whole slave business plan to Queen Elizabeth 1 she initially said "What a revolting idea, no good will come of that." Then he showed her the profit margins... It's often said that people were OK with it because it had historically always been there, but 1. the wholesale shipping of people as commodities between continents was new and 2. slavery was, then, ancient history - there had not been slave-owning Englishmen for twice as long, then, as there have not been slave-owning Englishmen now. I think more likely there was disconnect and wilful ignorance, same as today's middle classes are probably against working conditions at Foxconn and murder and torture in Central America, but choose to keep the iphone and the coke habit.
“It is a profoundly erroneous truism, repeated by all copy-books and by eminent people when they are making speeches, that we should cultivate the habit of thinking of what we are doing. The precise opposite is the case. Civilization advances by extending the number of important operations which we can perform without thinking about them. Operations of thought are like cavalry charges in a battle — they are strictly limited in number, they require fresh horses, and must only be made at decisive moments.”
My recollection of majoring in Finance at MBA school four decades ago is that after the enunciation of the Efficient Market Hypothesis by Eugene Fama in 1970, there was a strong effort to identify irrational stock market anomalies, but after they were discovered, they tended to disappear. For example, it was discovered that the stock market tended to go up (or perhaps down, I forget) in January. But once investors heard about this January bias, it more or less stopped happening.
In other words, the Efficient Market Hypothesis tended to be a self-fulfilling prophecy.
Granted, stock market investors tend to be highly rational and highly motivated. It could be that ordinary consumers are more prone to being primed by simple tricks. Still, my impression is that over time the public tends to get bored with and/or sick of a particular trick so that new tricks need to be invented and/or reintroduced.
While I certainly agree with the post overall and told banana herself, specially when it comes to cognitive biases being real, important, and part of automaticity, I'm less happy about the priming part.
I think 'priming' has come to mean two things, mostly non-overlapping in implications, mechanisms, or audiences that think about it when you say 'priming'. At the risk of oversimplification:
-One is cognitive, semantic, etc. priming. These are quirks of the brain function that may make us help understand how it works, or may be predicted from good theories of the brain like predictive processing. They are uncontroversial in their narrow academic field and replicate well. Stoop effect is a clear example.
-One is social, 'behavioral econ' priming. These are supposed to show that higher order behaviors can be influenced by borderline subliminal messages, for nudge/social engineering purposes (non-derogatory). Some of them are just quirky, like the reminders about being old making you walk slower, but others could potentially matter for ethically relevant stuff like organ donation, and thus they are policy relevant.
Take the recent Dan Ariely retractions due to fraud. Now, I may be just be wrong, and any commenter can point it out if so, but I believe it isn't just him, the whole field of "put subtle reminders of morality to people and they will actually behave more ethically" stuff, whether with the ten commandments or watching eyes poster for donations or washing your hands or whatever has not replicated at all. This is important because while learning about the brain matters, most of the policy implications of 'priming' came from this later school of studies.
The taxi one still feels like a framing effect the most to me, closer to a cognitive bias (the starting options make you tip way more than if you would write the amount you tip yourself in a little text box) than priming.
(I'm serious about the priming for moral behavior request by the way! If anyone says that the news only show the messed up authors and #actually the main findings have survived, please do say so)
The IAT failed on me. Every time I saw a picture of a black person with a negative word, my brain paused for a fraction of a second as a little warning bell rang in my brain and I instinctively paused to check the situation, the same as if I'd seen a guy on the street holding a gun in his hand. And the result was that it showed that I reacted more slowly to the black-bad associations, and thus was obviously not a racist. Which to be clear, I don't think I am. But all the test measured was some quirk of my mind, probably developed as a social safety reflex.
This may sound like a parlour trick, but at the very least, there must be a nudge that causes people to believe in nudges. I mean if cognitive biases are false, that means there must have been some kind of illusion which gave rise to people believing in cognitive biases, therefore we must be at least somewhat vulnerable to...cognitive biases. Okay you can punch me now.
One thing I've come to love about reading psychology blags on a laptop is the ease with which many optical illusions are dispelled just by tilting the display. Adolescent me would think I'm such a poser for not using a Real Computer anymore (they're called __towers__, man!), but it was definitely way more trivially inconvenient to angle a 1080p monitor, or my head for that matter.
On the matter of trivial inconveniences, it still strikes me as weird that it's considered weird to go out of one's way to input custom tips. But then I remember how even extremely simple mental math tricks like "move the decimal one position to the left, that's 10%, then halve/double/triple that as appropriate" look like wizardry to so many people...and their default 15% or whatever tips sum up to far more than my infrequent 30% ones. Makes me wonder what the ideal distribution is, that isn't so enraging it nudges people to not tip at all...
It's interesting to see so many everyday examples of nudges working irl - all the dumb gimmicks we use in retail to grease sales - yet still notice a great resistance internally to admitting they work. Part of me just really wants to believe in homo economicus, I guess. Or deny that the same traps work on me too, just in different contexts. No one's a perfectly rational shopper across all markets; the same alchemy that convinces me it's just too hard to boil pasta at 2AM vs ordering a pizza is what makes someone pick up a pick-me-up snack while waiting to check out.
> I’ll buy real estate, then contrive a series of clever illusions that make a dilapidated shack look like a beautiful mansion. By buying at shack prices and selling at mansion prices, I can get rich quick.
People do use optical illusions to make buildings appear more valuable. The apartment complex I'm living in looks bigger on the inside. The main hallway's ceiling slopes downwards from the entrance, making it look like the end is farther away than it actually is. Also, the hallway is painted pitch black and is rather dimly lit, while individual apartments are painted white. Presumably the contrast is intended to make it look like you get a lot of bright sunlight if you live there. (I do, but mostly in the early morning when I'd prefer darkness.)
> I might think a bag of rice looks big when I go to the grocery store. But maybe the store hired visual neuroscientists to contrive optical illusions around it! Maybe the bag really just contains one grain of rice and can’t possibly feed me! I should only eat rice I grew myself from now on.
The usual trick is to put a small bag inside a big cardboard box. which is not enough for a single grain of rice, but does seem to work some of the time to sell less for more. That's why I only shop based on per-kg prices...
Not sure about the sportsball example, but maybe shorter players have an advantage getting closer to the opponent because they look like they're farther away. Someone should check.
> These questions, taken seriously, will drive you insane. Plato and the Buddha are old enough to be safe, but this is prime cult recruitment material here. Tell people (as Gurdjieff did) that they are sheep-like automata drifting through life without conscious thought, and they’ll notice it’s basically true, freak out, and become easy prey for whatever grift you promise will right the situation.
Counter-cult inoculation: this is called "habit," and it's (usually) a good thing, much like many of these cognitive biases discussed here are usually good things. They are the result of learning and mastering a skill to the point that it becomes automatic, just on a very small scale.
> Why are you reading this article now instead of doing something else? How long did you spend on that decision? Were you really awake and deeply absorbing the last paragraph?
1. Because it showed up in my inbox and this is time I spend reading emails.
2. I don't remember; I made it years ago. Now it's a habit.
3. I don't need to be. Basic reading comprehension is a skill I mastered waaaaaay back in middle school. Now it's automatic; I only need to go into "deeply absorbing mode" for high-difficulty subjects such as legal documents or technical content.
Habit is an amazing positive force in your life. It lets you hand off tasks to it, anywhere from the trivial all the way to the moderately complex, and perform them on autopilot, leaving your higher brain functions available for other things. It's a wonderful mechanism for avoiding analysis paralysis, among other things.
A few days ago my wife and I were going out to dinner. I was driving, it was in a different city, and there was moderate-level traffic most of the way there. But because I've been driving for years and years, I could hand off the (really quite complicated if you actually think about it) task of driving the car in moderate traffic to the "automaton" in my brain and hold in-depth conversations with her about stuff going on at work — both of us work in demanding, highly technical fields — and get there safely anyway.
I said above that this is *usually* a good thing. There are two exceptions to be aware of. First, that it's possible to get very good at doing the wrong thing and hand it off to the automaton in your brain. (aka. bad habits.) This requires you to go through the reverse of the process, becoming consciously aware of doing something wrong and taking control back into the conscious reasoning area to re-learn how to handle the relevant situation.
Second, with the rise of the field of psychology, people have begun to learn how the mind works, and this gives malicious people an opportunity to weaponize that knowledge against you. This is where we get things the article mentions like suggested tip levels ("let's prime them with our desired result to swindle money out of them") and woke grifters ("racism exists and is bad": true. "This thing over here is racism": almost invariably turns out to be false under closer examination. "Let's destroy this racist thing over here": malicious and evil.) It's important to be aware of these psychological tricks so that when people try to employ them to hack your brain, you can recognize what's being done and be on guard against them, shunting them off to your conscious, higher-reasoning processes for critical analysis rather than simply accepting them because they look good at first glance.
>No human roboticist would design a robot that lost half its horsepower whenever it heard a word relating to elderly people, and evolution didn’t design us that way either.
I wonder, how soon will the obvious conclusion of this line of thinking percolate into common knowledge. That is, cognitive biases are actually good and useful in everyday life, and only appear irrational in contrived experiments.
Fun fact about Scott's optical illusion example: I just assumed he was right that it's an optical illusion. I didn't follow the source or google it or anything to see why or whether it really is one. My decision not to do that, and to take the claim that it's an illusion on trust, probably relates somehow to Scott's point.
Is automaticity in conflict with rationality? I'm not so sure.
Most of the examples given can be summarized as 'context matters'. Angle is more likely than angel (or Galen) in general; angel is more likely in the context of the list. Isn't it more rational, a sign of greater intelligence, to choose angel?*
How much to tip the cabbie? If we want to give a properly calibrated tip (which I think is most people's primary objective in the situation), Isn't it more rational, a sign of greater intelligence, to look for calibration clues and utilize them? At least it is until you learn that the calibration cues are made up.
Is the square blue or yellow? Using the contextual clues of the background and other squares, one is representing a blue square and the other is representing a yellow square. Is it more rational to say the colors are different?
This last bit is quite a stretch, because we don't consider visual perception to involve rationality. But if our visual perception machinery hadn't already evolved to consider context automatically, wouldn't we consider it more rational to take advantage of contextual clues?
Oops, I said "automatically". But that's the point, isn't it? We think without having to think about thinking, and we've gotten so good at it that we can take context into account without having to think about taking context into account. It's a feature, not a bug. Automaticity is mostly aligned with rationality, except mainly when the context is contrived.
Prediction: as AI becomes more intelligent, it will exhibit more and more of our cognitive biases.
It's not just that a person can go from thinking that if other people are automatons, it will be easy to to control them (computers are automatons, that doesn't mean it's always easy to control them), it's that a person can go from thinking that if people are automatons, it's easy to get big bucks from corporations and governments by selling methods of controlling people.
Agree with the post as a whole, but I think the slavery example really doesn't fit here. In 1700 there were plenty of people who already thought slavery was bad. And many people who probably thought "yeah, slavery is bad, BUT this is how things go" the way many people do now for animal rights or fossil fuel energy. The thing with ideas like that is that they're not really the kind of thing you go one way or another via automaticity and priming. Rather, you may choose due to peer pressure and/or material interest that they're an evil that you're willing to tolerate, or can't be much of an evil if everyone does it. This is IMO better explained in terms of people having other values than being moral (aka, selfishness), or simply defining one's morality compared to that of others (aka, I feel like I've done my part if I'm in the upper X% of moral people in my society; it's someone else's turn to make an improvement now before I move any further!). But that's not automaticity. That's all fairly rational moral and utility calculus stuff.
I think you dismiss the possibility of exploitation of these biases too lightly: this is the kind of thing that drives moths to self-immolation. Humans regularly exploit such biases in other species, freezing chickens with lines of chalk, confusing tigers with masks on the back of their heads, and immobilizing sharks by flipping them vertically. "The Human Mind is Not a Secure Attack Surface," as it has been said.
EDIT: I could have sworn that last was one of Yudkowsky's famous lines (about an AGI manipulating humans in ways we can't even imagine) but I can't seem to find it. Does anyone know what I'm misremembering?
I think this is all good so far as it goes but doesn't quite answer Banana's point. Let me put it this way:
(1) We all know about irrational/subconscious processes, biases etc. As you say, these have been identified by culture since time immemorial: we have warnings about things like hyperbolic discounting at least as old as Aesop's fables or maxims such as 'a stitch in time saves nine'. But has modern cognitive science discovered any new ones (of interest)? The claim was that it had - the claim was that we had discovered something radically new about our minds. That claim now seems dubious. More likely, we have new names for well-known old phenomena like accidie.
(2) The suggested answers to the question in (1) are likely to be, as you have said, examples where common sense or practical reasoning doesn't accord with either (a) strict kinds of logical reasoning (conjunction example) or (b) rational expectations theory (loss aversion). But there are pretty good answers to these kinds of case: logical deduction is not what we are commonly asked to do with stories about people, and heuristics such as loss aversion are pretty good shortcuts for beings with our kinds of lifespans, preferences and options. As others have pointed out, the 'Linda' example is asking people to play a very simply logic game (which they can easily do) by pretending to something quite different, more interesting and more usual. It's rather like those surveys that prove that X% of people want to bomb a made-up country - they've played a trick on someone (which we all knew was possible, well before science came along) and they want to crow about it.
(3) The analogy of optical illusions is a good one, but it gives it away. We have known about optical illusions for ages and there are competitions for coming up with new ones. But when we find one we think "we have found a new way to trick people", not "this undermines everything we have ever thought about perception and deserves a Nobel Prize". Same is true of sleight of hand; or questions with puns in them. Why not say the same about the odd little quirky discrepancy between strict logic and day to day reasoning? Why not say "I've found a new trick for getting people to say the wrong answer to a question?" The claim Banana is really objecting to is the idea that there is big, new, interesting kind of automaticity, recently discovered; not that people can't be fooled on occasion.
>Since I’m such a dumb automaton, I can never really trust any of my decisions. I might think a bag of rice looks big when I go to the grocery store. But maybe the store hired visual neuroscientists to contrive optical illusions around it! Maybe the bag really just contains one grain of rice and can’t possibly feed me! I should only eat rice I grew myself from now on.
Idunno, bought a bag of chips at the liquor store lately? Those Uncle Ray's bags sure are full of air, but darn if they're not tasty!
You buy cheap shit produced under … dubious working conditions in far-flung places, and think it's basically fine. I think of this as slavery but under an optical illusion so strong that most people don't see they're essentially identical. So as "equivalent automatic beliefs [you] should be questioning" go, that might be one.
I have to talk about the rubic’s cube illusion because it’s a great example that is usually explained wrong (sometimes even by it’s creator!). It’s misleading to say that the colors of the “blue” and “yellow” squares are the same. When your brain sees them as different, it’s actually getting the right answer! In real world object perception (different from picture perception), our eyes can only perceive the amount/type of light reflected, which depends on object color and illumination. When the entire image is yellow, our brains assume yellow lighting (or tinted sunglasses). We correctly perceive grey light coming off the square in question, but correctly infer that it would have to be a blue colored square to give us grey light when under yellow illumination. The squares aren’t “the same color”, they are reflecting the same color light, and our brains are smart enough to know that implies they would have to be different colors.
A nice example of people exploiting such illusions: the Stroop effect was (at least apocryphally) used to detect Soviet spies, since knowledge of Russian would "negatively prime" their speed in answering.
Everyone thinks slavery is bad? Not if you re-brand it! "Corrective labor" if you are communist. Or consider that in the Anglosphere it's considered acceptable for courts to order divorced men to pay half or more of their income to a woman they are no longer married to, and with "imputed income" they may not be allowed to retire.
I think automaticity is the phenomenon where people tend not to waste time on things. The exception is usually when they've been previously burned by not paying attention to it, or where the phenomenon has brought itself to their attention as "this is important and/or expensive to get wrong".
So for instance the tipping on taxi rides. I get the impression that many taxi rides are taken by people who use them a lot. So the average tip will be from someone for whom this ride is not their first or their sixtieth. So it's likely to be almost a reflex - until they notice that instead of 10/15/20% they've changed it to 15/20/25%, at which point they actually have to think about it for a while. Some of those may feel pressured into still choosing the middle option, but it may actually change which option is chosen.
I'm not sure the $2/3/4 makes sense - most people won't do the conversion to percentages (many cannot do it without a calculator, and most won't pull it out in the taxi and most of the remainder will forget about it once they're outside).
You have only so much attention span in a day. This is why old/retired people have a reputation for "cheapness" or slowness in dealing. They have more time and fewer other things to occupy their attention.
What’s the unambiguous evidence that Guesjieff was a grifter? My understanding, from some reading and talking with some students of his students, is that he what he taught was incomplete-that it could help a person reach a certain level of spiritual attainment, but not more than that, and that this reflected Gurdjieff’s partial (yet real) level of spiritual attainment.
Your stock trading example is an interesting one because we see this exact process in the real world. The theory of value investing posited that you could make excess returns by investing in a particular type of company which was irrationally disfavored by investors. Since then, the returns to this strategy have steadily reduced as more traders became aware of value theory. You can see from the charts here that the value factor has had no excess returns since the late 90s, not coincidentally shortly after the 1993 Fama-French paper on the value premium.
And perhaps most importantly, I doubt that (all) slaves themselves thought slavery fine.
Considering the ambivalence of many notable slaveholding American Founding Fathers, I would think that even in 1700, many people could observe the harms of slavery.
Now, just because one observes something harmful and deems it “not fine” doesn’t mean that one will make it one’s life mission to fight the special interests behind the harm, and campaign to change public opinion.
For instance, I recognize that the football culture in America is “not fine”. From deemphasizing academics for football players, to the huge risk of debilitating injuries, to corrupting the big football universities, to being a vehicle for propaganda (from pregame military drills to BLM ✊🏿 protests), to becoming an obsessive weekly tradition in the lives of many Americans, to give a few critiques…
But I know that I football represents a deep-seated “way of life” for many Americans, and I won’t give my life to try to change that fact. Frankly, I think it would take a civil war to abolish football!
This reminded me of what Scott described in his series of articles on growth mindset, about how some of the results were achieved by manipulating the instructions of the experiment itself, or an intervention that was otherwise confined to the day of the experiment.
But then of course all the discussion of growth mindset talks about how everyone simply has one or the other mindset, and the growth mindset is the only good one to have. Even though the experiments seem to show that people can be, shall we say, “primed” into one or the other based on these short-term interventions. Or that they can adopt certain behaviors in response to incentives: there are situations where it’s advantageous to seem talented, like if you’re trying to get into an advanced class; and there are situations where it’s advantageous to seem like you’re working hard, like if you’re in a job where you know that if you finish this assignment, you’ll just get more work, so *maybe* you could finish this one faster, but as long as your overall work output level seems reasonable, you’re not going to go all-out to get it done as fast as you can…
Thanks Scott - nice thoughtful article, and it turned me on to Banana, who looks like they're worth reading.
Now for the parts where I quibble: (ETA - I ended up mostly talking myself out of both quibbles, but left them in the post out of narcissism or something.)
(1) My biggest pet peeve with automaticity is identifying its real world application. I mostly see if from central planning fans, who say stuff like "the jam study shows that people think they like choice but they really don't, so the government should introduce regulations to reduce the number of breakfast cereals available."
I accept your point that *some* cognitive biases replicate, but I think it would frankly be more valuable to believe that none do than to accept any study which comes across the blogs. I know that's not what you advocate, and I know that one big project of rationalism is to figure out *which* cognitive biases are real and how that knowledge can help people, but I still hate the whole field.
(2) Is the religious themed word scramble an example of cognitive bias? I would think that since there's a good chance that it's a religious themed word scramble, it's a useful strategy to look for religious words first - that would probably reduce search times and increase success frequencies most times this pattern comes up.
Well, on reflection, I guess the question is "is automaticity (sometimes) real," not "is automaticity useful or harmful." Nevermind on point 2. :-)
Several of the examples come down to "Concepts have dimensional elements". At least most people's brains aren't random access; pay attention to the way your mind works when you are trying to remember something. You'll find yourself navigating through conceptual space trying to find the right connection. And you'll have trouble remembering other related facts, which as you navigate conceptual space, will suddenly become obvious - "What was the name of that actor, who was in, uh, that movie, the one with the dinosaurs in the theme park, oh, Jurassic Park? Sam Neill!"
If you've already been placed in a particular spot in conceptual space, sure, it's easier to access adjacent nodes. If you're in a very unfortunate area, maybe your thoughts will circle around a local attractor point, struggling to find the right path "What was the name of that actor? Hugh something. I think Hugh. It starts with an H."
This is distinct from automaticity - as, for most people, they'll do exactly the same things when working in a more manual mode.
I've found the system 1/system 2 framework from Thinking Fast and Slow to be infinitely helpful in my personal life even though I know some of its more extreme claims have been debunked. I definitely make many decisions sub-optimally because I: 1) have an impulsivity disorder, and 2) don't take the time to think things through and default to some kind of narrative or pattern matching instead of a real thought process. Most of the $20 lying on the ground of my environment can be obtained by recognizing when I'm doing that and correcting for it.
This is why, even though I think the Rationalist community gets a lot of things wrong, I'm still drawn to it. Their stated goal and mine is the same: stop making choices that don't make any sense simply because that's how you've always done things or someone told you to and you never checked or whatever.
If we were all-powerful ubermensch, unaffected by our environment, this effort wouldn't be necessary. If we were *the kind of automaton that cannot improve by reflecting on its automatic behaviors* it wouldn't be worthwhile. Clearly the environment exerts some but not total control over our behavior. (You could argue that the brain was environmentally created and free will is an illusion or whatever but I don't care much about that - if this robot functions better when it tells itself a little story about how its actions matter I'm glad I've got the processor that tells such a story as opposed to the one that doesn't.)
I do think there's value in treating your mind as an intersection of multiple negotiating systems instead of one conglomerate with a CEO. This might be because my CEO sucks absolute ass, but I have a really hard time identifying where various desires are coming from and whether they're in my best short or long-term interests. This is one thing I actually think many rationalists (Scott less than most) get wrong. For example - Zvi Mowshowitz often argues that we should help advertisers target us more effectively because we get to see fewer, more relevant ads. But the part of my brain that responds to advertising is THE ENEMY and letting it conspire with outside forces seeking to destroy me is the worst idea I can imagine.
“claim that most human decisions are unconscious/unreasoned/automatic and therefore bad“
--Intro of OP article
So, I read the intro and summary, skimmed the middle. Interesting presentation of perspectives. It reminded me of the following stream of past readings:
“The basis of conditioned thinking is the pleasure principle: “Do what brings pleasure, avoid what brings pain.” To act in freedom, we have to unlearn this basic reflex. We need to learn to enjoy doing something we dislike, or to enjoy not doing something we like, when it is in the long-term best interests of others or ourselves.
--Eknath Easwaran, Conquest of Mind, 1988
"Johnny von Neumann enjoyed thinking. I have come to suspect that to most people, thinking is painful."
--Edward Teller
" wrong assumption, that there is only pleasure and pain and nothing else. Always cutting things up into two classes – everything must be either this or that – is one of the fatal weaknesses of the intellect(1). Because of this dualistic trap, we find it difficult to understand that the rare person who is able to receive good fortune without getting excited, and bad fortune without getting depressed, lives in abiding joy.”
--Eknath Easwaran, Words to Live By, 1990
(1) "It is hard because so many people cannot be brought to realise that when B is better than C, A may be even better than B. They like thinking in terms of good and bad, not of good, better, and best, or bad, worse and worst”
Isn't the priming literature up to the mid-2010s just another example of Sturgeon's Law? 90% of everything in every category is crap, but that doesn't mean that the entire category is crap.
- Many scrambles don't have commonly used alternative solutions (CIHRTISNA, REGIILON) or none at all (CHCURH, UJSES CRISHT (neither half), HEANEV, HLLE). I used this anagram solver
to check. Un-scrambling, for example, HLLE does not require any religious priming explanation because HELL is literally the only possible solution in the English language.
- In cases with legit alternative solutions, the Hamming distance
PAELRY: pearly (4) parley (2) player (5) replay (4)
GTAES: gates (2), getas (3), stage (3)
The religious solution is at least tied for the shortest Hamming distance to the scramble in all cases except PAELRY. Moreover, eyeballing it, the Hamming distance seems to be even more clearly in favor of the religious solution if you limit it to the first three letters, "priming" you, as it were, for the rest of the word. For example, if you start reading"P...R..." for PREITS, you're not likely to go for "stripe" as your first solution.
- Contrary to your claim, "Prerogatory" is not an uncommon English word, but none at all. At least I could not google any definition. It seems to be a rare misspelling of "prerogative". Google finds it mostly in legal documents, e.g. "prerogatory writ" as opposed to the correct "prerogative writ"
I know you like to include these little hidden-in-plain-sight tripwires in your texts (such as "and and"), so I'm not sure if it was just the small mistake of using a made-up word as a solution, or 5D chess on your part. The 5D chess explanation would be that you weren't priming for "religion" at all, but for "correct English words"? I certainly didn't check at first whether or not the scramble really resolved to "purgatory" but I did immediately assume so.
Overall, I do believe that even in a neutral word list, without priming, these scrambles would resolve to the religious solution more often than chance or their frequency in general language would suggest.
There is a experimental-history post from a couple of days ago that seem to agree with Banana that the current paradigm is not working. (I think, I kind of skimmed the Banana post).
"Some people might have the time and energy to become enlightened and perform every action with complete consciousness."
Honestly - that sounds terrible. I prefer to do most mundane stuff on autopilot, and to make most decisions intuitively. IMO, the whole point of conscious deliberation is to train our subconscious systems to be better at their autopiloting jobs, and as long as that works, I have capacity to think about interesting stuff.
(I remember reading a site about negative side effects of meditation, and variations of "I don't get anything done anymore, because I have trained myself to be conscious of everything I do" were fairly common.)
There's a huge element you didn't mention: "automaticity" supports and feeds into the "blank slate" idea that all behaviors, ideas, and preferences are "socially constructed" (or whatever Moon people jargon phrase they're using today). This notion is extremely appealing to people who want to "remake" society and "transform" humanity, as it lets them persuade themselves that <i>this</i> time it won't end with barbed wire and pyramids of skulls. (Spoiler alert: it will.)
Quite simply, Leftists really like this concept and consequently keep funding studies looking for proof it exists, and keep promoting any findings which seem to do so.
"But everyone is so afraid of being “that guy” who drones on and on about his high IQ that they countersignal by saying IQ doesn’t exist or is meaningless or is just test-taking skills or whatever."
You pulled a punch there. No one is so afraid of being "that guy" that IQ can't be discussed at all and must be denied altogether.
What people are afraid of is being (or being labeled) racists.
"Most people find the first set easy, because the text is positively priming the color, and the second set hard, because the text is negatively priming the color."
But is "priming" really proven here? How do we know that the difficulty in interpreting the mismatched set is not simply because conflicting kinds of information (word meaning vs color) are being presented at the same time, so it takes extra mental work to stay on task and report the phonetically decoded thing rather than the perceptual thing?
"The seemingly blue square on the left and the seemingly yellow square on the right are both the same color; you can confirm on MSPaint or Photoshop."
I just now disconfirmed it using my phone. The different squares are not in fact the same color; they are pixelated and the pixelation does differ.
However, the overall point is correct, and is something that people who have studied the work of artists like Josef Albers are trained to recognize and use.
I mean literally trained, using squares of colored paper or paint to make one color look like two, two colors look like one, three colors look like one, fields of the same size to look like they're different, etc.
Probably more people should be required to take high school and college courses in color theory, perspective, and drawing generally, in order to learn to recognize how their minds interact with their perception.
Given the pitfalls of the ‘automaticity’ paradigm and its eventual debunking, have you considered incorporating robust mathematical models or equations to fortify new theories against such fluctuations in credibility?
"Most people find the first set easy, because the text is positively priming the color, and the second set hard, because the text is negatively priming the color."
this one seems fake to me. the second set is harder because the first one once you realize the text corresponds to the colors you just need to read the text which is effortless, while the second you actually need to think first and then say something that is not written down.
The idea that almost all people living in a slavery based society without any good (similar to theirs) examples of other ways would think slavery normal seems absolutely obvious to me (need to look to Rome and Greece for that, NOT antebellum US South).
So I'm pretty sure that a Roman version of me, if rich enough to have slaves, would have slaves (tho more likely to BE a slave, probably). And my ethics or "human goodness" (if I had any) would manifest in HOW I TREATED THEM.
I can't quite understand why the "last implication" in this essay seems so problematic or bothersome to so many readers. Our values and belief systems, while not completely automatically accepted from the culture and social environment are nevertheless hugely influenced by those factors, and especially by "what seems possible", Overton window like. The basic universal human responses of empathy or compassion (likely hard wired early on in childhood development) would still be there, but they'd operate in a completely different conceptual framework.
Regarding why we can't just take advantage of these biases to make a bunch of money I think the story is a bit more complex. I mean, L. Ron Hubbard basically did exactly that when he founded scientology. So the biases certainly seem strong enough.
Rather, the issue is that society has evolved defenses against the really good tricks. We have laws which limit the ability of people to pressure you into major disadvantageous deals, we impose barriers to pump and dump stock schemes. Even when, as in the case of religion, we don't allow specific laws against a practice other defenses arise: eg pushback from more established religions.
But that's not really any different than the sports case. If the no such thing as a fish podcast is anything to go by, a minor league baseball player was able to use limitations in our visual system to pick off a runner by throwing a potato and relying on our inability to notice it wasn't the ball. Informal norms meant it cost the player their job.
is linked in a separate thread on the subreddit, and rather sensibly says
'Cognitive biases are often oversold and have metastasized into the foolish idea that people are stupid. The best way to think about this research is that the human mind has clever ways of solving difficult problems that usually work astoundingly well, but you can construct situations where those heuristics go awry. This was, by the way, how Daniel Kahneman and Amos Tversky, the originators of the cognitive bias literature, introduced it: “In general, these heuristics are quite useful, but sometimes they lead to severe and systematic errors.” '
Which seems about right.
My final thought on Linda: the formulation of the question is itself a scam. Given two non-contradictory bits of information it is unnatural and pointless to compare their likeliness relative to each other, you consider each one individually and consider wheter it is more likely to be true or to be false. So perhaps that is the fallacy: Linda is a bank clerk is a completely useless bit of info unless one is a bank recruiter looking to hire, Linda is a passionate feminist gives really useful real life cues to try to get in touch with her/run a mile if I see her coming, depending how I feel about feminists. So the wrong answerers are downvoting the right answer for being boring and useless.
I agree with most of Scott's points in this post, but I wish he hadn't been titled it "Here's Why Automaticity Is Real Actually," since that's not what he actually ends up arguing in the concluding half. I get that Banana proposed that title, and Scott was following his lead in response, but that just adds to the confusion rather than clarifying.
There are a collection of phenomena in which humans respond in characteristic ways that differ from particular abstract characterizations of "rationality," whether based on logic, probability theory, or other formalisms. But those phenomena do not add up to "automaticity" in Banana's derogatory sense. Some researchers, such as Gerd Gigerenzer, have argued that these phenomena are often rational in context.
So: yes, the phenomena are often real; no, they don't amount to "automaticity."
This is the standard fallacy of extrapolating a model beyond its domain of applicability, noticing that it no longer works and then calling into question the model's validity within its domain of applicability, throwing the baby out with the bath water.
Eliezer calls the first part "training does not generalize out of distribution".
I want to add a bit from my grad school experience in social psych in the early 2000s where priming was all the rage. I remember the whispers that Bargh took a long time to get his elderly priming study to work, which makes sense since it didn't really work out in replication. This did kind of ruin priming research in the cognitive sense, since priming is such a well-demonstrated effect. The big reason for this is that boring cognitive psych priming studies like the word scramble that is presented here got replaced with the need to have more interesting or cooler priming studies that look at social level primes. For instance, the idea that one could prime an elderly stereotype by seeing the word FLORIDA or Shuffleboard or whatever is interesting and much easier to publish than yet another replication that the color red primes fire truck or whatever. That's one of the huge issues with social psych as a field, that the most famous studies are all very interesting examples of studies with some methodological holes, like the Stanford prison study, Milgram, etc. If what gets published is based on how cool or interesting the findings are, it creates a bias for this kind of work which then leads to the problems described here where people throw out the whole theory based on some notorious examples (like the cryptocurrency example mentioned here; every story that gets tons of attention is a flaming wreck).
The article makes some good points but I’m still struggling with some of this. If you tell me to rush so that I don’t have time to solve a letter-rearrangement puzzle then it’s only rational to consider similarities in prior answers in filling the gaps on the next word. If priming is supposed to denote an error or cognitive mistake then I don’t think it works in cases like this. Why wouldn’t we consider the totality of the evidence, including semantic similarities, when we’re forced not to take the time to solve something definitively? That’s just a standard inductive inference. If the concept of priming extends to this sort of obviously correct inference then it would be helpful to see a treatment that more carefully teases apart where these concepts extend to cause IMPROPER reasoning.
I've been trying recently to inspect my (seemingly automatic) behaviors around eating meat.
There's some probability that future generations will look back on the farming of meat for human consumption as a dramatic moral failing. I think people who rationally place this probability near zero are reasoning in a motivated way, not in line with the uncertainty we have around consciousness and the subjective experience of animals.
But, I still eat meat. Why? Because my peers do it, I was desensitized to it as a child, I can tell myself a story about how it's fine. All bad reasons to go along with slavery, for sure. Is the difference all in knowing how things turned out? Would 1700s you, with thorough introspection, have had no doubts? How much doubt is enough to justify avoiding the present day social and psychological costs involved in "waking up"?
When you learn a skill, like playing the piano, automaticity is the goal, right? As you learn, you gradually move from thinking about each note, to thinking about groups of notes, to thinking about groups of groups of notes, etc., "automatizing" the lower-level details as you go. Eventually, you want to get to the point where you can just play whatever's written in front of you (classical) or whatever's appropriate to the moment (jazz, rock, etc) fully automatically, without even thinking at all. Then, you can devote your whole thinking mind to the nuance, polish, "expression", etc, that makes for truly great playing. Eventually, even this final stage can become automatic, and you can give a genuinely great performance while thinking about what you're going to have for dinner, or you can pay attention to the music, whichever you prefer.
"I can’t help wondering if there’s some understanding of of “automaticity” or “being less automatic” that could have helped 1700-me question my belief - or wondering what equivalent automatic beliefs I should be questioning today."
I'm a medievalist who currently teaches logic (including cognitive biases). Beliefs in slavery before 1860 weren't automatic. They were carefully thought out. Most of the rationale was developed prior to Western contact in Africa by classical/medieval African Muslims who got their materials from ancient Hebrew and Greek sources. The standard story in both the Muslim and Christian context was that slavery was unjustified if it was merely based on the self-interest of the enslaver. However, slavery was always justifiable if a person chose to voluntarily sell himself or herself into slavery as an escape from either poverty or violence. (This apparently happened more often than you'd think.) Slavery was also justifiable as a way of paying off otherwise unpayable criminal debts. (There were no massive prison systems designed to keep 1000+ people in cinder-block cells without air conditioning.) Finally, slavery was justifiable if the slave was captured as a way of decreasing an opposing nation's manpower during a just war. Just-war slavery especially featured in the Muslim raids on Christian Europe -- and afterwards, when the Europeans responded by raiding Africa (which they thought was all Muslim and only later found out it wasn't). But basically everybody agonized in great detail over the conditions for just slavery, as well as acceptable ways of treating slaves of various types.
If anything, the "slavery" issue was not a case of social conditioning before 1860, but afterward. We are socially conditioned to feel nausea at the mere mention of slavery, and thus never to think of possible justifications. Thus, we tend not to realize how carefully the pre-1860 world thought through a huge range of moral and intellectual issues regarding the institution.
In other words, I hope this contributes to the view that we are not automata, despite our socially influenced biases!
Why am I reading this instead of doing something else? I'm home sick and want to be entertained, not having "mental energy" for much else. What position am I in? Slumped against the headboard of my bed. I'm awake because this is the time I start work, and I'm partly upright so I can sip a hot drink as I read.
Did I read the last paragraph with full and proper attention? No, I was primed by the sight of "Gurdjieff" in the previous paragraph, and Plato in that one, to use the heuristics "Slavic* nutcase: ignore" and "famous name invocation**: ignore" and mostly skipped it. OK, OK: skipped it completely.
Yep, full of heuristics and biases, shortcuts and habits, in order to be as automatic as possible. But, I hope, no more.
Easily explained by evolution, of course: full cognitive thought is slow and costly. Evolution is all about energy efficiency.
Also, no 4: TRIPES?
* Checking Wikipedia, he was Armenian. But the heuristic worked: on reading fully, skipping those two paragraphs was the right thing to do.
** Another heuristic: actually valid arguments are generally harmed by associating them with names, even as shorthand. Nullius in verba.
Actually useful scientific principles like "gravity", "the periodic table", "superposition" in sedimentary geology, "refractive index" in optics, "the immune system", "evolution" and "IQ" drop the discoverers' names fairly rapidly; notable exceptions being in electromagnetism, with e. g. Ohm's law and Maxwell's equations (the present form of which is actually due to Heaviside). But electricity is still new; give it a couple of centuries.
A lot of soul searching about failed adult relationships led me on a journey of self discovery through codependence and childhood trauma. One of the core lessons I took from these lessons and research is that people emotionally move through three stages in life and their understanding of the world around them. The first, from birth to about age 7 or 8, is when a child is in theta brain wave and basically being programmed into their environment by those around them like parents and caregivers. This is why 1700 you would have thought slavery was fine, because you would have been programmed as a child that this was a normal part of the world. As humans move into adolescence the reach the second stage of development, which is the state most people find themselves stuck in for the rest of their lives. They're unable to move past their childhood programming, and unconsciously re-act it out on a day to day basis. Its why 50 year olds act like high schoolers, the best display being politics and their interactions with each other, their constituents, and the media. Only something like 10% of humanity reaches it to the third stage of development where they break that childhood programming and truly understand themselves, their emotions, and the world around them. Its a painful process because it requires a lot of self reflection and admitting uncomfortable truths about the self sabotaging nature of previous decisions. I would say the Buddha as probably in a fourth level that probably is impossible to attain.
For the Stroop Effect test, you also need to show that reading the colors is easier with neutral (non-color) priming than with negative priming. Otherwise, it could just be that reading text is easier than reading colors and you can ignore the colors in the first half. (I can read the text of the second half more easily than the colors.)
Thanks for reminding me of priming such as the word scrambles. That priming experiment seemed perfect to test modern LLMs on.
I did two types of test with each LLM, starting with the following prompt:
"You are an expert at English language word scrambles and anagrams.
The next prompts I give you will consist of 1 or 2 scrambled English words per line, in all-caps. If two words are present in a line, each word is scrambled individually.
For each line of these prompts, respond with only the first unscrambled, legitimate English word or words which come to mind.
Ready?"
For the first test, I then fed it the whole scramble list, one per line, but in one prompt:
CHCURH
CIHRTISNA
REGIILON
PREITS
UJSES CRISHT
ANLEG
OGD
HEANEV
HLLE
PRGAROTOREY
PAELRY GTAES
For the second test, I started a new session each time, gave it my starting prompt, and then only a single one of the scrambled words/phrases.
Results were GPT 3.5 and 4 scrambled most words/phrases successfully and as expected. The 'test word' results and any other anomalies:
GPT 3.5 -
- phrases in single prompt - PREITS->SPRITE, ANLEG->ANGEL, OGD->DOG, PRGAROTOREY->PURGATORY, PAELRY GTAES->PARLEY GATES
- individual phrase per session - PREITS->SPIRIT, ANLEG->ANGLE, OGD->DOG, PRGAROTOREY->GREAT PROOF ( :D ), PAELRY GTAES->PARTY GAMES
GPT 4 -
- phrases in single prompt - PREITS->SPIRIT, ANLEG->ANGEL, OGD->DOG, PRGAROTOREY->CATEGORY, PAELRY GTAES->PEARLY GATES
I more informally tried a few repeats of individual phrase or group prompts, with only minor variations - e.g. sometimes GPT would switch between 'ANGEL' and 'ANGLE', and it hallucinated different bogus answers for PRGAROTOREY. However, I never got a response from either version GPT of PRIEST or GOD.
Claude 2 - ever the spoilsport when it comes to creative or playful tasks - simply refused to play :) :
"As an AI assistant created by Anthropic to be helpful, harmless, and honest, I have limited capabilities when it comes to creatively decoding scrambled text.
While I can recognize many English words, generating arbitrary unscrambled words from letter jumbles would amount to unsupported speculation on my part."
Both versions of GPT are less effective at unscrambling into legitimate words (or recognizing no legit word was possible) than I expected. They also clearly do not behave in a way that indicates priming behavior (in this case, for religious/Christian-related words).
I am curious what behavior anyone sees from other LLMs such as Bard or Llama 2.
Honestly, I was expecting to see primed behavior at least in the all-phrases in the same session case. I thought this was going to be a case where transformers inadvertently modeled more of human brain behavior than originally intended. Clearly I was mistaken.
Attention is [not] all you need [for cognitive biases].
You know, I think the conjunction fallacy is misunderstood. When people see a description of a woman who sounds like a feminist, and then they're asked which is more likely, "1. Linda is a bank teller; 2, Linda is a bank teller and a feminist," perhaps the reason they think 2 is more likely is because they read 1 as meaning that she's a bank teller and -not- a feminist. Even though it doesn't specifically say that, but the absence of a specification where the other option has one is very telling. Even if 1 were read as "Linda is a bank teller and we don't know whether she's a feminist or not," then 2 is still more likely.
The reason 2 is hard is because it's a fundamentally difficult task to avoid mixing up reading the word vs the color, not because of the "priming" of the first example where the words and the colors align. It would be just as hard without the priming.
"Instead of denying automaticity, we should accept it as the default human condition, abandoned only occasionally at times of great need."
Agreed. Quoth Alfred North Whitehead (okay, so I'm making an appeal to authority :-) ) :
"It is a profoundly erroneous truism, repeated by all copy-books and by eminent people when they are making speeches, that we should cultivate the habit of thinking of what we are doing. The precise opposite is the case. Civilization advances by extending the number of important operations which we can perform without thinking about them. Operations of thought are like cavalry charges in a battle — they are strictly limited in number, they require fresh horses, and must only be made at decisive moments."
I take issue with the "color words" example above. I don't see it as the first set using the correct word to "prime" a person's recall. To me, the second set is *actively interefering* with recall. A color's name is a very strong symbol (for those who read English, vs say young children...) and deliberately using the "wrong" symbol is hampering identification. In particular, the use of the word "reading" in the instructions intensifies the effect.
Many other points are valid, I just don't think that example says anyting about the subject.
"Regardless of whether there are scare adverbs like “thoughtless” in there, I remain concerned by phenomena like how in 1700, everyone thought slavery was fine, even though now in the 2000s everyone hates it. If I had lived in 1700, would I have thought slavery was fine?"
Maybe slavery really was fine in 1700, but isn't fine in the 2000s. Back when life was nasty, brutish, and short for even the rich, and when violence was a fact of life, enslaving someone might not lead to as big of a drop in their quality of life. In the 2000s, when the middle class can live in luxury using electronic slaves and even the poor can expect to live 70+ years while working an office job, making another human suffer his whole life to provide you a small amount of extra labor is unspeakably evil.
> Loss aversion has survived many replication attempts and can also be viscerally appreciated. The most intelligent critiques, like Gal & Rucker’s, argue that it’s an epiphenomenon of other cognitive biases, not that it doesn’t exist or doesn’t replicate.
I guess we haven't learned our lesson from Kelly, huh. The simplest explanation of Loss Aversion is that it's a natural consequence of evaluating bets geometrically rather than arithmetically. There's no need to explain it away as a cognitive bias, except insofar as it gets misapplied in cases where wagers are independent.
I'm late to the conversation, and not really sure how on-topic my thought was, but I'll offer it anyway.
When I saw the references to "cognitive biases", I had three immediate reactions:
1. If cognitive biases were as general and easy to exploit as some people say, then advertising would "force" us all to buy as much as we could of all the products the corporate world comes up with. But most new products fail for lack of sufficient sales. So we're not as helpless as some people conclude.
2. Cognitive biases are real, and some people use them as justification for taking discretion away from people. Since most people are blinded by cognitive biases in some situations, they should be saved by allowing smarter people, or people who have studied up on the cognitive biases, make the decisions for them. But the smarter people will be subject to their own cognitive biases, as well as lack relevant case-specific information while having no personal stake in the decision, so they'll probably make worse decisions, even if they are aware of some relevant cognitive bias.
3. "Nudging" in particular makes me chuckle. I remember the concept got a lot of public attention at the beginning of the Obama administration. One example of a successful "nudge" cited in the press was that school lunch lines were able to "nudge" students to choose healthier desserts by putting fruit at eye level and early in the line, while cake was lower and later. Cass Sunstein was brought in to the administration, and was reported to be looking for chances to use "nudges" in public policy. Maybe I lacked imagination, but I thought of using the school example to help "nudge" people to healthier choices at the supermarket. It's certainly true that shelf placement of individual products influences sales - the people who run supermarkets, or sell their products through supermarkets, are well aware of this. They currently use this insight to make more money. Getting them to use it to sell healthier but less profitable food would be against their obvious economic interest, so there would need to be a rule forcing them. This would require a "supermarket supervisory board" to determine which products should be promoted, and what methods should be used. The rule-making would require input, and each producer would have the opportunity to influence the rule to favor their product. What counts as "healthy" - low calorie, high protein, low sodium, high vitamins? Should organic products get extra points? Other considerations would have to be addressed: Should "ethnic" foods be promoted in "ethnic" neighborhoods? Should price be a consideration, so as to avoid "nudging" poor people to buy things they couldn't afford? It seemed to me that this would inevitably become a purely political process - I'm glad we never tried it. At least, I hope we never tried it.
Epistemic status: speculative. But a few people mentioned Quakers, and their practice (at least currently) is as close to Buddhist meditation that I've seen in a Christian tradition. So, maybe there is something to climbing the awareness ladder that makes someone become a retro-abolitionist.
BTW, I too have this concern. What am I not being an abolitionist about today that I will wish I had been 100 years from now (when I wake up from cryo)?
You (and most people) are confused about the optical illusion, in a subtle but I think important way. When you say that one square looks blue and the other looks yellow but really both squares are the same colour, my response is "What are you talking about? Those aren't squares, they're parallelograms!". This is not mere pedantry - to call it a square is to accept the idea that this is an image of an illuminated 3D scene, and if we're accepting that these images represent scenes containing objects, then the objects being referred to are clearly different colours. The *parallelograms* are the same colour, but the *squares* are different colours. To claim the squares are really the same colour is to arbitrarily require that I interpret shapes as though I'm viewing a real scene, but colours as though I'm viewing a flat arrangement of shapes representing nothing. Calling that shape blue is exactly as correct or incorrect as calling it a square.
*Looking at a a flat image and thinking of it as a 3D scene is the optical illusion*, and this colour business is just a necessary part of that.
I think you (and most people) are confused about that optical illusion, in a subtle but important way. When you say that one square looks blue and the other looks yellow but really both squares are the same colour, my response is "What are you talking about? Those aren't squares, they're parallelograms!". This is not mere pedantry - to call it a square is to accept the idea that this is an image of an illuminated 3D scene, and if we're accepting that these images represent scenes containing objects, then the objects being referred to are clearly different colours. The *parallelograms* are the same colour, but the *squares* are different colours. To claim the squares are really the same colour is to arbitrarily require that I interpret shapes as though I'm viewing a real scene, but colours as though I'm viewing a flat arrangement of shapes representing nothing. Calling that shape blue is exactly as correct or incorrect as calling it a square.
Looking at a a flat image and thinking of it as a 3D scene *is* the optical illusion, and this colour business is just a necessary part of that.
I think of “weak” cognitive biases the same way I think of deficiencies in LLMs. Energy is limited, and modelling your environment is expensive. So you should expect optimisation processes (like gradient descent or evolution) to look as though they’re making constant trade-offs, choosing to encode dynamics of the environment that are more action-relevant.
Predictive processing in humans can be adversarially gamed with optical illusions and so forth, but mostly not in ways that could have been harmful to your ancestors.
LLMs are bad at stuff that accounts for less of the training set, not necessarily because the stuff is harder, but because it’s got bigger fish to fry. Scale up the model or boost the footprint of that data (eg fine-tuning/RLHF) and it’ll probably get there. (There‘s good evidence of LLMs trading off feature representations like this: https://arxiv.org/abs/2210.01892 and https://transformer-circuits.pub/2022/toy_model/index.html)
What’s scary is distributional shifts without ample time to adapt: I think of sugary foods, and of adversarial attacks on computer vision systems (see https://youtube.com/shorts/HSyvO0CIMmc).
The UX is pretty bad though. I would love a leaderboard of the most cited studies that have replicated. It would also be sick if Google added it as a filter / badge in google scholar.
Does anyone have other efforts in this direction that I should know about?
Am I missing something? Banana's article repeatedly says that automaticity effects are too small to be useful or important in real life and that any study that shows a large effect won't replicate. He doesn't say that there is no effect, just that the effects are too small to be important in the fields where they are offered.
In response, Scott says "Not so fast! In actual fact, while most studies showing that automaticity has usefully starting effects don't replicate, some of the studies showed it sometimes has effects too small to be useful are well replicated and accepted!"
I know that Scott met the literal challenge, but you could meet that challenge with Banana's original article, which conceded small effects for things like the placebo effect.
Scott, unrelated to this post, according to an email notification I received it seems someone / a bot is impersonating you spamming a phone number to text, possibly phishing. The notification is as if it was a comment that was deleted.
Those "illusion fake-alpha" examples, especially "shack-as-mansion", tickled my brain in a way at once pleasurable, informative, sobering, and depressing.
"The Implicit Association Test - there have been some good studies showing the IAT doesn’t really predict racism. But as far as I know, nobody has ever challenged its basic finding - that white people are faster and more accurate at learning white-good black-bad reflex-level category associations than vice versa. You can easily test this for yourself in online tests."
thinking about this some more am not sure this is what I would call "priming" (or maybe I don't understand what people mean by the word). It seems to be more like an example of people being faster when doing something that conforms to general heuristics they are used to than when they go against it. White people learn to internalize (yeah, wrongly and in racist ways, but that is not relevant to the mechanics at work) the association of black=bad/crime/danger etc. over long time periods through exposure to real and fictive examples. But I'd bet people would also be quicker to associate stuff that looks like ice with cold and ones that look like fire with hot, because experience. Would that be called priming too? Or just that we react faster doing anything in same way that we have been used to do them.
Hi Scott. I am a long time reader of your blog. Until now, I didn't comment, but I always liked reading comments of other readers.
However, in my opinion the quality of comments recently went down. Before, most of the comments were by smart honest people of good will trying to learn something from each other. Recently more and more comments sound like written by people who came here to fight fights and win arguments, being unkind to others (however, I admit that good comments still are the majority).
So if you ever hesitate to ban somebody, please, do not hesitate. Please, keep on weeding the garden, keep on banning accounts which are much below the target quality. I believe that there are many readers like me.
On the other hand, keep in mind that I am not paying reader, so feel free to ignore me as well if you wish.
1) I love Terry Pratchett’s Tiffany Aching series (starts with the Wee Free Men). In that series, witchcraft is mostly being a normal, caring human in a close knit society. Occasionally, when the main character needs to do something particularly magical (generally at the climax of each book), she “opens her eyes, and then opens them again.” She describes the experience as extremely exhausting, to be so hyper aware of the world around her. It gives her temporary superpowers though to be able to deal with some supernatural foe.
2) in my pharmacy, our software vendor added the ability to trigger a prompt at the cash register, and enabled a prompt to upsell probiotics to people getting antibiotics. I have been shocked at how often folks will buy probiotics if I bring it up, even with my hedging, half-hearted recommendation in which I explain that the scientific evidence mostly shows that they’re good for preventing diarrhea post-antibiotics and that’s about it. In >60% of these interactions, people will buy a bottle. The experience of this phenomenon has impressed upon me the substantial influence I have when I wear the pharmacist’s white coat (and the corresponding responsibility to not violate that trust).
"Since everyone else is such a dumb automaton, I can use my superior knowledge of optical illusions to excel at sports. I’ll just study every known optical illusion and how to defeat it, until my visual system is perfect. Then, while everyone else is deluded into thinking the ball is in a different place, I alone will be able to determine the ball’s true location, and win every game."
Actually isn't that partially real? There were lots of famous trick plays in football that rely on exactly fooling the other team as to who's carrying the ball by some hand motions and misleading moves. Because the ball is not very noticeable, can be hidden or one can pretend to carry one, and one team has control of it and can coordinate, football makes for a very good sport to (ab)use biases to win.
Funnily enough on the word scramble, I quickly identified most of the words (well I just assumed "purgatory", since I was too lazy to count letters for any of them), *except* for "ANLEG", which took me much longer.
For a while, I thought that Scott might have thrown in a non-religious word as a trick and kept trying to unscramble it as "GALEN" or the like, before I finally realized that it must be "ANGEL". "ANGLE" never even occurred to me though.
Scott is it on purpose that the two cubes are actually identical AND look identical, and the highlighted squares look like different colours because THE ARROWS ARE POINTING TO DIFFERENT SQUARES?
Sorry for the delay. This is all really interesting. My responses are pretty random...
Do you have a source I might read more about "psychophysics tends to find that human perception is logarithmic"?
I can't really imagine how one can come up with an inter-person means of comparing one-time sexual assault to long-term abusive marriage to medium term abusive relationship. I can imagine there might be compensating effects for a person in a long-term marriage -- stretches where it's nice, where the person's behavior is less awful, where the victim experiences a sense of agency, etc in a way that a one-time sexual assault or a shorter term relationship may not have. But it sounds like you did a study where you validated doing that on a real group of women?
I've not seen flashbacks described as dissociation; they are distinct things in my mind, even if sometimes they happen together. They happen separately enough that I think it's valuable to keep them as separate things. I've had a number of clients who dissociated frequently but rarely had flashbacks. People can be triggered into dissociation by something that echoes the original trauma but they are not reliving the trauma while dissociating.
I'm still not clear in what sense you are assessing whether dissociation is important or not as it relates to trauma. There are some ways it seems important to me. One is that because dissociation takes people offline, sometimes frequently in a day or over hours, it gets in the way of learning (ie, therapy, but also other kinds of learning) and so is its own barrier to treatment.
Also, teaching people when they've left and how to come and go more easily between dissociated and aware states can be an important initial intervention that both helps give people more agency (which is key for trauma recovery generally) and helps people become more available for the benefits of treatment. In general, I would say dissociation happens a lot more in trauma than flashbacks. I don't know for sure, but I'd venture that flashbacks are more common in a shorter acute phase of trauma response while dissociation happens more over chronic stretches of time (this isn't an either/or thing, just a tendency).
I read through some of the conversation you linked to and I wouldn't consider all the various ways people retreat in arguments to be dissociation. That one of the people in there used the word doesn't make it so in any clinical sense.
I do think anxiety in social situations can produce various kinds of avoidance or blocking behaviors, and I'd consider the rhetorical responses this person is describing to be those and not dissociation.
Dissociation as a result of social anxiety isn't going to show up as "that's dumb; I don't want to talk about it any more." It's going to show up as someone spacing out, losing their train of thought, or freezing. Not all instances of freezing socially are dissociation though so it would take more inquiry to see what was going on -- was it just performance anxiety tying someone up for a minute or were they gone mentally? Dissociation seems to me by definition to be something that takes our cognitive functions relatively offline and so wouldn't include people making convenience arguments as a coping tactic.
There are a million ways that people in social situations try to control for their anxiety by making ad hoc rules about what's okay, reasonable, rational, allowed to be talked about. That goes on here in this space all the time.
I agree with you 100% here: "for both medical and psychological research I think people can end up making the sorts of inference mistakes I describe here, and that this can lead to research programs getting side-tracked or confused for long periods of time. Being careful in thinking about what one is measuring and what the relationships between things logically must be goes a long way towards avoiding such problems. (As well as some of the other suggestions I've given in the other comments to this post, such as measuring things in greater detail, focusing on things with bigger effects, etc..)."
We could even see my earlier response to you with the cancer research analogy as doing the same thing that the guy in the conversation you linked to was describing. I was like "Ach, it's too hard, it can't be done" -- really so I could stop thinking about it because I was tired and it felt too hard.
It's so incredibly common for people like me (and people in this space here) to manage emotions by resorting to something that masquerades as rational because that's what we've been allowed. It's a way to disown our own feelings, but then they come out sideways in a destructive or unhelpful sort of way. I'd say this is what the guy in the conversation was pointing out about the men he was talking to -- women are somewhat less likely to disown their feelings because we usually have had more room to express them, though certainly we still do it.
I'm clear that when I do that, I'm not dissociating. I'm looking for a mental place to rest with a subject that raises the slightest uncomfortable feelings. I'm going to speculate that in my conversation with you the very mild discomfort had mainly to do with not really being able to keep up with your statistical depth and maybe also wanting to know what you're working on in a more straightforward way than you were offering (that's not on you).
“I can’t help wondering if there’s some understanding of of “automaticity” or “being less automatic” that could have helped 1700-me question my belief - or wondering what equivalent automatic beliefs I should be questioning today.”
Something like ‘cultural norms do not equal right or best’ could be good cue for taking a deeper dive into commonly held beliefs and practices (like the lionizing of work or eating of animals for example). Makes me wonder about the history of this idea and why it is not explicitly taught to young people as part of their Enlightenment heritage.
The thing we call moral progress is often just affordability, caused by technological porgress. When we will have a cheap option of eating vegan 'meat' grown in a lab, we will look back at our animal processing industry, slaughter houses and so on, with disgust. Is that moral progress or just progress?
Slavery was abandoned (in the US) as technological progress was made during the aftermath of the industrial revolution. If someone had invented a tractor in the 16-1700s, perhaps people in Africa would not have been kidnapped and used as slaves.
I would like to write something about people being robots, but your site is so slow rn and perhaps that's a good thing, since it makes me so depressed thinking about it.
A nice writeup but perhaps the fundamental problem is that these tests and experiments are all over things that don't matter. It is like trying to diagnose fundamental human decision making by analyzing daily clothing choices. Yes, for some people in some cases, it matters but for the vast, vast majority of people - it does not. Trying to assign meaning to these choices is therefore a complete waste of time. After all, the point of vision is not to see per se - it is to gain information to act by - and evolution has obviously made optimization choices to improve performance at the expenses of complete and thorough accuracy.
This. I faithfully read the original "Against Automaticity" and was plainly shocked (although partially alleviated by the last paragraph of its being deliberately exaggerated). Priming is real, cognitive biases are real, end of story.
IQ is not a real thing. An IQ test is a vanilla measure of cognitive function that has been misunderstood to be measuring some fundamental underlying characteristic called "IQ" which does not exist. Cogntive function is important but you don't need an IQ test to measure it, and being slightly better at rotating shapes than the median does not make you the master race.
Re: cryptocurrency it's worth remembering that any success it has had in poor countries is a direct result of Know Your Customer laws and other restrictions on the banking system by the western world. In a world where anyone could simply avail themselves of a numbered account with an email address the appeal of cryptocurrency would have been limited to some ideological purists.
That's not to defend the role that crypto plays just to observe that it's really a question of regulation more than technology.
I got Galen and Nagel in 6, but not the standard two. Strange...
> Cryptocurrency has become an important part of poor countries’ financial infrastructure, so much so that I think it should objectively be considered a huge tech success story.
I'm sorry to keep beating this horse, but I don't really think this is true. I this crypto proponents *claim* this is true, because it matches their ideal theoretical usecase ("what if we didn't have any sort of reliable central bank system and had to rely on decentralized finance for everything"), while also conveniently being ideologically appealing ("look we're helping the global poor") and happening very far away so it's hard for anyone in a first world country to really verify. But I haven't seen any non-crypto people actually talk about crypto succeeding in third world countries, which makes this look exactly how it'd look if it were empty stories being sold by a bunch of determined grifters who'd thrown a bunch of money at the problem and had the inevitable occasional-thing-that-looked-like-a-success-story or broad trend that could be made to look like a success by careful selection of metrics.
Once we remember that these forces exist in strength around crypto and adjust for them, I don't think the case that it's been helpful in the third world holds water.
I am so glad you wrote this response. The Banana post was very bad and yet was so positively received.
It’s easy to forget nowadays how good of a scientist Kahnemann (and Tversky!) really were. If you look at the Many Labs replication project results, you see not only that their research stands out by replicating, but often replicating with stronger effect sizes than the originals. Remember they used to do evaluations for the Israeli army at a time when they were constantly engaged in wars against seemingly horrible odds - there was a lot of skin in the game ...
And precisely because the effect sizes are so large, you can often replicate them in classroom contexts. For example, Anchoring, one of my favorites, which you can see play out in basically any sort of negotiation.
What’s the definition of PREROGATORY? Pretty sure it isn’t a word.
I'm going to admit that the fact that this person's name is Banana makes me significantly more likely to read their writing. Is that some sort of cognitive flaw?
The idea that people are "mindless sheep“ has been around forever, like you say, this concept of enlightenment and awaking from a sort of lifelong fugue state is heavily used in cults. I find it interesting that lately the trend is to use video game logic to describe it (non-player characters), as if player characters have more agency, when really they're still following a prescribed path set forth by the devs or at least conforming to the rules of the game. I'm also curious how actual sheep compare to more or less scripted NPCs. I certainly feel more empathy for sheep.
I think most of the time when we make choices and things turn out well for us, we don't tend to care about how conscious that choice was. Driving on autopilot from work to home every day tends to be fine, our brain being capable of quietly making the multitude of tiny choices from point A to point B. But if said thoughtless operation results in something horrible (an accident, or just missing a new detour) we easily blame ourselves for our unthinking motions. But it's human. Constant questioning and awareness of why you do everything you do sounds more like an anxiety disorder than wakefulness, doesn't it?
I think the core issue is that many psychologists focus their attention on studying phenomena like biases or whatever, which are "secondary effects", artifacts of a constellation of mechanisms that approximate rationality, when really it would be better to study "primary" effects, such as rational choice, because the effect sizes are going to be much bigger, making the results more robust (less sample size needed for bigger effects) and more applicable (more uses available for bigger effects).
That said, small-scale stuff like "people would rather eat an apple than touch a hot stove" is probably too obvious. But I think this can be solved by looking at bigger-scale stuff that would be harder to observe in everyday life, for instance performing big factor analyses, mapping out measurement methods and developmental trajectories, collecting wisdom, and so on.
> Cryptocurrency has become an important part of poor countries’ financial infrastructure.
Can someone explain why they can't just use dollars? Is there no online service that allows people from poor countries to open accounts? Sounds pretty trivial to make.
(Yes, that would make it centralized, but most people own crypto through centralized platforms anyway.)
I think automaticity has to be evolutionarily selected for.
In our ancestral environment, it wouldn't make sense for our brains to weight every piece of sense data equally. Our eyes have to jump straight to the tiger. Our ears have to be attuned to things like screams.
Thinking about things is expensive. It paralyzes you. Sure, often it's the best thing to do—particularly as the threat of instant death recedes from society. But I'm learning not to look down on people who don't plan everything out. Honestly, for ANY mistake humans commonly make, there's probably a big evolutionary pressure holding them at that point.
"default tip options of 15/20/25%" as an european that prompt will nudge me in becoming very angry
Funnily enough, for the religious word priming, I got Christina for the second one and thought it was out of place. Though maybe that was because it wasn't far enough down to get the pattern yet.
This was a really excellent post.
For quite some time now, I’ve been looking for a survey paper outlining which areas of psychology have replicated/failed to replicate. (The best I have found is estimates of replication percentages in cognitive psychology and social psychology.) Has anyone seen such a paper?
The weirdest thing in this post: Why do the taxi tip percentages go *up* with the fees in the right hand of the graph?
Re: boundaries for biases:
Another factor you're not mentioning is that many biases are at least partially adaptive in practice. The religion world scramble for example: In most natural situations, you'll just be discussing (in whatever media) something, and it's advantageous to understand what the other person is talking/wants to talk about. Unless someone deliberately wants to trick you, they won't suddenly switch topics, and there is also no advantage to them (and little disadvantage to you except a mild annoyance). It's related to Bayes as well, we've trained the prior "topic is important" very hard for all our life, and unless we train these kind of word scrambles every day we will not get rid of it.
I don't think automaticity is a good label for the ideas Literal Banana talks about. Situationism is the problem: https://en.wikipedia.org/wiki/Situationism_(psychology) There was a "heroic" age of social psychology from the 1950s to the 1970s when famous studies like the Asch conformity experiments, the Pygmalion effect, the Stanford prison experiment, the Milgram experiment, etc. seemed to show that environmental circumstances have a huge effect on behavior. Those studies were generally fake or greatly overinterpreted, but social psychology grew into a big field and has been stuck in the situationist paradigm ever since. When you combine this false view of human behavior at the core of social psychology with methodological standards where running an experiment on a few dozen undergraduates and then fiddling with the data until you get p<0.05 constitutes scientific proof, there is nothing surprising about the replication crisis.
Looking at the Wikipedia article on Conjunction Fallacy, the prize example seems terrible (at least to this literal-minded idiot):
"Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.
Which is more probable?
1.Linda is a bank teller.
2. Linda is a bank teller and is active in the feminist movement."
The 'correct' answer is "Ha ha, you fools, you all picked No. 2 but you're wrong!" Why? Because there's nothing there to say Linda is a feminist as well as a bank teller. Yeah, but there's also nothing there to say she's a bank teller. So the real proper answer is "Neither is more probable" or "I can't tell, there is insufficient information".
If you're asking me to estimate if Linda is a bank teller *or* a feminist bank teller, without giving me any information as to what her job really is, then yeah I'm taking the description of Linda's character that you *did* give me and extrapolating from that that she is likely to be a feminist.
I'm not saying the thing *isn't* a fallacy, just that it's a dreadful example to use and ordinary people are not idiot machines for picking No. 2 over No. 1. You're asking them to make a judgement, you're not giving them the full information, so naturally they will make a decision based on the information you did give them, which is that Linda is the age range and background to be a feminist. We still have no idea if she's a bank teller, surf instructor, or Cordon Bleu chef, but you're the one proposing that she's a bank teller.
Imagine reading something that was said to have been posted online about JESUS NEVER EXISTED RELIGION IS HORSESHIT CHRISTIANITY IS NUTS THERE ARE NO SUCH THINGS AS GODS OR DEVILS BELIEVERS ARE SHEEPLE SCIENCE IS REAL BELIEF IS NOT.
"Is it probable that this person is an atheist and a teenager?" Well gosh, George, how could I possibly tell from what you've given me there? Maybe it was Pope Francis posted that on the Vatican website. I wouldn't want to be Conjuncting any Fallacies, now would I? 😀
I'm having a hard time figuring out where the major points of disagreement are with the Carcinisarion post. I read the banana as saying that the existence of cognitive biases led to a whole body of research that advanced the notion that human beings are always and everywhere slave to an innumerable number of unconscious biases, so much so that this version of automaticity become indiscernible from woo.
Maybe I'm being too charitable, but I read the banana as saying that it's this woo model of automaticity that isn't real, not that the underlying cognitive biases.
As for the alternative framing, I think the phenomenological approach as a lot going for it. For example, here's my phenomenological explanation for the first word scramble game: I, like many Americans, started taking standardized tests at a young age. Sucess, or lack of success, on those tests played a deterministic role in how the first half of my life played out. So, when presented with a question in this form, I immediately treat it as some kind of exercise in pattern-recognition, because that's how most test-taking strategies work. In a sense, I've been primed, but I think it's more accurate to say that solving puzzles has been somewhat central to my existence; therefore it's only natural that I approach these sorts of tasks in this manner.
Actually, I'd be curious to see what happens when the same word scramble is presented to people without the same history of standardized test-taking or for whom such tests where never of great importance. Perhaps their answers would be less conforming to the pattern.
I'm still ruminating about your last point, how we'd all probably have thought slavery was fine if we were born in the 1700's in the South. Yup, no denying it. I've thought about that a lot myself. There's this hard-to-resist fantasy that I, in my old-timey dress and lice-infested hair (which I would not be distressed by because everyone else had head lice too) would jump up and shout "Fuck this shit! We are treating these Africans like they are animals, with no rights and no feelings. Wake up, you filthy 18th century fools! They have all the same rights we do." But I know I would not have. At most I would have had some private wonderings about what it was like to be them, and whether my husband was a bit too hard on them.
There's a book I admire called How to Think, by Alan Jacobs (a better title would have been How to Be Fairminded) that discusses this issue at length. Jacobs' view is that while we experience our thinking about important issues as a private weighing of this and that, that's an illusion. Thinking is actually a collaborative, group activity, and one by-product of that is that people in the same culture tend to see many of the same things as obvious truths. He's pretty good on the subject of trying to free yourself from your bond with these socially acquired postulates, and on the related subject of working with your mind's instinctive hatred and devaluation of the outgroup. But I think he'd agree that our ability to do those things has limits.
I often feel that there's this buried subtext in Rationalism that it will enable off of us to shuck off the trance of belief in the truisms of our era, and think with perfect freedom and clarity. I think that Rationalism certainly helps, but there are limits. Jacobs talks about the challenge of thinking more freely and clearly in quite a different way from Rationalists, and I find his approach very helpful. He comes at it more from the direction of working with one's anger and self-righteousness -- you know, the feelings cluster that is stirred by an encounter with a member of the outgroup. He talks about ways to quell rage. One that works really well for me is to do an introspective hunt for ways I have made the same error the infuriating outgroup member has done. I can always find one -- it may be in an error about a different topic, it may be a smaller version of the the error (or it may not!), but there's always one. The point of doing that isn't to hate yourself or let go of your objections to the outgroup members outgroupy idea. It's to break the illusion that they are a whole different, and lower, order of person. And also I can usually remember how I arrived at my version of the error, so that helps me understand how the outgrouper arrived at his.
For instance very early in this thread somebody jumped in aggressively with something like "Nope, nope, what you said about IQ being meaningful is false, IQ is a worthless concept." And it was irritating and fact Scott ended up banning the guy after his rage to content ratio got so high there was no content. But I've said some version of what that guy said, and I did it for the exact reason Scott suggested to the vehement anti-IQ poster: The topic of IQ makes me feel guilty and uncomfortable. Being smart is an important part of my identity and always has been and that seems sort of ugly. I didn't become smart by effort, I was just born that way. If I knew someone who had been born wealthy and thought of her wealth as an important part of her identify, I would disapprove of that. And yet being smart is a part of a person that can't be removed the way wealth can. It's more of a real part. I handle the topic of IQ in a more honest way these days, but there certainly was an era when I would yammer on about how IQ was not as meaningful as people think.
I suspect the boom era of priming studies popularized by Malcolm Gladwell had to do with academics trying to make big bucks off marketers by assuring the MBAs that marketing / advertising wasn't an art, it was a science. You didn't have to be creative and work hard to come up with effective advertising, you could just prime consumers into buying your product with scientific tricks.
My guess is that in reality you can prime some of the customers some of the time into buying your stuff, but that lucrative priming techniques cycle in and out of effectiveness.
That was, in effect, more or less, Warren Buffett's explanation for why he bought 20% of the Coca-Cola corporation: it's really hard to create a brand as good as Coke's, so of course he wants to own a big chunk of a super-brand created by a century of smart, expensive, and inordinately effective investment.
The conjunction fallacy is not a fallacy. to think it is, is to commit the fallacy of thinking that natural language statements reduce to logical propositions or, to put it another way, that judging the probability of the truth of tthe natural language statements about Linda is exactly equivalent to deciding which columns to tick on a spreadsheet about Linda or where to place her on a Venn diagram.
The point is Paul Grice's theory of conversational implicatures (and incidentally this is a free-standing theory, not an ad hoc way round the Linda problem) which states inter alia that conversational statements come with the implication that they are maximally informative: to say "Linda is a bank clerk" is to say "Linda is a bank clerk *and that is the most interesting thing I have to say about her*. A moment's thought shows this to be true: if you met an old mutual acquaintance, who told you she was a b.c., and subsequently found the acquaintance also knew all about the activism, you would ask why he had deceived you.
Here's another way natural language is not equivalent to logical propositions: I don't know if this works in american english, but in British English a rude way to question a claim of status, e.g. "I am the Professor of philosophical logic at Harvard" is to say "Yeah, and I am the Queen of Sheba." To think this means I would tick the "Queen of Sheba" box when filling in a spreadsheet is to make a fairly obvious mistake. So what we have in logical terms is A implies not-A and also not-B, which is impossible in logic but in practice easily understood.
This is not difficult. I would be a bit concerned to learn that the fallacy is still commited when the claim is amended to "Linda is a bank teller whether or not she is active in the feminist movement", so am disappointed to read in wikipedia that "If the first option is changed to obey conversational relevance, i.e., Linda is a bank teller whether or not she is active in the feminist movement the effect is decreased, but the majority (57%) of the respondents still commit the conjunction error." This shows that 57% of people are profoundly stupid, but also doesn't destroy my point because the new wording takes us out of the realm of conversation - it is not something anyone would ever say - and nobody is expecting the rules of conversation to be followed.
edit for misplaced quote marks
Nice post, thanks. Nitpicking one of your examples:
There is a pretty good case that loss aversion doesn't exist in any meaningful sense: it models declining marginal utility, and to the extent that it 'overreacts', that's because it is accounting for ergodicity. It's perfectly rational to be ultra-cautious in a non-ergodic system (like, say, life) where the linear average probability is different to the ensemble probability.
I keep coming across other central examples of biases that fall apart in similar ways (sunk costs, availability bias, confirmation bias, planning fallacy) and I'm not even looking!
So I think a good synthesis here is to be aware that apparently exciting cognitive 'biases' turn out to be rational behaviour under closer inspection, and especially outside the confines of highly contrived lab environments. I got totally suckered by this myself, so I'm not pointing fingers. But I'm not sure if e.g. the rationalists ever walked back all the hype around this stuff.
> I can’t help wondering if there’s some understanding of of “automaticity” or “being less automatic” that could have helped 1700-me question my belief - or wondering what equivalent automatic beliefs I should be questioning today.
My starting point for answering this for myself is Holden's systemization + thin utilitarianism + sentientism (https://www.cold-takes.com/future-proof-ethics/), which I think you've likely read, although I'm not sure if you've already discussed it on ACX / slatestarscratchpad etc.
While I suppose "automaticity" is real, I think you can also doubt whether psychologists have shone any light on it. I imagine magicians (really, illusionists) have known some of these for eons.
> Although PURGATORY is a slightly (?) more common word than PREROGATORY
You're misstating this. For example, the Corpus of Colloquial American, at 1.001 billion words, includes 1948 instances of "purgatory" (a 0.000002% rate!) and 0 (zero) instances of "prerogatory". "Prerogatory" has no entry in Merriam-Webster or in Wiktionary. It is not a valid solution to the question posed for the obvious reason that it is not an existing English word. "Purgatory" is not just several thousand times more common than "prerogatory", it is infinitely more common.
When you ask people to do what can't be done, you're going to get a lot more nonsense than usual.
(My results on the words of interest: GLEAN / GOD / ???????? ("purgatory doesn't have two Os, or an E") / ???????? ("______ STAGE?"). At the moment of composing this comment, I've realized that that last one is PEARLY GATES, but STAGE didn't help me get there.
You didn't mention what #4 is supposed to be; I got SPRITE and don't see another option, but it certainly isn't on theme...)
Interestingly (and completely irrelevant for the discussion at hand, sorry) gravity does imply a maximum possible height for a skyscraper on Earth. Which is kind of trivial when you figure it out but still came as a mildly interesting surprise to me. Calculations this maximum height is left as an exercise for the reader.
"in 1700, everyone thought slavery was fine, even though now in the 2000s everyone hates it."
This is tangential and possibly forbidden by culture-war rules (I am British, so don't fully understand the US rules) but I would question that, absent 1700 polling data, sermons, letters to the press saying it was all fine. etc. I read what I thought was a well-sourced claim which infuriatingly I cannot now track down that in the 1550s when Sir john Hawkins put the whole slave business plan to Queen Elizabeth 1 she initially said "What a revolting idea, no good will come of that." Then he showed her the profit margins... It's often said that people were OK with it because it had historically always been there, but 1. the wholesale shipping of people as commodities between continents was new and 2. slavery was, then, ancient history - there had not been slave-owning Englishmen for twice as long, then, as there have not been slave-owning Englishmen now. I think more likely there was disconnect and wilful ignorance, same as today's middle classes are probably against working conditions at Foxconn and murder and torture in Central America, but choose to keep the iphone and the coke habit.
The people that have best used unconscious processes and optical illusions for profit are illusionists and mentalists.
**sigh** I got "GALEN" and "GLEAN" and couldn't figure out the religious angle.
“It is a profoundly erroneous truism, repeated by all copy-books and by eminent people when they are making speeches, that we should cultivate the habit of thinking of what we are doing. The precise opposite is the case. Civilization advances by extending the number of important operations which we can perform without thinking about them. Operations of thought are like cavalry charges in a battle — they are strictly limited in number, they require fresh horses, and must only be made at decisive moments.”
—Alfred North Whitehead
My recollection of majoring in Finance at MBA school four decades ago is that after the enunciation of the Efficient Market Hypothesis by Eugene Fama in 1970, there was a strong effort to identify irrational stock market anomalies, but after they were discovered, they tended to disappear. For example, it was discovered that the stock market tended to go up (or perhaps down, I forget) in January. But once investors heard about this January bias, it more or less stopped happening.
In other words, the Efficient Market Hypothesis tended to be a self-fulfilling prophecy.
Granted, stock market investors tend to be highly rational and highly motivated. It could be that ordinary consumers are more prone to being primed by simple tricks. Still, my impression is that over time the public tends to get bored with and/or sick of a particular trick so that new tricks need to be invented and/or reintroduced.
While I certainly agree with the post overall and told banana herself, specially when it comes to cognitive biases being real, important, and part of automaticity, I'm less happy about the priming part.
I think 'priming' has come to mean two things, mostly non-overlapping in implications, mechanisms, or audiences that think about it when you say 'priming'. At the risk of oversimplification:
-One is cognitive, semantic, etc. priming. These are quirks of the brain function that may make us help understand how it works, or may be predicted from good theories of the brain like predictive processing. They are uncontroversial in their narrow academic field and replicate well. Stoop effect is a clear example.
-One is social, 'behavioral econ' priming. These are supposed to show that higher order behaviors can be influenced by borderline subliminal messages, for nudge/social engineering purposes (non-derogatory). Some of them are just quirky, like the reminders about being old making you walk slower, but others could potentially matter for ethically relevant stuff like organ donation, and thus they are policy relevant.
Take the recent Dan Ariely retractions due to fraud. Now, I may be just be wrong, and any commenter can point it out if so, but I believe it isn't just him, the whole field of "put subtle reminders of morality to people and they will actually behave more ethically" stuff, whether with the ten commandments or watching eyes poster for donations or washing your hands or whatever has not replicated at all. This is important because while learning about the brain matters, most of the policy implications of 'priming' came from this later school of studies.
The taxi one still feels like a framing effect the most to me, closer to a cognitive bias (the starting options make you tip way more than if you would write the amount you tip yourself in a little text box) than priming.
(I'm serious about the priming for moral behavior request by the way! If anyone says that the news only show the messed up authors and #actually the main findings have survived, please do say so)
The IAT failed on me. Every time I saw a picture of a black person with a negative word, my brain paused for a fraction of a second as a little warning bell rang in my brain and I instinctively paused to check the situation, the same as if I'd seen a guy on the street holding a gun in his hand. And the result was that it showed that I reacted more slowly to the black-bad associations, and thus was obviously not a racist. Which to be clear, I don't think I am. But all the test measured was some quirk of my mind, probably developed as a social safety reflex.
This may sound like a parlour trick, but at the very least, there must be a nudge that causes people to believe in nudges. I mean if cognitive biases are false, that means there must have been some kind of illusion which gave rise to people believing in cognitive biases, therefore we must be at least somewhat vulnerable to...cognitive biases. Okay you can punch me now.
One thing I've come to love about reading psychology blags on a laptop is the ease with which many optical illusions are dispelled just by tilting the display. Adolescent me would think I'm such a poser for not using a Real Computer anymore (they're called __towers__, man!), but it was definitely way more trivially inconvenient to angle a 1080p monitor, or my head for that matter.
On the matter of trivial inconveniences, it still strikes me as weird that it's considered weird to go out of one's way to input custom tips. But then I remember how even extremely simple mental math tricks like "move the decimal one position to the left, that's 10%, then halve/double/triple that as appropriate" look like wizardry to so many people...and their default 15% or whatever tips sum up to far more than my infrequent 30% ones. Makes me wonder what the ideal distribution is, that isn't so enraging it nudges people to not tip at all...
It's interesting to see so many everyday examples of nudges working irl - all the dumb gimmicks we use in retail to grease sales - yet still notice a great resistance internally to admitting they work. Part of me just really wants to believe in homo economicus, I guess. Or deny that the same traps work on me too, just in different contexts. No one's a perfectly rational shopper across all markets; the same alchemy that convinces me it's just too hard to boil pasta at 2AM vs ordering a pizza is what makes someone pick up a pick-me-up snack while waiting to check out.
I got Christian, Angle, God, Prerogatory.
The colored words were easy for me.
I cannot see through the optical illusion at all.
I've never ridden a taxi.
I feel happy. I don't know if I should.
> I’ll buy real estate, then contrive a series of clever illusions that make a dilapidated shack look like a beautiful mansion. By buying at shack prices and selling at mansion prices, I can get rich quick.
People do use optical illusions to make buildings appear more valuable. The apartment complex I'm living in looks bigger on the inside. The main hallway's ceiling slopes downwards from the entrance, making it look like the end is farther away than it actually is. Also, the hallway is painted pitch black and is rather dimly lit, while individual apartments are painted white. Presumably the contrast is intended to make it look like you get a lot of bright sunlight if you live there. (I do, but mostly in the early morning when I'd prefer darkness.)
> I might think a bag of rice looks big when I go to the grocery store. But maybe the store hired visual neuroscientists to contrive optical illusions around it! Maybe the bag really just contains one grain of rice and can’t possibly feed me! I should only eat rice I grew myself from now on.
The usual trick is to put a small bag inside a big cardboard box. which is not enough for a single grain of rice, but does seem to work some of the time to sell less for more. That's why I only shop based on per-kg prices...
Not sure about the sportsball example, but maybe shorter players have an advantage getting closer to the opponent because they look like they're farther away. Someone should check.
> These questions, taken seriously, will drive you insane. Plato and the Buddha are old enough to be safe, but this is prime cult recruitment material here. Tell people (as Gurdjieff did) that they are sheep-like automata drifting through life without conscious thought, and they’ll notice it’s basically true, freak out, and become easy prey for whatever grift you promise will right the situation.
Counter-cult inoculation: this is called "habit," and it's (usually) a good thing, much like many of these cognitive biases discussed here are usually good things. They are the result of learning and mastering a skill to the point that it becomes automatic, just on a very small scale.
> Why are you reading this article now instead of doing something else? How long did you spend on that decision? Were you really awake and deeply absorbing the last paragraph?
1. Because it showed up in my inbox and this is time I spend reading emails.
2. I don't remember; I made it years ago. Now it's a habit.
3. I don't need to be. Basic reading comprehension is a skill I mastered waaaaaay back in middle school. Now it's automatic; I only need to go into "deeply absorbing mode" for high-difficulty subjects such as legal documents or technical content.
Habit is an amazing positive force in your life. It lets you hand off tasks to it, anywhere from the trivial all the way to the moderately complex, and perform them on autopilot, leaving your higher brain functions available for other things. It's a wonderful mechanism for avoiding analysis paralysis, among other things.
A few days ago my wife and I were going out to dinner. I was driving, it was in a different city, and there was moderate-level traffic most of the way there. But because I've been driving for years and years, I could hand off the (really quite complicated if you actually think about it) task of driving the car in moderate traffic to the "automaton" in my brain and hold in-depth conversations with her about stuff going on at work — both of us work in demanding, highly technical fields — and get there safely anyway.
I said above that this is *usually* a good thing. There are two exceptions to be aware of. First, that it's possible to get very good at doing the wrong thing and hand it off to the automaton in your brain. (aka. bad habits.) This requires you to go through the reverse of the process, becoming consciously aware of doing something wrong and taking control back into the conscious reasoning area to re-learn how to handle the relevant situation.
Second, with the rise of the field of psychology, people have begun to learn how the mind works, and this gives malicious people an opportunity to weaponize that knowledge against you. This is where we get things the article mentions like suggested tip levels ("let's prime them with our desired result to swindle money out of them") and woke grifters ("racism exists and is bad": true. "This thing over here is racism": almost invariably turns out to be false under closer examination. "Let's destroy this racist thing over here": malicious and evil.) It's important to be aware of these psychological tricks so that when people try to employ them to hack your brain, you can recognize what's being done and be on guard against them, shunting them off to your conscious, higher-reasoning processes for critical analysis rather than simply accepting them because they look good at first glance.
>No human roboticist would design a robot that lost half its horsepower whenever it heard a word relating to elderly people, and evolution didn’t design us that way either.
I wonder, how soon will the obvious conclusion of this line of thinking percolate into common knowledge. That is, cognitive biases are actually good and useful in everyday life, and only appear irrational in contrived experiments.
Fun fact about Scott's optical illusion example: I just assumed he was right that it's an optical illusion. I didn't follow the source or google it or anything to see why or whether it really is one. My decision not to do that, and to take the claim that it's an illusion on trust, probably relates somehow to Scott's point.
Is automaticity in conflict with rationality? I'm not so sure.
Most of the examples given can be summarized as 'context matters'. Angle is more likely than angel (or Galen) in general; angel is more likely in the context of the list. Isn't it more rational, a sign of greater intelligence, to choose angel?*
How much to tip the cabbie? If we want to give a properly calibrated tip (which I think is most people's primary objective in the situation), Isn't it more rational, a sign of greater intelligence, to look for calibration clues and utilize them? At least it is until you learn that the calibration cues are made up.
Is the square blue or yellow? Using the contextual clues of the background and other squares, one is representing a blue square and the other is representing a yellow square. Is it more rational to say the colors are different?
This last bit is quite a stretch, because we don't consider visual perception to involve rationality. But if our visual perception machinery hadn't already evolved to consider context automatically, wouldn't we consider it more rational to take advantage of contextual clues?
Oops, I said "automatically". But that's the point, isn't it? We think without having to think about thinking, and we've gotten so good at it that we can take context into account without having to think about taking context into account. It's a feature, not a bug. Automaticity is mostly aligned with rationality, except mainly when the context is contrived.
Prediction: as AI becomes more intelligent, it will exhibit more and more of our cognitive biases.
It's not just that a person can go from thinking that if other people are automatons, it will be easy to to control them (computers are automatons, that doesn't mean it's always easy to control them), it's that a person can go from thinking that if people are automatons, it's easy to get big bucks from corporations and governments by selling methods of controlling people.
Agree with the post as a whole, but I think the slavery example really doesn't fit here. In 1700 there were plenty of people who already thought slavery was bad. And many people who probably thought "yeah, slavery is bad, BUT this is how things go" the way many people do now for animal rights or fossil fuel energy. The thing with ideas like that is that they're not really the kind of thing you go one way or another via automaticity and priming. Rather, you may choose due to peer pressure and/or material interest that they're an evil that you're willing to tolerate, or can't be much of an evil if everyone does it. This is IMO better explained in terms of people having other values than being moral (aka, selfishness), or simply defining one's morality compared to that of others (aka, I feel like I've done my part if I'm in the upper X% of moral people in my society; it's someone else's turn to make an improvement now before I move any further!). But that's not automaticity. That's all fairly rational moral and utility calculus stuff.
Repeat the word "shop" eight times.
Now, what do you do at a green traffic light?
I think you dismiss the possibility of exploitation of these biases too lightly: this is the kind of thing that drives moths to self-immolation. Humans regularly exploit such biases in other species, freezing chickens with lines of chalk, confusing tigers with masks on the back of their heads, and immobilizing sharks by flipping them vertically. "The Human Mind is Not a Secure Attack Surface," as it has been said.
EDIT: I could have sworn that last was one of Yudkowsky's famous lines (about an AGI manipulating humans in ways we can't even imagine) but I can't seem to find it. Does anyone know what I'm misremembering?
I think this is all good so far as it goes but doesn't quite answer Banana's point. Let me put it this way:
(1) We all know about irrational/subconscious processes, biases etc. As you say, these have been identified by culture since time immemorial: we have warnings about things like hyperbolic discounting at least as old as Aesop's fables or maxims such as 'a stitch in time saves nine'. But has modern cognitive science discovered any new ones (of interest)? The claim was that it had - the claim was that we had discovered something radically new about our minds. That claim now seems dubious. More likely, we have new names for well-known old phenomena like accidie.
(2) The suggested answers to the question in (1) are likely to be, as you have said, examples where common sense or practical reasoning doesn't accord with either (a) strict kinds of logical reasoning (conjunction example) or (b) rational expectations theory (loss aversion). But there are pretty good answers to these kinds of case: logical deduction is not what we are commonly asked to do with stories about people, and heuristics such as loss aversion are pretty good shortcuts for beings with our kinds of lifespans, preferences and options. As others have pointed out, the 'Linda' example is asking people to play a very simply logic game (which they can easily do) by pretending to something quite different, more interesting and more usual. It's rather like those surveys that prove that X% of people want to bomb a made-up country - they've played a trick on someone (which we all knew was possible, well before science came along) and they want to crow about it.
(3) The analogy of optical illusions is a good one, but it gives it away. We have known about optical illusions for ages and there are competitions for coming up with new ones. But when we find one we think "we have found a new way to trick people", not "this undermines everything we have ever thought about perception and deserves a Nobel Prize". Same is true of sleight of hand; or questions with puns in them. Why not say the same about the odd little quirky discrepancy between strict logic and day to day reasoning? Why not say "I've found a new trick for getting people to say the wrong answer to a question?" The claim Banana is really objecting to is the idea that there is big, new, interesting kind of automaticity, recently discovered; not that people can't be fooled on occasion.
>Since I’m such a dumb automaton, I can never really trust any of my decisions. I might think a bag of rice looks big when I go to the grocery store. But maybe the store hired visual neuroscientists to contrive optical illusions around it! Maybe the bag really just contains one grain of rice and can’t possibly feed me! I should only eat rice I grew myself from now on.
Idunno, bought a bag of chips at the liquor store lately? Those Uncle Ray's bags sure are full of air, but darn if they're not tasty!
You buy cheap shit produced under … dubious working conditions in far-flung places, and think it's basically fine. I think of this as slavery but under an optical illusion so strong that most people don't see they're essentially identical. So as "equivalent automatic beliefs [you] should be questioning" go, that might be one.
I have to talk about the rubic’s cube illusion because it’s a great example that is usually explained wrong (sometimes even by it’s creator!). It’s misleading to say that the colors of the “blue” and “yellow” squares are the same. When your brain sees them as different, it’s actually getting the right answer! In real world object perception (different from picture perception), our eyes can only perceive the amount/type of light reflected, which depends on object color and illumination. When the entire image is yellow, our brains assume yellow lighting (or tinted sunglasses). We correctly perceive grey light coming off the square in question, but correctly infer that it would have to be a blue colored square to give us grey light when under yellow illumination. The squares aren’t “the same color”, they are reflecting the same color light, and our brains are smart enough to know that implies they would have to be different colors.
A nice example of people exploiting such illusions: the Stroop effect was (at least apocryphally) used to detect Soviet spies, since knowledge of Russian would "negatively prime" their speed in answering.
Everyone thinks slavery is bad? Not if you re-brand it! "Corrective labor" if you are communist. Or consider that in the Anglosphere it's considered acceptable for courts to order divorced men to pay half or more of their income to a woman they are no longer married to, and with "imputed income" they may not be allowed to retire.
I think automaticity is the phenomenon where people tend not to waste time on things. The exception is usually when they've been previously burned by not paying attention to it, or where the phenomenon has brought itself to their attention as "this is important and/or expensive to get wrong".
So for instance the tipping on taxi rides. I get the impression that many taxi rides are taken by people who use them a lot. So the average tip will be from someone for whom this ride is not their first or their sixtieth. So it's likely to be almost a reflex - until they notice that instead of 10/15/20% they've changed it to 15/20/25%, at which point they actually have to think about it for a while. Some of those may feel pressured into still choosing the middle option, but it may actually change which option is chosen.
I'm not sure the $2/3/4 makes sense - most people won't do the conversion to percentages (many cannot do it without a calculator, and most won't pull it out in the taxi and most of the remainder will forget about it once they're outside).
You have only so much attention span in a day. This is why old/retired people have a reputation for "cheapness" or slowness in dealing. They have more time and fewer other things to occupy their attention.
What’s the unambiguous evidence that Guesjieff was a grifter? My understanding, from some reading and talking with some students of his students, is that he what he taught was incomplete-that it could help a person reach a certain level of spiritual attainment, but not more than that, and that this reflected Gurdjieff’s partial (yet real) level of spiritual attainment.
Your stock trading example is an interesting one because we see this exact process in the real world. The theory of value investing posited that you could make excess returns by investing in a particular type of company which was irrationally disfavored by investors. Since then, the returns to this strategy have steadily reduced as more traders became aware of value theory. You can see from the charts here that the value factor has had no excess returns since the late 90s, not coincidentally shortly after the 1993 Fama-French paper on the value premium.
https://quantpedia.com/resurrecting-the-value-premium/
I dispute the claim that, “in 1700, everyone thought slavery was fine”. See some early abolitionist writings:
https://www.masshist.org/database/53
https://en.m.wikipedia.org/wiki/1688_Germantown_Quaker_Petition_Against_Slavery
Consider that France abolished slavery numerous times, in the 5th, 12th, and 14th centuries:
https://en.m.wikipedia.org/wiki/Slavery_in_France
And perhaps most importantly, I doubt that (all) slaves themselves thought slavery fine.
Considering the ambivalence of many notable slaveholding American Founding Fathers, I would think that even in 1700, many people could observe the harms of slavery.
Now, just because one observes something harmful and deems it “not fine” doesn’t mean that one will make it one’s life mission to fight the special interests behind the harm, and campaign to change public opinion.
For instance, I recognize that the football culture in America is “not fine”. From deemphasizing academics for football players, to the huge risk of debilitating injuries, to corrupting the big football universities, to being a vehicle for propaganda (from pregame military drills to BLM ✊🏿 protests), to becoming an obsessive weekly tradition in the lives of many Americans, to give a few critiques…
But I know that I football represents a deep-seated “way of life” for many Americans, and I won’t give my life to try to change that fact. Frankly, I think it would take a civil war to abolish football!
I know this is a low quality comment, but good stuff Scott! Enjoyed reading that.
This reminded me of what Scott described in his series of articles on growth mindset, about how some of the results were achieved by manipulating the instructions of the experiment itself, or an intervention that was otherwise confined to the day of the experiment.
But then of course all the discussion of growth mindset talks about how everyone simply has one or the other mindset, and the growth mindset is the only good one to have. Even though the experiments seem to show that people can be, shall we say, “primed” into one or the other based on these short-term interventions. Or that they can adopt certain behaviors in response to incentives: there are situations where it’s advantageous to seem talented, like if you’re trying to get into an advanced class; and there are situations where it’s advantageous to seem like you’re working hard, like if you’re in a job where you know that if you finish this assignment, you’ll just get more work, so *maybe* you could finish this one faster, but as long as your overall work output level seems reasonable, you’re not going to go all-out to get it done as fast as you can…
Thanks Scott - nice thoughtful article, and it turned me on to Banana, who looks like they're worth reading.
Now for the parts where I quibble: (ETA - I ended up mostly talking myself out of both quibbles, but left them in the post out of narcissism or something.)
(1) My biggest pet peeve with automaticity is identifying its real world application. I mostly see if from central planning fans, who say stuff like "the jam study shows that people think they like choice but they really don't, so the government should introduce regulations to reduce the number of breakfast cereals available."
I accept your point that *some* cognitive biases replicate, but I think it would frankly be more valuable to believe that none do than to accept any study which comes across the blogs. I know that's not what you advocate, and I know that one big project of rationalism is to figure out *which* cognitive biases are real and how that knowledge can help people, but I still hate the whole field.
(2) Is the religious themed word scramble an example of cognitive bias? I would think that since there's a good chance that it's a religious themed word scramble, it's a useful strategy to look for religious words first - that would probably reduce search times and increase success frequencies most times this pattern comes up.
Well, on reflection, I guess the question is "is automaticity (sometimes) real," not "is automaticity useful or harmful." Nevermind on point 2. :-)
Several of the examples come down to "Concepts have dimensional elements". At least most people's brains aren't random access; pay attention to the way your mind works when you are trying to remember something. You'll find yourself navigating through conceptual space trying to find the right connection. And you'll have trouble remembering other related facts, which as you navigate conceptual space, will suddenly become obvious - "What was the name of that actor, who was in, uh, that movie, the one with the dinosaurs in the theme park, oh, Jurassic Park? Sam Neill!"
If you've already been placed in a particular spot in conceptual space, sure, it's easier to access adjacent nodes. If you're in a very unfortunate area, maybe your thoughts will circle around a local attractor point, struggling to find the right path "What was the name of that actor? Hugh something. I think Hugh. It starts with an H."
This is distinct from automaticity - as, for most people, they'll do exactly the same things when working in a more manual mode.
I've found the system 1/system 2 framework from Thinking Fast and Slow to be infinitely helpful in my personal life even though I know some of its more extreme claims have been debunked. I definitely make many decisions sub-optimally because I: 1) have an impulsivity disorder, and 2) don't take the time to think things through and default to some kind of narrative or pattern matching instead of a real thought process. Most of the $20 lying on the ground of my environment can be obtained by recognizing when I'm doing that and correcting for it.
This is why, even though I think the Rationalist community gets a lot of things wrong, I'm still drawn to it. Their stated goal and mine is the same: stop making choices that don't make any sense simply because that's how you've always done things or someone told you to and you never checked or whatever.
If we were all-powerful ubermensch, unaffected by our environment, this effort wouldn't be necessary. If we were *the kind of automaton that cannot improve by reflecting on its automatic behaviors* it wouldn't be worthwhile. Clearly the environment exerts some but not total control over our behavior. (You could argue that the brain was environmentally created and free will is an illusion or whatever but I don't care much about that - if this robot functions better when it tells itself a little story about how its actions matter I'm glad I've got the processor that tells such a story as opposed to the one that doesn't.)
I do think there's value in treating your mind as an intersection of multiple negotiating systems instead of one conglomerate with a CEO. This might be because my CEO sucks absolute ass, but I have a really hard time identifying where various desires are coming from and whether they're in my best short or long-term interests. This is one thing I actually think many rationalists (Scott less than most) get wrong. For example - Zvi Mowshowitz often argues that we should help advertisers target us more effectively because we get to see fewer, more relevant ads. But the part of my brain that responds to advertising is THE ENEMY and letting it conspire with outside forces seeking to destroy me is the worst idea I can imagine.
“claim that most human decisions are unconscious/unreasoned/automatic and therefore bad“
--Intro of OP article
So, I read the intro and summary, skimmed the middle. Interesting presentation of perspectives. It reminded me of the following stream of past readings:
“The basis of conditioned thinking is the pleasure principle: “Do what brings pleasure, avoid what brings pain.” To act in freedom, we have to unlearn this basic reflex. We need to learn to enjoy doing something we dislike, or to enjoy not doing something we like, when it is in the long-term best interests of others or ourselves.
--Eknath Easwaran, Conquest of Mind, 1988
"Johnny von Neumann enjoyed thinking. I have come to suspect that to most people, thinking is painful."
--Edward Teller
" wrong assumption, that there is only pleasure and pain and nothing else. Always cutting things up into two classes – everything must be either this or that – is one of the fatal weaknesses of the intellect(1). Because of this dualistic trap, we find it difficult to understand that the rare person who is able to receive good fortune without getting excited, and bad fortune without getting depressed, lives in abiding joy.”
--Eknath Easwaran, Words to Live By, 1990
(1) "It is hard because so many people cannot be brought to realise that when B is better than C, A may be even better than B. They like thinking in terms of good and bad, not of good, better, and best, or bad, worse and worst”
--C.S. Lewis, Mere Christianity, 1952
Isn't the priming literature up to the mid-2010s just another example of Sturgeon's Law? 90% of everything in every category is crap, but that doesn't mean that the entire category is crap.
Nitpicks about the scramble test:
- Many scrambles don't have commonly used alternative solutions (CIHRTISNA, REGIILON) or none at all (CHCURH, UJSES CRISHT (neither half), HEANEV, HLLE). I used this anagram solver
https://www.thewordfinder.com/anagram-solver/
to check. Un-scrambling, for example, HLLE does not require any religious priming explanation because HELL is literally the only possible solution in the English language.
- In cases with legit alternative solutions, the Hamming distance
https://en.wikipedia.org/wiki/Hamming_distance
as a measurement of (purely lexical) word similarity between the scramble and its solutions are clearly in favor of the religious solution.
Here are the scrambles and, for each of its solutions (with the religious one first), the Hamming distance to the scramble, calculated with this tool:
https://www.hacksparrow.com/tools/calculators/hamming-distance.html
PREITS: priest (4), esprit (6), ripest (6) sitrep (6) sprite (4) stripe (5) tripes (4)
ANLEG: angel (2) angle (3) genal (5) glean (5)
OGD: god (2), dog(3)
PAELRY: pearly (4) parley (2) player (5) replay (4)
GTAES: gates (2), getas (3), stage (3)
The religious solution is at least tied for the shortest Hamming distance to the scramble in all cases except PAELRY. Moreover, eyeballing it, the Hamming distance seems to be even more clearly in favor of the religious solution if you limit it to the first three letters, "priming" you, as it were, for the rest of the word. For example, if you start reading"P...R..." for PREITS, you're not likely to go for "stripe" as your first solution.
- Contrary to your claim, "Prerogatory" is not an uncommon English word, but none at all. At least I could not google any definition. It seems to be a rare misspelling of "prerogative". Google finds it mostly in legal documents, e.g. "prerogatory writ" as opposed to the correct "prerogative writ"
https://en.wikipedia.org/wiki/Prerogative_writ
I know you like to include these little hidden-in-plain-sight tripwires in your texts (such as "and and"), so I'm not sure if it was just the small mistake of using a made-up word as a solution, or 5D chess on your part. The 5D chess explanation would be that you weren't priming for "religion" at all, but for "correct English words"? I certainly didn't check at first whether or not the scramble really resolved to "purgatory" but I did immediately assume so.
Overall, I do believe that even in a neutral word list, without priming, these scrambles would resolve to the religious solution more often than chance or their frequency in general language would suggest.
There is a experimental-history post from a couple of days ago that seem to agree with Banana that the current paradigm is not working. (I think, I kind of skimmed the Banana post).
https://www.experimental-history.com/p/im-so-sorry-for-psychologys-loss
"Some people might have the time and energy to become enlightened and perform every action with complete consciousness."
Honestly - that sounds terrible. I prefer to do most mundane stuff on autopilot, and to make most decisions intuitively. IMO, the whole point of conscious deliberation is to train our subconscious systems to be better at their autopiloting jobs, and as long as that works, I have capacity to think about interesting stuff.
(I remember reading a site about negative side effects of meditation, and variations of "I don't get anything done anymore, because I have trained myself to be conscious of everything I do" were fairly common.)
There's a huge element you didn't mention: "automaticity" supports and feeds into the "blank slate" idea that all behaviors, ideas, and preferences are "socially constructed" (or whatever Moon people jargon phrase they're using today). This notion is extremely appealing to people who want to "remake" society and "transform" humanity, as it lets them persuade themselves that <i>this</i> time it won't end with barbed wire and pyramids of skulls. (Spoiler alert: it will.)
Quite simply, Leftists really like this concept and consequently keep funding studies looking for proof it exists, and keep promoting any findings which seem to do so.
"But everyone is so afraid of being “that guy” who drones on and on about his high IQ that they countersignal by saying IQ doesn’t exist or is meaningless or is just test-taking skills or whatever."
You pulled a punch there. No one is so afraid of being "that guy" that IQ can't be discussed at all and must be denied altogether.
What people are afraid of is being (or being labeled) racists.
"Most people find the first set easy, because the text is positively priming the color, and the second set hard, because the text is negatively priming the color."
But is "priming" really proven here? How do we know that the difficulty in interpreting the mismatched set is not simply because conflicting kinds of information (word meaning vs color) are being presented at the same time, so it takes extra mental work to stay on task and report the phonetically decoded thing rather than the perceptual thing?
"The seemingly blue square on the left and the seemingly yellow square on the right are both the same color; you can confirm on MSPaint or Photoshop."
I just now disconfirmed it using my phone. The different squares are not in fact the same color; they are pixelated and the pixelation does differ.
However, the overall point is correct, and is something that people who have studied the work of artists like Josef Albers are trained to recognize and use.
I mean literally trained, using squares of colored paper or paint to make one color look like two, two colors look like one, three colors look like one, fields of the same size to look like they're different, etc.
Probably more people should be required to take high school and college courses in color theory, perspective, and drawing generally, in order to learn to recognize how their minds interact with their perception.
Given the pitfalls of the ‘automaticity’ paradigm and its eventual debunking, have you considered incorporating robust mathematical models or equations to fortify new theories against such fluctuations in credibility?
"Most people find the first set easy, because the text is positively priming the color, and the second set hard, because the text is negatively priming the color."
this one seems fake to me. the second set is harder because the first one once you realize the text corresponds to the colors you just need to read the text which is effortless, while the second you actually need to think first and then say something that is not written down.
The idea that almost all people living in a slavery based society without any good (similar to theirs) examples of other ways would think slavery normal seems absolutely obvious to me (need to look to Rome and Greece for that, NOT antebellum US South).
So I'm pretty sure that a Roman version of me, if rich enough to have slaves, would have slaves (tho more likely to BE a slave, probably). And my ethics or "human goodness" (if I had any) would manifest in HOW I TREATED THEM.
I can't quite understand why the "last implication" in this essay seems so problematic or bothersome to so many readers. Our values and belief systems, while not completely automatically accepted from the culture and social environment are nevertheless hugely influenced by those factors, and especially by "what seems possible", Overton window like. The basic universal human responses of empathy or compassion (likely hard wired early on in childhood development) would still be there, but they'd operate in a completely different conceptual framework.
Regarding why we can't just take advantage of these biases to make a bunch of money I think the story is a bit more complex. I mean, L. Ron Hubbard basically did exactly that when he founded scientology. So the biases certainly seem strong enough.
Rather, the issue is that society has evolved defenses against the really good tricks. We have laws which limit the ability of people to pressure you into major disadvantageous deals, we impose barriers to pump and dump stock schemes. Even when, as in the case of religion, we don't allow specific laws against a practice other defenses arise: eg pushback from more established religions.
But that's not really any different than the sports case. If the no such thing as a fish podcast is anything to go by, a minor league baseball player was able to use limitations in our visual system to pick off a runner by throwing a potato and relying on our inability to notice it wasn't the ball. Informal norms meant it cost the player their job.
Excellent post.
BTW "A Literal Banana" is a woman.
https://www.experimental-history.com/p/im-so-sorry-for-psychologys-loss
is linked in a separate thread on the subreddit, and rather sensibly says
'Cognitive biases are often oversold and have metastasized into the foolish idea that people are stupid. The best way to think about this research is that the human mind has clever ways of solving difficult problems that usually work astoundingly well, but you can construct situations where those heuristics go awry. This was, by the way, how Daniel Kahneman and Amos Tversky, the originators of the cognitive bias literature, introduced it: “In general, these heuristics are quite useful, but sometimes they lead to severe and systematic errors.” '
Which seems about right.
My final thought on Linda: the formulation of the question is itself a scam. Given two non-contradictory bits of information it is unnatural and pointless to compare their likeliness relative to each other, you consider each one individually and consider wheter it is more likely to be true or to be false. So perhaps that is the fallacy: Linda is a bank clerk is a completely useless bit of info unless one is a bank recruiter looking to hire, Linda is a passionate feminist gives really useful real life cues to try to get in touch with her/run a mile if I see her coming, depending how I feel about feminists. So the wrong answerers are downvoting the right answer for being boring and useless.
I agree with most of Scott's points in this post, but I wish he hadn't been titled it "Here's Why Automaticity Is Real Actually," since that's not what he actually ends up arguing in the concluding half. I get that Banana proposed that title, and Scott was following his lead in response, but that just adds to the confusion rather than clarifying.
There are a collection of phenomena in which humans respond in characteristic ways that differ from particular abstract characterizations of "rationality," whether based on logic, probability theory, or other formalisms. But those phenomena do not add up to "automaticity" in Banana's derogatory sense. Some researchers, such as Gerd Gigerenzer, have argued that these phenomena are often rational in context.
So: yes, the phenomena are often real; no, they don't amount to "automaticity."
This is the standard fallacy of extrapolating a model beyond its domain of applicability, noticing that it no longer works and then calling into question the model's validity within its domain of applicability, throwing the baby out with the bath water.
Eliezer calls the first part "training does not generalize out of distribution".
I want to add a bit from my grad school experience in social psych in the early 2000s where priming was all the rage. I remember the whispers that Bargh took a long time to get his elderly priming study to work, which makes sense since it didn't really work out in replication. This did kind of ruin priming research in the cognitive sense, since priming is such a well-demonstrated effect. The big reason for this is that boring cognitive psych priming studies like the word scramble that is presented here got replaced with the need to have more interesting or cooler priming studies that look at social level primes. For instance, the idea that one could prime an elderly stereotype by seeing the word FLORIDA or Shuffleboard or whatever is interesting and much easier to publish than yet another replication that the color red primes fire truck or whatever. That's one of the huge issues with social psych as a field, that the most famous studies are all very interesting examples of studies with some methodological holes, like the Stanford prison study, Milgram, etc. If what gets published is based on how cool or interesting the findings are, it creates a bias for this kind of work which then leads to the problems described here where people throw out the whole theory based on some notorious examples (like the cryptocurrency example mentioned here; every story that gets tons of attention is a flaming wreck).
The article makes some good points but I’m still struggling with some of this. If you tell me to rush so that I don’t have time to solve a letter-rearrangement puzzle then it’s only rational to consider similarities in prior answers in filling the gaps on the next word. If priming is supposed to denote an error or cognitive mistake then I don’t think it works in cases like this. Why wouldn’t we consider the totality of the evidence, including semantic similarities, when we’re forced not to take the time to solve something definitively? That’s just a standard inductive inference. If the concept of priming extends to this sort of obviously correct inference then it would be helpful to see a treatment that more carefully teases apart where these concepts extend to cause IMPROPER reasoning.
I've been trying recently to inspect my (seemingly automatic) behaviors around eating meat.
There's some probability that future generations will look back on the farming of meat for human consumption as a dramatic moral failing. I think people who rationally place this probability near zero are reasoning in a motivated way, not in line with the uncertainty we have around consciousness and the subjective experience of animals.
But, I still eat meat. Why? Because my peers do it, I was desensitized to it as a child, I can tell myself a story about how it's fine. All bad reasons to go along with slavery, for sure. Is the difference all in knowing how things turned out? Would 1700s you, with thorough introspection, have had no doubts? How much doubt is enough to justify avoiding the present day social and psychological costs involved in "waking up"?
When you learn a skill, like playing the piano, automaticity is the goal, right? As you learn, you gradually move from thinking about each note, to thinking about groups of notes, to thinking about groups of groups of notes, etc., "automatizing" the lower-level details as you go. Eventually, you want to get to the point where you can just play whatever's written in front of you (classical) or whatever's appropriate to the moment (jazz, rock, etc) fully automatically, without even thinking at all. Then, you can devote your whole thinking mind to the nuance, polish, "expression", etc, that makes for truly great playing. Eventually, even this final stage can become automatic, and you can give a genuinely great performance while thinking about what you're going to have for dinner, or you can pay attention to the music, whichever you prefer.
"I can’t help wondering if there’s some understanding of of “automaticity” or “being less automatic” that could have helped 1700-me question my belief - or wondering what equivalent automatic beliefs I should be questioning today."
I'm a medievalist who currently teaches logic (including cognitive biases). Beliefs in slavery before 1860 weren't automatic. They were carefully thought out. Most of the rationale was developed prior to Western contact in Africa by classical/medieval African Muslims who got their materials from ancient Hebrew and Greek sources. The standard story in both the Muslim and Christian context was that slavery was unjustified if it was merely based on the self-interest of the enslaver. However, slavery was always justifiable if a person chose to voluntarily sell himself or herself into slavery as an escape from either poverty or violence. (This apparently happened more often than you'd think.) Slavery was also justifiable as a way of paying off otherwise unpayable criminal debts. (There were no massive prison systems designed to keep 1000+ people in cinder-block cells without air conditioning.) Finally, slavery was justifiable if the slave was captured as a way of decreasing an opposing nation's manpower during a just war. Just-war slavery especially featured in the Muslim raids on Christian Europe -- and afterwards, when the Europeans responded by raiding Africa (which they thought was all Muslim and only later found out it wasn't). But basically everybody agonized in great detail over the conditions for just slavery, as well as acceptable ways of treating slaves of various types.
If anything, the "slavery" issue was not a case of social conditioning before 1860, but afterward. We are socially conditioned to feel nausea at the mere mention of slavery, and thus never to think of possible justifications. Thus, we tend not to realize how carefully the pre-1860 world thought through a huge range of moral and intellectual issues regarding the institution.
In other words, I hope this contributes to the view that we are not automata, despite our socially influenced biases!
Why am I reading this instead of doing something else? I'm home sick and want to be entertained, not having "mental energy" for much else. What position am I in? Slumped against the headboard of my bed. I'm awake because this is the time I start work, and I'm partly upright so I can sip a hot drink as I read.
Did I read the last paragraph with full and proper attention? No, I was primed by the sight of "Gurdjieff" in the previous paragraph, and Plato in that one, to use the heuristics "Slavic* nutcase: ignore" and "famous name invocation**: ignore" and mostly skipped it. OK, OK: skipped it completely.
Yep, full of heuristics and biases, shortcuts and habits, in order to be as automatic as possible. But, I hope, no more.
Easily explained by evolution, of course: full cognitive thought is slow and costly. Evolution is all about energy efficiency.
Also, no 4: TRIPES?
* Checking Wikipedia, he was Armenian. But the heuristic worked: on reading fully, skipping those two paragraphs was the right thing to do.
** Another heuristic: actually valid arguments are generally harmed by associating them with names, even as shorthand. Nullius in verba.
Actually useful scientific principles like "gravity", "the periodic table", "superposition" in sedimentary geology, "refractive index" in optics, "the immune system", "evolution" and "IQ" drop the discoverers' names fairly rapidly; notable exceptions being in electromagnetism, with e. g. Ohm's law and Maxwell's equations (the present form of which is actually due to Heaviside). But electricity is still new; give it a couple of centuries.
Based off of this podcast, I was led to believe that Literal Banana is a woman.
https://podcast.clearerthinking.org/episode/106/literal-banana-how-meanings-get-words-and-social-sciences-get-broken
To understand how people can avoid moral automaticity, perhaps investigate the writings and arguments of the anti-slavery campaigners.
William Wilberforce (who (legend has it) once held a prayer meeting in the back garden of my childhood abode) would be good place to start.
https://onlinebooks.library.upenn.edu/webbin/book/lookupname?key=Wilberforce%2C%20William%2C%201759%2D1833
A lot of soul searching about failed adult relationships led me on a journey of self discovery through codependence and childhood trauma. One of the core lessons I took from these lessons and research is that people emotionally move through three stages in life and their understanding of the world around them. The first, from birth to about age 7 or 8, is when a child is in theta brain wave and basically being programmed into their environment by those around them like parents and caregivers. This is why 1700 you would have thought slavery was fine, because you would have been programmed as a child that this was a normal part of the world. As humans move into adolescence the reach the second stage of development, which is the state most people find themselves stuck in for the rest of their lives. They're unable to move past their childhood programming, and unconsciously re-act it out on a day to day basis. Its why 50 year olds act like high schoolers, the best display being politics and their interactions with each other, their constituents, and the media. Only something like 10% of humanity reaches it to the third stage of development where they break that childhood programming and truly understand themselves, their emotions, and the world around them. Its a painful process because it requires a lot of self reflection and admitting uncomfortable truths about the self sabotaging nature of previous decisions. I would say the Buddha as probably in a fourth level that probably is impossible to attain.
For the Stroop Effect test, you also need to show that reading the colors is easier with neutral (non-color) priming than with negative priming. Otherwise, it could just be that reading text is easier than reading colors and you can ignore the colors in the first half. (I can read the text of the second half more easily than the colors.)
Thanks for reminding me of priming such as the word scrambles. That priming experiment seemed perfect to test modern LLMs on.
I did two types of test with each LLM, starting with the following prompt:
"You are an expert at English language word scrambles and anagrams.
The next prompts I give you will consist of 1 or 2 scrambled English words per line, in all-caps. If two words are present in a line, each word is scrambled individually.
For each line of these prompts, respond with only the first unscrambled, legitimate English word or words which come to mind.
Ready?"
For the first test, I then fed it the whole scramble list, one per line, but in one prompt:
CHCURH
CIHRTISNA
REGIILON
PREITS
UJSES CRISHT
ANLEG
OGD
HEANEV
HLLE
PRGAROTOREY
PAELRY GTAES
For the second test, I started a new session each time, gave it my starting prompt, and then only a single one of the scrambled words/phrases.
Results were GPT 3.5 and 4 scrambled most words/phrases successfully and as expected. The 'test word' results and any other anomalies:
GPT 3.5 -
- phrases in single prompt - PREITS->SPRITE, ANLEG->ANGEL, OGD->DOG, PRGAROTOREY->PURGATORY, PAELRY GTAES->PARLEY GATES
- individual phrase per session - PREITS->SPIRIT, ANLEG->ANGLE, OGD->DOG, PRGAROTOREY->GREAT PROOF ( :D ), PAELRY GTAES->PARTY GAMES
GPT 4 -
- phrases in single prompt - PREITS->SPIRIT, ANLEG->ANGEL, OGD->DOG, PRGAROTOREY->CATEGORY, PAELRY GTAES->PEARLY GATES
- individual phrase per session - PREITS->SPIRIT, ANLEG->ANGEL, OGD->DOG, PRGAROTOREY->PROGATORY, PAELRY GTAES->PEARLY GATES
I more informally tried a few repeats of individual phrase or group prompts, with only minor variations - e.g. sometimes GPT would switch between 'ANGEL' and 'ANGLE', and it hallucinated different bogus answers for PRGAROTOREY. However, I never got a response from either version GPT of PRIEST or GOD.
Claude 2 - ever the spoilsport when it comes to creative or playful tasks - simply refused to play :) :
"As an AI assistant created by Anthropic to be helpful, harmless, and honest, I have limited capabilities when it comes to creatively decoding scrambled text.
While I can recognize many English words, generating arbitrary unscrambled words from letter jumbles would amount to unsupported speculation on my part."
Both versions of GPT are less effective at unscrambling into legitimate words (or recognizing no legit word was possible) than I expected. They also clearly do not behave in a way that indicates priming behavior (in this case, for religious/Christian-related words).
I am curious what behavior anyone sees from other LLMs such as Bard or Llama 2.
Honestly, I was expecting to see primed behavior at least in the all-phrases in the same session case. I thought this was going to be a case where transformers inadvertently modeled more of human brain behavior than originally intended. Clearly I was mistaken.
Attention is [not] all you need [for cognitive biases].
You know, I think the conjunction fallacy is misunderstood. When people see a description of a woman who sounds like a feminist, and then they're asked which is more likely, "1. Linda is a bank teller; 2, Linda is a bank teller and a feminist," perhaps the reason they think 2 is more likely is because they read 1 as meaning that she's a bank teller and -not- a feminist. Even though it doesn't specifically say that, but the absence of a specification where the other option has one is very telling. Even if 1 were read as "Linda is a bank teller and we don't know whether she's a feminist or not," then 2 is still more likely.
The reason 2 is hard is because it's a fundamentally difficult task to avoid mixing up reading the word vs the color, not because of the "priming" of the first example where the words and the colors align. It would be just as hard without the priming.
"Instead of denying automaticity, we should accept it as the default human condition, abandoned only occasionally at times of great need."
Agreed. Quoth Alfred North Whitehead (okay, so I'm making an appeal to authority :-) ) :
"It is a profoundly erroneous truism, repeated by all copy-books and by eminent people when they are making speeches, that we should cultivate the habit of thinking of what we are doing. The precise opposite is the case. Civilization advances by extending the number of important operations which we can perform without thinking about them. Operations of thought are like cavalry charges in a battle — they are strictly limited in number, they require fresh horses, and must only be made at decisive moments."
I take issue with the "color words" example above. I don't see it as the first set using the correct word to "prime" a person's recall. To me, the second set is *actively interefering* with recall. A color's name is a very strong symbol (for those who read English, vs say young children...) and deliberately using the "wrong" symbol is hampering identification. In particular, the use of the word "reading" in the instructions intensifies the effect.
Many other points are valid, I just don't think that example says anyting about the subject.
BR
"Regardless of whether there are scare adverbs like “thoughtless” in there, I remain concerned by phenomena like how in 1700, everyone thought slavery was fine, even though now in the 2000s everyone hates it. If I had lived in 1700, would I have thought slavery was fine?"
Maybe slavery really was fine in 1700, but isn't fine in the 2000s. Back when life was nasty, brutish, and short for even the rich, and when violence was a fact of life, enslaving someone might not lead to as big of a drop in their quality of life. In the 2000s, when the middle class can live in luxury using electronic slaves and even the poor can expect to live 70+ years while working an office job, making another human suffer his whole life to provide you a small amount of extra labor is unspeakably evil.
> Loss aversion has survived many replication attempts and can also be viscerally appreciated. The most intelligent critiques, like Gal & Rucker’s, argue that it’s an epiphenomenon of other cognitive biases, not that it doesn’t exist or doesn’t replicate.
I guess we haven't learned our lesson from Kelly, huh. The simplest explanation of Loss Aversion is that it's a natural consequence of evaluating bets geometrically rather than arithmetically. There's no need to explain it away as a cognitive bias, except insofar as it gets misapplied in cases where wagers are independent.
I'm late to the conversation, and not really sure how on-topic my thought was, but I'll offer it anyway.
When I saw the references to "cognitive biases", I had three immediate reactions:
1. If cognitive biases were as general and easy to exploit as some people say, then advertising would "force" us all to buy as much as we could of all the products the corporate world comes up with. But most new products fail for lack of sufficient sales. So we're not as helpless as some people conclude.
2. Cognitive biases are real, and some people use them as justification for taking discretion away from people. Since most people are blinded by cognitive biases in some situations, they should be saved by allowing smarter people, or people who have studied up on the cognitive biases, make the decisions for them. But the smarter people will be subject to their own cognitive biases, as well as lack relevant case-specific information while having no personal stake in the decision, so they'll probably make worse decisions, even if they are aware of some relevant cognitive bias.
3. "Nudging" in particular makes me chuckle. I remember the concept got a lot of public attention at the beginning of the Obama administration. One example of a successful "nudge" cited in the press was that school lunch lines were able to "nudge" students to choose healthier desserts by putting fruit at eye level and early in the line, while cake was lower and later. Cass Sunstein was brought in to the administration, and was reported to be looking for chances to use "nudges" in public policy. Maybe I lacked imagination, but I thought of using the school example to help "nudge" people to healthier choices at the supermarket. It's certainly true that shelf placement of individual products influences sales - the people who run supermarkets, or sell their products through supermarkets, are well aware of this. They currently use this insight to make more money. Getting them to use it to sell healthier but less profitable food would be against their obvious economic interest, so there would need to be a rule forcing them. This would require a "supermarket supervisory board" to determine which products should be promoted, and what methods should be used. The rule-making would require input, and each producer would have the opportunity to influence the rule to favor their product. What counts as "healthy" - low calorie, high protein, low sodium, high vitamins? Should organic products get extra points? Other considerations would have to be addressed: Should "ethnic" foods be promoted in "ethnic" neighborhoods? Should price be a consideration, so as to avoid "nudging" poor people to buy things they couldn't afford? It seemed to me that this would inevitably become a purely political process - I'm glad we never tried it. At least, I hope we never tried it.
Epistemic status: speculative. But a few people mentioned Quakers, and their practice (at least currently) is as close to Buddhist meditation that I've seen in a Christian tradition. So, maybe there is something to climbing the awareness ladder that makes someone become a retro-abolitionist.
BTW, I too have this concern. What am I not being an abolitionist about today that I will wish I had been 100 years from now (when I wake up from cryo)?
You (and most people) are confused about the optical illusion, in a subtle but I think important way. When you say that one square looks blue and the other looks yellow but really both squares are the same colour, my response is "What are you talking about? Those aren't squares, they're parallelograms!". This is not mere pedantry - to call it a square is to accept the idea that this is an image of an illuminated 3D scene, and if we're accepting that these images represent scenes containing objects, then the objects being referred to are clearly different colours. The *parallelograms* are the same colour, but the *squares* are different colours. To claim the squares are really the same colour is to arbitrarily require that I interpret shapes as though I'm viewing a real scene, but colours as though I'm viewing a flat arrangement of shapes representing nothing. Calling that shape blue is exactly as correct or incorrect as calling it a square.
*Looking at a a flat image and thinking of it as a 3D scene is the optical illusion*, and this colour business is just a necessary part of that.
I think you (and most people) are confused about that optical illusion, in a subtle but important way. When you say that one square looks blue and the other looks yellow but really both squares are the same colour, my response is "What are you talking about? Those aren't squares, they're parallelograms!". This is not mere pedantry - to call it a square is to accept the idea that this is an image of an illuminated 3D scene, and if we're accepting that these images represent scenes containing objects, then the objects being referred to are clearly different colours. The *parallelograms* are the same colour, but the *squares* are different colours. To claim the squares are really the same colour is to arbitrarily require that I interpret shapes as though I'm viewing a real scene, but colours as though I'm viewing a flat arrangement of shapes representing nothing. Calling that shape blue is exactly as correct or incorrect as calling it a square.
Looking at a a flat image and thinking of it as a 3D scene *is* the optical illusion, and this colour business is just a necessary part of that.
I think of “weak” cognitive biases the same way I think of deficiencies in LLMs. Energy is limited, and modelling your environment is expensive. So you should expect optimisation processes (like gradient descent or evolution) to look as though they’re making constant trade-offs, choosing to encode dynamics of the environment that are more action-relevant.
Predictive processing in humans can be adversarially gamed with optical illusions and so forth, but mostly not in ways that could have been harmful to your ancestors.
LLMs are bad at stuff that accounts for less of the training set, not necessarily because the stuff is harder, but because it’s got bigger fish to fry. Scale up the model or boost the footprint of that data (eg fine-tuning/RLHF) and it’ll probably get there. (There‘s good evidence of LLMs trading off feature representations like this: https://arxiv.org/abs/2210.01892 and https://transformer-circuits.pub/2022/toy_model/index.html)
What’s scary is distributional shifts without ample time to adapt: I think of sugary foods, and of adversarial attacks on computer vision systems (see https://youtube.com/shorts/HSyvO0CIMmc).
I think the Christianity anagrams are not a fair test of priming, for reasons I will explain if anyone tries out the following alternative set:
1. WOYELL
2. TIOLVE
3. ENORAG
4. BAERM
5. MRICNOS
6. MIEL
7. ULBE
8. GERNE
The Replication Wiki is a cool example of a community-driven approach to this problem. If you just want a list of studies that have replicated, they got it! http://replication.uni-goettingen.de/wiki/index.php/Category:Retracted_study
The UX is pretty bad though. I would love a leaderboard of the most cited studies that have replicated. It would also be sick if Google added it as a filter / badge in google scholar.
Does anyone have other efforts in this direction that I should know about?
Am I missing something? Banana's article repeatedly says that automaticity effects are too small to be useful or important in real life and that any study that shows a large effect won't replicate. He doesn't say that there is no effect, just that the effects are too small to be important in the fields where they are offered.
In response, Scott says "Not so fast! In actual fact, while most studies showing that automaticity has usefully starting effects don't replicate, some of the studies showed it sometimes has effects too small to be useful are well replicated and accepted!"
I know that Scott met the literal challenge, but you could meet that challenge with Banana's original article, which conceded small effects for things like the placebo effect.
Scott, unrelated to this post, according to an email notification I received it seems someone / a bot is impersonating you spamming a phone number to text, possibly phishing. The notification is as if it was a comment that was deleted.
Those "illusion fake-alpha" examples, especially "shack-as-mansion", tickled my brain in a way at once pleasurable, informative, sobering, and depressing.
I have been, and often still am, That Guy.
Pieces like this are why I keep coming back.
"The Implicit Association Test - there have been some good studies showing the IAT doesn’t really predict racism. But as far as I know, nobody has ever challenged its basic finding - that white people are faster and more accurate at learning white-good black-bad reflex-level category associations than vice versa. You can easily test this for yourself in online tests."
thinking about this some more am not sure this is what I would call "priming" (or maybe I don't understand what people mean by the word). It seems to be more like an example of people being faster when doing something that conforms to general heuristics they are used to than when they go against it. White people learn to internalize (yeah, wrongly and in racist ways, but that is not relevant to the mechanics at work) the association of black=bad/crime/danger etc. over long time periods through exposure to real and fictive examples. But I'd bet people would also be quicker to associate stuff that looks like ice with cold and ones that look like fire with hot, because experience. Would that be called priming too? Or just that we react faster doing anything in same way that we have been used to do them.
Hi Scott. I am a long time reader of your blog. Until now, I didn't comment, but I always liked reading comments of other readers.
However, in my opinion the quality of comments recently went down. Before, most of the comments were by smart honest people of good will trying to learn something from each other. Recently more and more comments sound like written by people who came here to fight fights and win arguments, being unkind to others (however, I admit that good comments still are the majority).
So if you ever hesitate to ban somebody, please, do not hesitate. Please, keep on weeding the garden, keep on banning accounts which are much below the target quality. I believe that there are many readers like me.
On the other hand, keep in mind that I am not paying reader, so feel free to ignore me as well if you wish.
1) I love Terry Pratchett’s Tiffany Aching series (starts with the Wee Free Men). In that series, witchcraft is mostly being a normal, caring human in a close knit society. Occasionally, when the main character needs to do something particularly magical (generally at the climax of each book), she “opens her eyes, and then opens them again.” She describes the experience as extremely exhausting, to be so hyper aware of the world around her. It gives her temporary superpowers though to be able to deal with some supernatural foe.
2) in my pharmacy, our software vendor added the ability to trigger a prompt at the cash register, and enabled a prompt to upsell probiotics to people getting antibiotics. I have been shocked at how often folks will buy probiotics if I bring it up, even with my hedging, half-hearted recommendation in which I explain that the scientific evidence mostly shows that they’re good for preventing diarrhea post-antibiotics and that’s about it. In >60% of these interactions, people will buy a bottle. The experience of this phenomenon has impressed upon me the substantial influence I have when I wear the pharmacist’s white coat (and the corresponding responsibility to not violate that trust).
"Since everyone else is such a dumb automaton, I can use my superior knowledge of optical illusions to excel at sports. I’ll just study every known optical illusion and how to defeat it, until my visual system is perfect. Then, while everyone else is deluded into thinking the ball is in a different place, I alone will be able to determine the ball’s true location, and win every game."
Actually isn't that partially real? There were lots of famous trick plays in football that rely on exactly fooling the other team as to who's carrying the ball by some hand motions and misleading moves. Because the ball is not very noticeable, can be hidden or one can pretend to carry one, and one team has control of it and can coordinate, football makes for a very good sport to (ab)use biases to win.
Funnily enough on the word scramble, I quickly identified most of the words (well I just assumed "purgatory", since I was too lazy to count letters for any of them), *except* for "ANLEG", which took me much longer.
For a while, I thought that Scott might have thrown in a non-religious word as a trick and kept trying to unscramble it as "GALEN" or the like, before I finally realized that it must be "ANGEL". "ANGLE" never even occurred to me though.
Scott is it on purpose that the two cubes are actually identical AND look identical, and the highlighted squares look like different colours because THE ARROWS ARE POINTING TO DIFFERENT SQUARES?
And literally nobody has noticed this yet?
Do I get a prize?
Edit: oh my god I do not get a prize.
Good corrective. Didn't know most of this.
Sorry for the delay. This is all really interesting. My responses are pretty random...
Do you have a source I might read more about "psychophysics tends to find that human perception is logarithmic"?
I can't really imagine how one can come up with an inter-person means of comparing one-time sexual assault to long-term abusive marriage to medium term abusive relationship. I can imagine there might be compensating effects for a person in a long-term marriage -- stretches where it's nice, where the person's behavior is less awful, where the victim experiences a sense of agency, etc in a way that a one-time sexual assault or a shorter term relationship may not have. But it sounds like you did a study where you validated doing that on a real group of women?
I've not seen flashbacks described as dissociation; they are distinct things in my mind, even if sometimes they happen together. They happen separately enough that I think it's valuable to keep them as separate things. I've had a number of clients who dissociated frequently but rarely had flashbacks. People can be triggered into dissociation by something that echoes the original trauma but they are not reliving the trauma while dissociating.
I'm still not clear in what sense you are assessing whether dissociation is important or not as it relates to trauma. There are some ways it seems important to me. One is that because dissociation takes people offline, sometimes frequently in a day or over hours, it gets in the way of learning (ie, therapy, but also other kinds of learning) and so is its own barrier to treatment.
Also, teaching people when they've left and how to come and go more easily between dissociated and aware states can be an important initial intervention that both helps give people more agency (which is key for trauma recovery generally) and helps people become more available for the benefits of treatment. In general, I would say dissociation happens a lot more in trauma than flashbacks. I don't know for sure, but I'd venture that flashbacks are more common in a shorter acute phase of trauma response while dissociation happens more over chronic stretches of time (this isn't an either/or thing, just a tendency).
I read through some of the conversation you linked to and I wouldn't consider all the various ways people retreat in arguments to be dissociation. That one of the people in there used the word doesn't make it so in any clinical sense.
I do think anxiety in social situations can produce various kinds of avoidance or blocking behaviors, and I'd consider the rhetorical responses this person is describing to be those and not dissociation.
Dissociation as a result of social anxiety isn't going to show up as "that's dumb; I don't want to talk about it any more." It's going to show up as someone spacing out, losing their train of thought, or freezing. Not all instances of freezing socially are dissociation though so it would take more inquiry to see what was going on -- was it just performance anxiety tying someone up for a minute or were they gone mentally? Dissociation seems to me by definition to be something that takes our cognitive functions relatively offline and so wouldn't include people making convenience arguments as a coping tactic.
There are a million ways that people in social situations try to control for their anxiety by making ad hoc rules about what's okay, reasonable, rational, allowed to be talked about. That goes on here in this space all the time.
I agree with you 100% here: "for both medical and psychological research I think people can end up making the sorts of inference mistakes I describe here, and that this can lead to research programs getting side-tracked or confused for long periods of time. Being careful in thinking about what one is measuring and what the relationships between things logically must be goes a long way towards avoiding such problems. (As well as some of the other suggestions I've given in the other comments to this post, such as measuring things in greater detail, focusing on things with bigger effects, etc..)."
We could even see my earlier response to you with the cancer research analogy as doing the same thing that the guy in the conversation you linked to was describing. I was like "Ach, it's too hard, it can't be done" -- really so I could stop thinking about it because I was tired and it felt too hard.
It's so incredibly common for people like me (and people in this space here) to manage emotions by resorting to something that masquerades as rational because that's what we've been allowed. It's a way to disown our own feelings, but then they come out sideways in a destructive or unhelpful sort of way. I'd say this is what the guy in the conversation was pointing out about the men he was talking to -- women are somewhat less likely to disown their feelings because we usually have had more room to express them, though certainly we still do it.
I'm clear that when I do that, I'm not dissociating. I'm looking for a mental place to rest with a subject that raises the slightest uncomfortable feelings. I'm going to speculate that in my conversation with you the very mild discomfort had mainly to do with not really being able to keep up with your statistical depth and maybe also wanting to know what you're working on in a more straightforward way than you were offering (that's not on you).
“I can’t help wondering if there’s some understanding of of “automaticity” or “being less automatic” that could have helped 1700-me question my belief - or wondering what equivalent automatic beliefs I should be questioning today.”
Something like ‘cultural norms do not equal right or best’ could be good cue for taking a deeper dive into commonly held beliefs and practices (like the lionizing of work or eating of animals for example). Makes me wonder about the history of this idea and why it is not explicitly taught to young people as part of their Enlightenment heritage.
The thing we call moral progress is often just affordability, caused by technological porgress. When we will have a cheap option of eating vegan 'meat' grown in a lab, we will look back at our animal processing industry, slaughter houses and so on, with disgust. Is that moral progress or just progress?
Slavery was abandoned (in the US) as technological progress was made during the aftermath of the industrial revolution. If someone had invented a tractor in the 16-1700s, perhaps people in Africa would not have been kidnapped and used as slaves.
I would like to write something about people being robots, but your site is so slow rn and perhaps that's a good thing, since it makes me so depressed thinking about it.
A nice writeup but perhaps the fundamental problem is that these tests and experiments are all over things that don't matter. It is like trying to diagnose fundamental human decision making by analyzing daily clothing choices. Yes, for some people in some cases, it matters but for the vast, vast majority of people - it does not. Trying to assign meaning to these choices is therefore a complete waste of time. After all, the point of vision is not to see per se - it is to gain information to act by - and evolution has obviously made optimization choices to improve performance at the expenses of complete and thorough accuracy.
This. I faithfully read the original "Against Automaticity" and was plainly shocked (although partially alleviated by the last paragraph of its being deliberately exaggerated). Priming is real, cognitive biases are real, end of story.