Yeah. To the degree that Scott is a psychiatrist so most of his favorite examples will inevitably come from there this makes sense, but if it's genuinely hard to find this level of solid science results in fields closer to what most laypeople think of as psychology it really might be close to the education system bar
Yup, I'm reminded of his old article that he chose to become psychiatrist instead of psychologist because it makes him feel that he's really doing something. Feels like everything he mentioned here is in the psychiatrist side instead of psychologist side. Which is okay, my priors have been like that all along.
I don't think it is hard to find this level of solid science results in fields closer to what most laypeople think of as psychology. Think about claims like how intermittent reinforcement tends to produce stronger learning than 100% reinforcement. Or that you aren't sensitive to color in your peripheral vision but it is part of your experience there anyway. Or that people are slower at Stroop tests when counting backwards (or whatever).
I also think it's a bit misleading for Scott to use the word "woke" so much in the original post. It's true that woke versions of these claims were more likely to be propagated through the media than non-woke ones, but the priming stuff was mostly independent of wokeness, and I suspect there's a non-trivial amount of woke stuff that is actually just fine.
a lot of cognitive psychology is very settled but not mentioned here! spreading activation theory and other theories of activation or cognitive load are similarly settled
Conversely, there’s probably a lot of genuinely interesting controversy in cognitive psychology. (I’m just guessing.) E.g. how does memory work, how do we map 3d spaces when we aren’t looking at them, etc.
Here's a column I wrote on a study of how the Big 5 personality traits differ between people who say that Kubrick's "2001" is one of their favorite movies and people who say Julia Roberts' "Pretty Woman" is one of their favorites.
I'd say that Big 5 looks pretty sensible and useful for applications like predicting your favorite movies:
I kind of went down a rabbit hole after reading your article, and the linked study, but had Claude code up a nice visualizer for the data. If any wants to check it out, here's a link: https://claude.ai/public/artifacts/f97936aa-1945-41d0-9a30-9b74a1ea5dbe It lets you pick a movie, and see what other movies are liked by people with similar big 5 traits, or input your own big 5 traits, and see what movies it predicts you'd like.
It could be that 50 years from now, psych majors will be taught a Big 3 or Big 7 model of personality. There's no absolute proof that the Big 5 model is the best, it just seems to work fairly okayish so far.
The Dark Triad/Tetrad stands as a nice counter to the whole Big 5 idea (yes, you can find some correlation to Big 5, but these are mostly orthogonal -- you don't say someone who is Highly Open is likely to be a Sadist).
>"No real effect from factors typically considered 'psychological'"
>Electrophysiology example
>Psychopharmacology example again
...
Maybe the "psych" in neuropsychopharmacology is enough that we should lump it in as a subcategory of psychology—certainly all the above fields are intellectual descendants of psychology.
But as each of these subfields has grown rigorous and interesting in its own right, they've left home—taking that rigor with them. So most of what's left in "true psychology" is, as jumpingjacksplash says, the mind-content stuff.
How are they intellectual descendants of psychology? They are descendants of anatomy and physiology and pharmacology. (Yes, I understand there isn’t a clear practical distinction in many cases).
It is a bit as if astronomy and astrology never separated.
"Hey, I know that horoscopes are entirely useless and yet used as a reason to push pro-Capricorn quotas on the entire economy. But the same science is giving us spectrums of stars, which is cool!"
And then I look at general relativity, and I say "what's the point in all of this work, if none of the work gets referenced in Real World Money Making Enterprises?"
(x265, for "visual interpolation and compression" as claimed by OP as part of psychology.).
“Fairness” itself is worth bringing up here. Why do so many non-psychologists, non-scientists, heck, non-STEM-degreed people have so much interest in the Replication Crisis that I can put it in caps and be assured most people here will know what I’m talking about?
Is any of this really about science, or one level down is it really about culture war?
Not if the culture war itself is irrelevant. Which I know it isn’t to all people, but my experience at least has been that the more you get out and “touch grass” as the kids say, the less relevant you discover it to be.
Coincidentally, I did a bit of lawn maintenance earlier today. Grass well and truly touched.
But I was once driven out of the community that was my online-and-occasional-meetups home before this one, by culture warriors. And I was here when the culture war fragmented this community in a fortunately-unsuccessful attempt to destroy it. So, yeah, the culture war is quite relevant to me.
I think that on this Substack, it's a lot more about culture war. But in academic contexts, it's really about science. I don't think many people in the general public, other than maybe those with degrees related to psychology in the past decade or so, would recognize the phrase "the Replication Crisis".
I’d be curious to know, if it is about science to some people, do they think the term is apt? Is there really a crisis, or does cutting-edge research in any field often yield low replication rates, and this one just happened to earn a catchy label that was widely circulated because it played into existing culture war tropes?
I've heard slightly different views on this - social psychology has been the field most commonly said to be affected, but I've also seen claims that nutrition and cancer biology are too, and sometimes some aspects of economics. I think it matters that all of these are fields that work with humans, though it's interesting that some of them are experimental and it's not all observational.
I read a blog post making the rounds about a decade ago, that I unfortunately can't find now, that put forth the theory that the fields vulnerable to the replication crisis are the ones that both (1) rely on controlled experiments, and (2) don't have a strong theory known to govern the phenomenon they study. The explanation for (1) was something like, if your field is entirely observational studies then that forces it to develop better metis about confounders, or something like that.
I was actually thinking exactly the same thing (about having read a blog post about a decade ago making something like that point), but you added a few more details I hadn't recalled, so I copy-pasted your comment into ChatGPT and it found it:
Paid research is often nothing more than advertising. End of story. When you need to have research to make $$$, then it becomes incumbent on the profiteers to find the most efficient, best ways to make $$$ -- if that's finding folks willing to lie in order to publish... well...
It's all very controversial, of course. Some believe that the whole thing is overhyped, there's nothing to see, it's business as usual. Others claim that the standards are too low, it's feasible to improve them, and it's a coordination problem/entrenched interests that block improvements.
My answer would be essentially yes, the term replication crisis was embraced by many scientists and was accurate, but I think that "is there really a crisis" simply uses the wrong tense.
In the early 2010s, when the news came up, people in science were in a huge genuine shock, and the term replication crisis was spot on. This was a crisis, people all over science re-thought hard about what they can change about it. I think that the general sentiment in science was that this was a good term.
But a crisis, by the very definition is something acute, not something that lasts for decades. The replication crisis WAS a crisis of science around 2010-2015.
This does not mean that everything has been solved. Certainly a lot of things have improved. A lot of journals have changed their policies in some ways to address the problems better. Perhaps the most lasting effect is a general awareness in science that even a "settled field" can break, and that it is more important to test it. On the other hand, crappy science continues to exist, and many problems have not been overcome. But the term crisis for the term of science nowadays is inadequate. Scientists have changed what they managed to change, and have otherwise moved on. For the state now, more adequate terms would be "good" or "bad", but not "crisis".
To clarify then, is it fair to say the replication crisis referred to stuff like “oops, we realized that a lot of our findings were based on observations of grad students who are W.E.I.R.D. and not representative of the general population” (which they’ve since better accounted for and fixed) but in the popular imagination it was mainly about “look at these silly ideologically-captured journals publishing papers about icebergs being literal instantiations of oppressive patriarchy”?
Hm, it was definitely not about the latter, but I don't think the former captures it either.
The problem was not that the findings were only for W.E.I.R.D. grad students. The problem was that when a second person tried to do exactly the same experiment for exactly the same type of W.E.I.R.D. grad students, they would not get the same results.
I agree that there has been a tendency in the last two decades or so to go beyond W.E.I.R.D. grad students, or to test drugs on women, too. Which is a good thing. But I wouldn't link that to the replication crisis. The replication crisis has not much to do with culture wars or woke topics, except that it was picked up later in culture war battles. (Perhaps because social priming was a popular concept that failed to replicate and that was also used in woke arguments. But there were many other examples, too.)
The difference is that if someone told pre-crisis scientists that the outcome would be different for women and for men, or different for people of different age, they would probably have said "Huh, interesting. I never thought about this aspect, but sure, could be." But the actual replication attempts really tried to replicate the original experiment as exactly as possible, and they still failed. And the reaction was more "What? No way! This can't be!". (Most) scientists genuinely didn't expect that to happen, and it was a big shock that raised fundamental questions of how we could even continue to do science in those fields.
It you want to get a feeling of this shock, I recommend an old article by our host: https://slatestarcodex.com/2019/05/07/5-httlpr-a-pointed-review/ . Scott was clearly shocked by this, and this is very similar to how the replication crisis hit many fields a few years earlier.
The big story in academic publishing I'm reading about on Twitter nowadays is Jason Arday's extensive plagiarism, fraud & obvious incompetence being covered up & rewarded. Of course, he's not a psychologist though. As Greg Cochran quipped, some fields didn't have a replication crisis because, unlike social psychologists, they don't care about being wrong or feel embarrassed when others notice https://westhunt.wordpress.com/2017/02/17/the-replication-crisis-in-sociology/
I think you guys are underestimating the general public, or at least the more literate segments of it. There have been mentions of the Replication Crisis and similar academic scandals in Vox, The Atlantic, and NYT over the last several years.
The discussion about replicability was of interest and important to a relatively small population of active researcher and journal editors. Reforms to the process were worth discussing within that small community.
But the spread into a wider cultural phenomenon is, to my mind, entirely about giving sides in the culture war yet another tool to justify dismissing any scientific findings or expert opinions that don't align with their own motivated beliefs.
Your description matches my suspicions, but I just don't know any of the specific details of the history to give it confirmation in my mind. I do think the vast majority of people are primarily motivated in their opinion-having by whatever they decide is their tribal identity, even if it seems (even to them!) like they got there by reasoning.
I directionally agree, but I'm also chronically charitable in some regards and fell the need to complicate that story.
I think that we overestimate the degree to which people accept whatever beliefs their tribal identity is 'supposed' to accept, and underestimate the degree to which people end up with a tribal identity based on the best epistemics available to them from their individual position and knowledge, and from their get stuck in filter bubbles and trusting sources in a way that traps their priors despite proper Bayesian reasoning.
Like, it's definitely a problem that most people end up believing whatever their tribal identity says they should believe, but I think in some sense it's trivializing the problem to conclude that people ignore logic and reason and just accept beliefs for social reasons. I think the much more daunting truth is that it's just really really hard to end up believing only true things through logic and reason, and the shape of our culture and politics and economy is designed to drive certain people away from certain truths even if they are trying to be rational and skeptical.
You can get a proper Bayesian to believe wrong things if you control the evidence they see well enough, and a lot of what we see is very very controlled by various motivated sources, starting in infancy with our parents and proceeding from there.
I appreciate what you’re saying, and there are indeed people who are resistant to tribal identity and its various tugs, but I think broadly I am correct and that the sort of charity you describe is unwarranted.
The problem isn’t that it’s hard to believe only true things (though I agree that the second clause is true: it is hard to believe only true things); the *problem* is that people feel compelled to have strong beliefs about lots of things they know little or nothing about, and to ignore or not pursue knowledge of counter-beliefs that might temper them, and they do this—they feel compelled in this way—because they have internalized a tribal identity.
It is difficult, I theorize, not to internalize a tribal identity, because of how humans are wired by evolution. Even I, a freak, a natural contrarian, a Groucho Marxist who “wouldn’t join any club that would have me as a member”, still have to stop and be intentional about my epistemology from time to time to make sure I’m not starting to reason from after having picked a side.
Ground Penetrating Radar is a fun one. You'd have to actually have studied the physics to know how easy it was to mistake tree roots for "buried children"....
The Americans were laughing about this the entire time, of course, because they knew the failure modes of the GPR, and this was an entirely obvious "RTFM" -- or listen to the Americans telling you how to use the system.
... but, sitting there on the outside, not being part of the GPR physics folks, I wouldn't have known boo. And the scare of "mass graves everywhere" does sound implausibleish...
Wrong culture war, I think. The STEM vs soft science beef is its own kind of culture war, and its the stemlords who turned the replication crisis into a wider cultural phenomenon. Of course people did then use it to reject findings that they didnt like *from specific disciplines*, but thats basically in line with the intent.
“The STEM vs soft science beef is its own kind of culture war”
Among whom? I would say it’s a theater in “The” culture war. In real life i know actual scientists/technologists/engineers/mathematicians as well as psychologists/sociologists/anthropologists/archeologists/etc., and they don’t seem to have this beef with each other’s fields at all; it’s only in the lay peanut gallery where you find this supposed confrontation. I might even say the STEM vs. soft science beef is primarily a plaything of the Very Online. (Everyone else is busy working, they ain’t got time for dat noise.)
In fact even there, the confrontation mostly seems to be one-sided, people motivated by a certain tribal identity who scowl at non-STEM unrequitedly.
Most don't have outright beefs, but many are finely attuned to the perceptions of their own fields and the fields of others and develop various psychological complexes around them of varying severity.
For instance, quite a few economists have "math envy" and have an instinctive need to feel mathematically sophisticated, at once feeling superior to the "softer" social sciences and insecure around hardcore physicists or mathematicians. This is a whole thing (I'm pretty sure there was a famous essay written about it), and of course the better economists understand the point is just to do good economics.
Every field has an analogous "story" of how it relates socially/psychologically to other fields. Mathematicians and physicists mostly have a cordial relationship but mathematicians feel physicists write down equations without knowing what they mean while physicists feel mathematicians get into the weeds about really pedantic things that don't matter or reflect reality. Mathematicians and humanities professors tend to respect each other at a distance as "pure fields", but not know much of what each one does. These are all stereotypes of course, but I think you get the idea. These days because of the larger culture war against the academy as a whole, there is a strong force to unite all the disciplines and faculty often identify strongly with each other as being simply an academic or scholar. But there certainly are complicated (and not all positive) feelings between the disciplines.
I've heard economics nobel prize winners say, "But, it was just economics!" to mean that they didn't actually have to do anything "real world" like an experiment proving that reality was XYZ.
Well, this blog has always been partly about psychology and psychiatry, and science in general, so it's not that weird that people here would recognize the term. It's also something I learned about just by reading fairly mainstream publications like The Atlantic or something, years and years ago. It's not really esoteric knowledge. Something like, say, the Sokal Hoax is more niche and political, and that's something I've heard about primarily through a culture war context, but the Replication Crisis is a pretty normal thing for a semi-educated adult to have some familiarity with.
People built a lot of sandcastles on top of the baseline priming results, and file-drawer problem meant that a lot of obscure/hyper-specific things which probably weren't real managed to get a few articles deep before petering out, but the central phenomenon is pretty solid.
A bunch of consultants and companies latched onto priming ideas to try to make products and sell services and they promulgated a lot of junk ideas, but that's a problem with capitalism, not psychology research.
Steve Sailer • February 5, 2013 • 2,700 Words • 80 Comments
One of the most popular social psychology studies of the Malcolm Gladwell Era has been Yale professor John Bargh’s paper on how you can “prime” students to walk more slowly by first having them do word puzzles that contain a hidden theme of old age by the inclusion of words like “wrinkle” and “bingo.” The primed subjects then took one second longer on average to walk down the hall than the unprimed control group. Isn’t that amazing!
This finding has electrified the Airport Book industry for years: Science proves you can manipulate people into doing what you want them to! Why you’d want college students to walk slower is unexplained, but that’s not the point. The point is that Science proves that people are manipulable.
Now, a large fraction of the buyers of Airport Books like Blink are marketing and advertising professionals, who are paid handsomely to manipulate people, and to manipulate them into not just walking slower, but into shelling out real money to buy the clients’ products.
Moreover, everybody notices that entertainment can prime you in various ways. For instance, well-made movies prime how I walk down the street afterwards. For two nights after seeing the Coen Brothers’ No Country for Old Men, I walked the quiet streets swiveling my head, half-certain that an unstoppable killing machine was tailing me. When I came out of Christopher Nolan’s amnesia thriller Memento, I was convinced I’d never remember where I parked my car. (As it turned out, I quickly found my car. Why? Because I needed to. But it was fun for thirty seconds to act like, and maybe even believe, that the movie had primed me into amnesia.)
Now, you could say, “That’s art, not marketing,” but the distinction isn’t that obvious to talented directors. Not surprisingly, directors between feature projects often tide themselves over directing commercials. For example, Ridley Scott made Blade Runner in 1982 and then the landmark 1984 ad introducing the Apple Mac at the 1984 Super Bowl.
So, in an industry in which it’s possible, if you have a big enough budget, to hire Sir Ridley to direct your next TV commercial, why the fascination with Bargh’s dopey little experiment?
One reason is that there’s a lot of uncertainty in the marketing and advertising game. Nineteenth Century department store mogul John Wanamaker famously said that half his advertising budget was wasted, he just didn’t know which half.
Worse, things change. A TV commercial that excited viewers a few years ago often strikes them as dull and unfashionable today. Today, Scott’s 1984 ad might remind people subliminally, from picking up on certain stylistic commonalities, of how dopey Scott’s Prometheus was last summer, or how lame the Wachowski Siblings 1984-imitation V for Vendetta was, and Apple doesn’t need their computers associated with that stuff.
Naturally, social psychologists want to get in on a little of the big money action of marketing. Gladwell makes a bundle speaking to sales conventions, and maybe they can get some gigs themselves. And even if their motivations are wholly academic, it’s nice to have your brother-in-law, the one who makes so much more money than you do doing something boring in the corporate world, excitedly forward you an article he read that mentions your work.
(“Priming” theory is also the basis for the beloved concept of “stereotype threat,” which seems to offer a simple way to close those pesky Gaps that beset society: just get everybody to stop noticing stereotypes, and the Gaps will go away!)
But why do the marketers love hearing about these weak tea little academic experiments, even though they do much more powerful priming on the job? I suspect one reason is because these studies are classified as Science, and Science is permanent. As some egghead in Europe pointed out, Science is Replicable. Once the principles of Scientific Manipulation are uncovered, then they can just do their marketing jobs on autopilot. No more need to worry about trends and fads.
But, how replicable are these priming experiments?
Idk, the one of these I understand the most of is the heritability of psychological attributes, and this seems sketchy to me (in both "directions" - heritability underestimated due to measurement error, biological determinism overestimated due to multiple factors). Can I assume the others are equally sketchy?
The difficulty is that if your discipline makes claims to lots of 'settled science' - as mine, ecology, does - then stuff gets accepted that shouldn't. And if a discipline is taught to undergraduates it will have a textbook...and you have to put something in the textbook. Everything put in will be treated as settled science unless you say different. One example, we teach logistic population growth in 1st year ecology. Logistic population growth makes quantitative predictions if you have estimates of r (intrinsic growth rate) and K (carrying capacity). But if we take 30 years of data, estimate r and K with the first 29 years of data then predict year 30 - a random walk usually makes better predictions. So, it's very questionable when or where this population growth model we teach in first year actually applies - if anywhere at any time. It wouldn't surprise me if social psychology in a similar predica(t)ment.
Logistic growth is one of those things like frictionless planes in physics, or the tradeoff between inflation and unemployment in economics, where it idealizes away a lot of real-world complications to get at real causal understandings, but isn't very useful for empirical prediction (unless you learn all the factors to add to get rid of the idealization).
Hi Kenny, sorry it took me so long to get back to this. I agree that it ignores lots of other causal factors, but shouldn’t an idealization still make better predictions thana random walk?
Hey NLRG, a grad student of mine published a paper showing this to be true. But that doesn't address why it's true. I expect the logistic population model is a poor predictor because it doesn't include the factors that are the important drivers of population fluctuations. Maybe it's predators or disease or resource availability...but we don't know as yet. But here's the paper that showed the logistic model made poor predictions. https://pubmed.ncbi.nlm.nih.gov/38424456/
When she was doing her master's degree, I helped my wife comb through the education literature. I was surprised how terrible the quality of the arguments and experimental designs were.
I remember one study on tracking (separating kids into gifted+slow learning classrooms). The authors compared the untracked classroom to the various tracked classrooms. By the end of the school year, they "found" average scores for "high track" (gifted) students were better than for the untracked class, but that the average scores of "low track" students were worse. They then attributed this to tracking as a practice ... not to the fact that their study design would have showed that no matter what.
I know what you're thinking, "sure, but what was the change from baseline?" LoL, this wasn't a rigorous study. You thought they'd have a baseline! Meanwhile the people on the other side of the debate had equally terrible studies.
Education researchers are, as best I can tell, worse at statistics than psychology researchers and are mostly dealing with stuff with more political and social salience, so it would be really amazing if educational research was in a better state than psychology research.
I don't want to paint with a broad brush in a field that is not my own. Especially since it's been a decade since I really looked into it. All I know is that what I saw in the late 2010's was shockingly bad.
I think there is both the problem that they have among the lowest GRE scores of any field, and also that they're so politicized they don't care about things like replication.
Out of curiosity, do you remember which study that was well enough to link it?
Education research is always hard because you can't really do controlled experiments, since you're dealing with real children who you have to care about and who live in the real world, and also because many of the effects you'd care about take longer to emerge than the publish-or-die pace of academia allows you to wait.
An additional problem, specifically in the U.S., is that a large fraction of students attending education schools are public school teachers who are there primarily because their collective bargaining agreements entitle them to an automatic pay raise if they get a postgraduate degree in education. This probably doesn't foster a healthy epistemic environment.
Don't remember, though I could go look up her master's thesis and try to find it ... but I won't. I was just chatting with Claude about the state of the evidence today, and it does not seem any better.
To be fair to the teachers, I think these master's programs are often poorly targeted. They don't teach future teachers skills for the classroom, and this exercise was no exception for my wife. I think there are plenty of teachers who would consult the research literature if it were useful and reliable. But since it's neither, I'm not surprised they don't take the exercise more seriously than a hoop to jump through so they can get back to their real work.
I am a bit suspicious whether this isn't just about a particularly bad sub-field of education research.
For example, the first "settled" question of education research that came to my mind is spaced repetition. I would assume that people remember facts better if they don't get presented them once, but with spacing, for example repeat it once after a day, once after a week, and once after a month. (Even if the total exposure time is the same.)
My naive assumption is that this is established by high standards. Like, that it is reproduced dozens of times with good study designs that vary all kind of details. And that it is also studied in practical classroom settings and holds up there. Is that not the case?
The second settled question that came to my mind is that you learn interconnected facts easier than isolated facts. The third one is that you learn better if you are emotionally engaged than if you are emotionless/bored. The fourth one that students learn better if they are activated. All of these are very relevant for practical teaching. Also here, I was assuming that all these are supported by high-quality studies. So what is actually the quality of studies for these questions?
Ok, I don't know what exactly counts to which fields. But surely these things have been tested in classrooms as well, so it is also studied by education researchers, or not? Or are the lab studies by psychologists good and the classroom studies bad?
Activation is the concept that the students should actively do something during lessons. For example, make them think for themselves instead of just listening to you, or have them explain the concept to their neighbor in their own words, or let them vote on multiple-choice questions. In the broadest sense it could even be something unrelated to the material, like have all students stand up that understood the last 5 minutes. Whatever prevents them from just passively sitting in their chairs.
Though GPT is skeptical of the last type of activation that is unrelated to the material. I think it is a common concept, but it does not seem well supported.
These is a subtle difference between the well established research on effective learning techniques, and much of the dubious research on effective teaching techniques - especially as they relate to K-12 education.
The difference is whether the question you're asking is, "How do I learn and retain this information?" which is different from, "How do I get a diverse group of reluctant children to learn and retain this information?"
I heard about a university research that tried to experimentally verify Growth Mindset.
They split students to two groups. One of them educated traditionally, examined at the end of semester. Another group got an extra lecture on the importance of growth mindset, and also they did the same exam three times, so that they could see how they are improving (to support the idea of growth mindset by personal evidence), but only the third attempt at the end of the semester actually counted for the final grade.
Amazingly, the second group did much better. Growth mindset proved, checkmate IQ believers!
I asked: "Wait a moment! Is it possible that the second group did better simply because they had two extra opportunities to *practice* their final test?"
Response: I guess that might have helped a little... but most of the effect came from the growth mindset. (No evidence for this statement provided.)
> Most supplements are stuck in Poor Quality Study Replication Crisis Hell, but melatonin and l-theanine stand out as especially promising
Also on supplements, most nootropics studies that I've seen seem to understandably focus on the metrics that are the easiest to measure, like reaction time or N-back performance, but these are only tangentially related with what someone taking nootropics wants.
"The dumb woke psychology was wrong and bad; it seems useful to punish/discredit psychology as a whole to assist my campaign against it, so for propaganda purposes I will try to discredit psychology as a whole"
I agree that doing this is bad, and that people who reason this way have the problems you describe.
However, 'I ought to punish bad actors who peddle partisan nonsense using my money' is at least reasonable. Cambridge delanda est, etcétera etcétera. I think it is reasonable to hold partisan actors to partisan standards. But with this must be drawn with a line protecting non-bad-actors, and, ah, certain figures who are making anti-Harvard decisions have not been doing a perfect job of it. I prefer their route, to put my cards on the table, but for other reasons. On the limited topic of science, I think it is useful to split 'these researchers signed on to statements saying x y z, either lying for power / their own gain to do so or genuinely believing it, and this is disqualifying' from '[insert field I dislike] is inherently bad'.
It's hard to disentangle those. A field may not be "inherently bad", but it may be in practice bad.
There's also the problem, not specifically of money, but that the dumb woke psychology is being used as an excuse for drastic social and political changes. Using my tax money is bad, but using policies based on bad psychology as an excuse for forcing your political beliefs on me is worse. Not a lot of people are demanding policies based on "oh, we found the receptor for psychedelics", so pointing out that that is genuine knowledge is beside the point.
True. Different kinds of science (and 'science') are likely more or less vulnerable to this sort of thing. At the same time, I think that nonsense will find a way (take, idk, rent control or anthropogenic climate change denialism), though coming from sources other than universities will greatly improve a country's epistemology.
But with AGW, you have denialists from one side saying it's all made up, but also fabulists from the other side saying global warming is going to destroy the world or extinct mankind or something. But it sure seems (from my outsider perspective, with many problems with my filters and information sources) like only one side of that gets a lot of organized feedback from climate scientists.
I think this is in part because the denialists are saying things that are confidently known to be false, while the alarmists are mostly stating as established fact things that are possibly true but uncertain. I agree that I would like experts to do a better job correcting the latter kind of misinformation, but it's harder to do since people are entitled to their own opinions about ongoing scientific controversies.
(Also, it takes time for things to be corrected; today I think most climate scientists are pretty sure that the most catastrophic climate-change scenarios aren't going to happen, but that wasn't true 15 years ago.)
Climate scientists aren't neutral truth-lovers, as a class. They generally believe that AGW must be mitigated, and if it takes a few white lies to move the clueless masses in that direction, then those are amply justified.
Climate scientist here. One important factor is that the false statements from denialists and fabulists tend to be different in kind. For example, a common talking point for many denialists is that carbon dioxide can't possibly be having a significant effect on present-day climate change. That's incorrect as a matter of physics. But you won't hear many fabulists saying microplastics are having a significant effect on present-day climate change, which would be similarly incorrect as a matter of physics.
The false (or at least unlikely-to-be-true) statements from fabulists come much farther downstream and are often a matter of interpretation. For example, a quick google turned up this reasonably nuanced discussion on whether climate change is an existential threat to humanity: https://climate.mit.edu/ask-mit/why-do-some-people-call-climate-change-existential-threat
While I agree with the general argument, I think that comparing the solidity of those facts to change of paradigm in astronomy is maybe a bit too much. To just single out one instance over which there is very good evidence that the mainstream interpretation is probably wrong, most of the simplified picture of cortical reorganisation that is included in the list actually seems to come mostly from the wrong notion of brain function localisation (ie, many areas being already partially involved in the functions they help recovering). For a very good exposition I can recommend https://elifesciences.org/articles/84716
re astrology: I've met a number of people who I believe would benefit from learning the first rule of astrology - different people are different - that doesn't make it a science, though, just entertaining and perhaps useful.
One difficulty is that it is hard for outsiders to distinguish between the parts of a scientific field that are pretty solid, the parts that are probably more-or-less right but still under construction, and the parts that are extremely shaky. This is obviously made worse by media outlets that report on the latest n=5 study showing that too much social media exposure turns frogs gay or something.
But also, a lot of high-prestige scientific organizations, publications, and universities have visibly and loudly make pretty nonsensical public statements as part of various culture war battles or current hot political issues. And again, from the outside, it's not so easy to tell that in this case, that's Dr Schmoe, whom everyone knows is kind-of a twit who will say whatever gets her in the papers, or that in this other case, the organization is making some public political statement and most experts in the field are kind of wincing and remaining silent rather than get dragged on social media.
A nice example of this was that infamous open letter by public health experts about how after George Floyd was killed by the cops, it was okay to protest because racism is a public health crisis, too. As I understand it, the actual people who signed the letter were mostly not very important or prominent people in the field, but of course as an outsider, I had no way to know that without doing a bunch of extra research.
But also, a bunch of gender ideology and race ideology stuff got put out during the Great Awokening that was obviously someone reading the room and saying the current thing without any concern for being factually accurate. And mostly that seemed to go without pushback from actual experts in those fields, though it's not so clear if that's a problem with who was heard or who was willing to speak. When I see a respectable institutions spread obvious nonsense for political/social reasons, I am much less willing to give their other statements much weight.
As an outsider, how should I evaluate what findings/expertise from anthropologists or sociologists or public health experts or psychologists is actually very solid, what is shaky, and what is just them saying what some donor or an angry Twitter mob wants them to say?
Wasn't there quite a bit of work on stereotype threat? And a lot on implicit bias tests? As I understand things, neither of those have held up too well....
Important note on implicit association tests: This refers to multiple different findings. "People have faster reaction times pairing socially privileged groups with positive-valence concepts, and socially oppressed groups with negative-valence concepts, than vice versa" has held up fine. "People who have larger reaction-time differences on those associations are more likely to behave in prejudiced ways" has not.
Yeah, that fits my understanding too. As I understand it, IATs repeatedly give consistent results, peoples' scores are pretty stable, etc., but they aren't useful for predicting behavior. Which makes them oversold (I think people were offering IAT testing/training to police departments to reduce racial bias), but not false.
I'll never forget about The Smithsonian infographic claiming that punctuality was an aspect of white supremacy.
Zvi's exhaustive covid coverage let me build a model with enough gears that I generally feel comfortable assessing public health claims...no idea about other fields though. Scott used to do this for psychology and various other science things, but that's been fewer and farther between as the years go by, and it feels epistemically shaky to keep referring back to SSC posts that are sometimes 10+ years old. Surely the consensus has changed since then! (There's a lot of good bonus content on his Lorien Psych site, but it's nowhere near as fun to read, and I always forget it exists.)
And again, it's hard to tell from the outside how much stuff like the Smithsonian infographic (which was embarrassingly bad) represents actual elite thought vs some dimwit given a bigger platform than they deserved. Though this also illustrates the problem with institutions giving a pass to low-quality scholars/contents with the right partisan or culture-war valence.
If I read some internet article about a published result that makes some culture-war arguments and clothes them as science, it's pretty easy for me to know that I shouldn't give that a ton of weight--the writer probably wasn't trying to find truth, they were trying to find clicks.
If I read an op-ed from a professor at Harvard making some culture war argument and clothing it in science, or see some display put up at the Smithsonian, or read an editorial in Nature doing that, the "it's just some nobody clown getting a megaphone to make you mad" defense doesn't work anymore. Those are prominent institutions putting their prestige on the line. If many Harvard scholars show up in public to speak nonsense, I'm going to lower my opinion of Harvard.
The Smithsonian infographic was actually a reasonably good and concise summary of the cultural norms of modern Western civilization. It just missed two critical details. First, Western culture *works*, and for empirical metrics like peace, prosperity, scientific and technological achievement, it works spectacularly well. And second. while invented by white people, it has long since adopted an open-admissions policy such that e.g. pretty much all the black people I know are full members.
I'll never forget going to the Smithsonian, and looking at exhibits shortly before they were reported (and removed) for wrongthink (Asbestos endorsing doctors, cigarette endorsing doctors, very bad things to be mentioning in a covid19 era where "trust the doctors" was the watchword).
Mostly because this occurred circa 2020, and around 2025, the Smithsonian decided to make a big stink about the government wanting to decide what it got to put up (as if this had never happened before).
That's why the "doctors endorse cigarettes" was such a thought-provoking exhibit. And also why it had to go (notably, without the Smithsonian complaining about the expungement of wrongthink).
You should assume that most findings are false, of course. Some are even planted by enemies. And sometimes, the idiots just don't read the directions (The Americans that had used the GPR first started laughing when they saw the Canadian conclusions).
"Instead, you can say the thing you mean - that woke partisans shouldn’t try to occupy and exploit the otherwise-noble science of psychology for their own political gain"
I think part of what's driving the Camp 2 people is that academic research psychologists -- of whom Scott, notably, is not one -- didn't say this. When the microaggression industry called itself science, did we see a meaningful movement of workaday academic psychologists denouncing them? We saw the opposite; that seems important.
Well the APA did their thing, just as you'd expect, and social media denizens did theirs. But the background professional sentiment was that microaggression work was good:
I seem to recall that some fraudulent studies involving people being committed to mental institutions (it was all made up, IIRC) were a big part of the movement to deinstitutionalize. This seems kind-of relevant. If the good science is there and known to the practitioners, but the bad science is used by congressmen and judges to make policy, we have a big problem!
>If the good science is there and known to the practitioners, but the bad science is used by congressmen and judges to make policy, we have a big problem!
Eh. My impression is generally that congressmen and judges are going to do whatever they want, and will use whatever they can find to justify it post-facto.
If there's some fringe science they can misinterpret to justify the thing they already wanted to do, great. If not, who cares, they'll appeal to traditions or christian values or public safety or whatever-the-fuck works in the moment.
I don't think the science has any casual role here most of the time, so I'm not worried about it existing for this reason.
Congressmen and judges follow public opinion, and it's possible for studies to shift public opinion. I suspect this happened with the Rosenhan experiment.
Surely sometimes people are just looking for a study to support what they want to do anyway, but also, I think that people who read about this result when it came out probably thought it was telling them something true about the world, and their subsequent decisions were bad partly because their model of the world was wrong.
"To people in the second camp, my response is: deliberately exaggerated propaganda campaigns are bad and this is a confession, not an excuse. Inability to view science through anything but a political lens was what got us into this problem; doubling down on it is a temporary solution but a recipe for permanent disaster. If you pronounce entire fields of science to be acceptable collateral damage in your campaign against bad political actors, you will alienate not just scientists but patients, clinicians, and general lovers of truth. Instead, you can say the thing you mean - that woke partisans shouldn’t try to occupy and exploit the otherwise-noble science of psychology for their own political gain - and wage the same campaign with a halo rather than a comic-book-villain goatee." I agree in theory--science should not be politicized but it always has been. There's a reason Gaileo was persecuted, his discoveries challenged the authority of the Church. Scientific findings and truth(and truth in general) are challenges to power of incumbents, the legitimacy of society and traditions etc etc. At some level bad actors will always use scientific findings to push their agenda or alternatively disregard science when it goes against their political interest. The only way out is for good scientific actors to have enough legitimacy in a political sense(democratically popular not just correct on merits or actually factually correct) to be able to counter bad actors. Science will always be politicized/weaponized for those who have or want power to enact a particular agenda--only actors who have both political and scientific legitimacy at the same time will be able to stop that.
The Church claimed authority over theology, rather than astronomy, and didn't approve of his use of theological arguments in support of his astronomical ones (which happened to be incorrect in the case of explaining the tides).
I agree that it's a disaster that science has become so politicized, and that the likely outcome of that is people making worse decisions and convincing themselves/others of more nonsense. This is a big problem that needs to be fixed!
I think most of the work here has to come from within the individual fields. If people are speaking politically-convenient nonsense in the name of your field, you need to try to oppose that, or the public (including very science-positive people like me--I'm a working researcher FFS) will discount your field's findings as being mainly about politics and not getting the best available picture of reality. If the people at the top of your field are making public statements about how activism is the core of your field, okay, I'm probably going to believe them. If it turns out that being known to have the wrong politics or publishing results / doing research in areas with the wrong political valence blights your career in that field, and I learn of it, then again, I'm going to assume that I can't take the findings of your field at face value. The main people in your field have made sure of that.
One of the many ways this goes wrong is that it's hard for outsiders to know which fields are real vs BS fields. A math professor may have silly political or social views, but his actual field is serious. Is the same true for a gender studies professor? That looks like a field that's mostly about ideology and politics, but maybe I'm being unfair--how would I tell?
Gender Studies is mostly about ideology, but so is Climate Change. And you wouldn't believe how much trouble you can get in for nominating someone for a Nobel Prize in Physics (again, see Climate Change -- notable nobel prizewinners have commented on the relative bunkness of the field).
As a Social Psychologist by training, I really appreciated the recent article. I also agree with this update. Relating to the microagression example, I would add, as someone whose graduate research involved aggression that the whole concept of microagressions isn't actually from social psychology. It originated in the 1970s, but wasn't really studied until the 2007. Those researchers come from clinical and counseling psychology. In my experience, social psychologists don't cite that research and it's considered pretty unconvincing among actual aggression researchers. Scott Lilienfeld wrote a great critique of the microagression literature that captures my thoughts on it pretty well: https://homepages.se.edu/cvonbergen/files/2018/05/Microaggressions_Strong-Claims-Inadequate-Evidence.pdf
This seems like a combination of (a) not psychology, (b) meaningless tautological statements that are unfalisifiable, (c) things already known or suspected to be false
(1) just says "there's a prediction-error-based learning algorithm in the brain". I never understood what predictions that makes and what it would mean for it to be false
(2) is not really psychology
(3) is false I think? Your links don't demonstrate that the brain can use neuroplasticity to recover from damage to Broca's area
(4) is controversial but seems likely to be false, as new molecular heritability studies place certain heritabilities closer to 30%, while old twin studies (and old SSC articles!) place them at 80%. Unless you mean that everything from 20%-80% is "about 50%", I guess. As for "environment does nothing", see birth order effects on SSC readership!
(5) is nonsense; your link doesn't actually give any evidence that brain waves have to do with consciousness, it's just pure woo and "woah, waves, isn't it cool" level stuff
some of the later ones are a bit better, I'll grant.
Isn't the heredity issue basically the difference between "we observe this heritability based on looking at identical vs fraternal twins" and "we found a set of genes that account for this level of heritability?" I wouldn't take GWAS studies as the final word on how heritable things are, just as a lower-bound on stuff we can pretty-much know.
No, the modern molecular studies are not GWAS. They look at random variation in how much DNA is shared by siblings (on average it's 50%, but it is sometimes 40% and sometimes 60%), then check whether more generically related sibling pairs are more correlated in phenotype; there's no need to find the actual set of genes responsible.
I will simply repeat my point from the previous post: we shouldn’t care about results before they have been replicated three or four times.
I suggest we in this community who value truth memorize and deploy this stock phrase: "yeah but has it been replicated?" And simply ignore any new studies or findings that have not yet been sufficiently replicated (no matter if it confirms our biases and hopes and dreams or not!)
This is a reasonable stance for the type of social psych experiments that can be done on any group of 50 undergraduates, but generalizes very badly to other fields.
I don’t think the excuse "it’s hard to do" makes it any better. There is an argument that if the effect size is large and also N is large in the original study, maybe you don’t need so many replications, but I will hold firm that you do need at least one independent replication in absolutely all cases. That should be table stakes for something we call "science".
It generalizes very well to fMRI studies! They're generally light on the N. Do ten of them, and you get a decent (replicated) result. Expensive studies mean you're trying to get the most out of the least amount of people (it's even worse if you need to get people who have already been admitted, and are as yet untreated.)
> > Freud was “settled science” for a long time ... Astrology? Phrenology? These seem different in that their entire orientation was flawed from the start...
I could be wrong, but from what I understand, Freud's conclusions rested on the kind of methodology that wasn't very much stronger than the methods behind astrology and phrenology. If that's what passes for "settled science" in psychology today, then maybe the entire field *should* be discarded.
I agree, but at the same time, modern medicine had discarded e.g. Humorism. If modern doctors still considered Humorism to be "settled science", then yeah, I'd argue that modern medicine is due for a serious scrubbing if not a total rebuild.
Traditionally, twin studies. In more recent decades, genotyping. (Lately the two have given seemingly contradictory results, which has created a spicy scientific controversy that Scott has blogged about a few times.)
>This isn’t a claim that settled science is always right - Freud was “settled science” for a long time
I always want to stand up for Freud a little bit. Today we hear about his wackiest bits of psychoanalytic theory as color commentary about how weird the field used to be. But his most substantive contributions were ideas like 'maybe significant events during your childhood affect your personality and behaviors as an adult' and 'maybe you are doing a lot of mental processes outside of direct conscious experience that still affect you in profound ways'.
These paradigms are so core to our understanding of everyday experience that they seem 'obvious' to us, like something no one would have had to 'invent' because obviously everyone would know them just from living their life. But AFAIK these were really not obvious and accepted truisms before Freud popularized them.
This is sort of a 'reading philosophy backwards' thing, it's hard to appreciate the contributions of people who popularized ideas that are now so widespread that they seem 'obvious' and sourceless.
To the extent that Freud was correct about these things, was this due to his robust methodology, or was it because even a stopped clock is right twice a day ? My impression is that the latter might be more accurate...
I think that the "stopped clock" framing is a bit unfair. Fields of study with strong methodologies don't emerge already fully formed - somebody has to obtain the initial vague good ideas through some other means. I'd say that it's fair to credit Freud with doing just that, and at the same time to blame him for constructing an enormous edifice of nonsense atop that meager foundation, although everybody else deserves much more blame for taking that nonsense seriously.
I hear what you're saying, but Freud wasn't born in Ancient Sumer. He was (roughly speaking) a contemporary of Maxwell. By that time the scientific method had already advanced beyound merely thinking really hard about aesthetically pleasing ideas, and toward rigorous measurements and hypothesis testing via probability theory. So Freud IMO has no excuse -- but you're right in saying this also applies to any people who took him seriously.
Some of this is likely hindsight bias - Popper's falsifiability was still a few decades away (largely in response to Freud & Co, as it happens), and it's inherently more difficult to be rigorous in "squishier" domains (something that's still clearly relevant, considering that "maybe psychology is all bunk" is a very topical idea).
That is a separate problem. Climate also changes over time, yet it is possible to study climate quantitatively, as opposed to just going on vibes, as psychology appears to be doing.
Well, I don't think comparing the climate to people's personalities is a very good comparison. However, I'm not entirely sure what you're throwing into the basket of psychology here, and I'm not trained in the field. I have my own experiences with it and my own opinions about it, but that's my own personal weather, isn't it?
How much of the good commonsense stuff did Freud actually originate, though? The idea of a subconscious/unconscious mind dates back to Plato, or at least that's what my own intro-psych textbook said.
Right, and I doubt Freud was the first person who thought childhood affects adult personality. Anyone who thought about education would fall into that category.
I don’t think you are interpreting Harris work correctly. I think her point was that peer groups are much more important than parents in one’s behavior. It was not that significant events had no effect. Of course, the whole thing exists on a Bell curve in that more normative upbringings, shall we say, tend to resolve themselves without any great influence, but psychology only really gets interesting when you’re dealing with the tails
One of her points was that even the Holocaust didn't seem to affect survivors that much, and that surely qualifies as a "significant event". And there's theory behind this: Trivers pointed out that there's a genetic conflict of interest between children and parents, so that if children could be molded by parents, the parental interest could win out. But if we've evolved to be resilient, then we'll continue to be whatever we were going to be regardless of "significant events".
Well, I suppose it depends on the survivor. There’s lots of accounts of people who were affected very deeply. I would say we are going to be what we will be including our significant events, not in spite of them. A lot of it depends on at what age a significant event occurred. I don’t think they had much patience for small children in concentration camps.
There's a common belief that there's a "formative age" that matters, but per Judith Harris the reason not to abuse the child is the same as the reason not to abuse an adult. Steve Sailer likes to quip that the relevant time keeps getting pushed back further as studies/interventions fall prey to Rossi's Iron Law, to the point that it can be nine months prior to birth BUT NOT MORE!
>the reason not to abuse the child is the same as the reason not to abuse an adult.
no doubt but the reason is not the issue.
I believe brain development in humans has been well studied and divided into stages. Things land very differently depending on age and circumstance. Or does Steve Sailer believe that a young child can be subjected to physical, sexual or extreme emotional abuse by their caregivers and it ain't no big deal?
This discussion feels like it's turning into a circular firing squad. I don't have a very high opinion of Harris and too much depends on the specifics for generalized conclusions. As I said before, it's the tails of the curve where things get interesting. Even "a significant event" is highly contextual.
> To put things in context, to show how diseased psychology was in Freud's era, let me just say that the most famous Freud's student and then competitor, Carl Gustav Jung, rejected much of Freud's teachings and replaced them with astrology / religion / magic, and this was considered by many people an improvement compared with the horribly offensive ideas that people could be predictably irrational, motivated by sexual desires, and generally frustrated with the modern society based on farmers' values. (Then there was also the completely different school of Vulcan psychologists who said: Thoughts and emotions cannot be measured, therefore they don't exist, and anyone who says otherwise is unscientific.)
> I think the bad PR comes from combination of two facts: 1) some of Freud's ideas were wrong, and 2) all of his ideas were controversial, including those which were correct. So, first we have this "Freud is stupid" meme most people agree with, however, mostly for wrong reasons. Then, the society gradually changes, and those Freud's ideas which happened to be correct become common sense and are no longer attributed to him; they are further developed by other people whom we remember as their authors. Only the wrong ideas are remembered as his legacy.
A lot of the parts of Freud that weren't crazy just sound like common sense today, but they mostly weren't at the time he was writing.
He was also one of the first people to say "hey, maybe we can study people's minds scientifically just like everything else" and actually have it catch on.
>I won’t dispute this as an accurate description of how the sausage gets made,
I would dispute it inasmuch as I don't think the science is the driving factor in most of these cases. Rather, I think the typical story is that there's a political movement that's looking for any rhetorical tools it can find to justify its own necessity and increasing power, and skimming journal articles for any finding it can appropriate or misrepresent to its own ends is just one of the dozens of avenues it uses to do this.
No matter what the scientists believe or prefer, I don't think there's much they can do to stop political movements from using them this way. I feel like I very often see the scientists in question going 'no, stop it, that's not what my research means, those claims aren't justified by my findings' and getting ignored as the political movement keeps on citing them.
Which isn't to say there aren't also cases where the scientist in question sees an opportunity for fame and fortune, and joins the political movement as a willing stooge. But even then, I still see the political movement as the driving factor; the existence of the science is not itself the causal force creating the political movement people are complaining about.
My perspective is undoubtedly skewed, as I wouldn't have a way to know that a scientist protested against misuse of their research. Given that, most examples I do see involve people jumping on the bandwagon. In fact, it's not just individual authors. Plenty of examples where major governing bodies (AMA, APA) and respectable publications were happy to do the same.
An example the other way from many years ago was the APA statement on IQ and intelligence, published during the controversy after _The Bell Curve_ was published. My impression (it's not my field) is that this was in fact a sober and honest snapshot of what was known at the time about intelligence and IQ.
Right. IQ is a field of which tens of millions of people with college degrees have 180 degree wrong notions of what is the settled science.
There are few easier ways to make yourself subject to rage and name-calling than to mention the settled science regarding IQ and race, even while carefully explaining that the fundamental causes of these endlessly replicated racial differences in average IQ are not at present settled science.
Someone on DSL had some line along the lines of "politics+science=politics," which I think is basically true.
Science is hard, and social science is so damned hard we can just barely do it. Adding in current political battles, ideological concerns, and threats of outrage campaigns is like hitting a delicate piece of equipment with a sledgehammer.
I don't think camp 1 is as simple as you make it out to be. Psychology as a field has been used for decades now to advance progressive social causes, primarily by medicalizing bad behavior in order to garner sympathy for the practicioners (the poster child is alcoholism). Was "woke psychology" meaningfully different or did the movement just get out in front of its skis a little bit? Apparently the scientific mechanisms can't be relied on to distinguish between the hypotheses.
I guess the biggest blow to popular psychology is, despite the frequent confirmation of genetic inheritance you have mentioned, it's still fought tooth and nail in every step, like you have covered. So not only popular psychology not have too much replicable results, they reject ones they do have.
Any suggestions for members of the general public who want to know if a particular claim is part of settled science rather than an unsettled part of the research frontier? In the absence of enough time to read through a textbook (which may or may not cover the particular claim...), and the absence of a professional in the field with time to spare to answer such questions, are SOTA LLMs fairly accurate or not on "Is <claim X> settled science?" questions?
I think they're pretty good. I don't ask whether something in psychology is settled science, though. I write something like "where I left off the prevailing view was this [and I describe it in a paragraph]. Is that still the prevailing view, or has more research and thought cast doubt on the it, or has some new view replaced it? Please search the last 20 years of studies and articles on the topic using only higher-quality sources such as juried journals, paying particular attention to well-done meta-analyses, then summarize the result, giving links to the most relevant studies."
I think there's a "the barman told the man in the labcoat with the 'I p-hack to achieve social justice aims' button to leave because he was a bad scientist, then took a funnel plot with nails in it from the back room and chased him away" story going on here, in that if you let bad scientists hang out in your bar then it's now a bad science bar.
Ironically, all the bad scientists getting chased out of the psychology bar has caused the general public to notice they were in there in the first place, uninterrupted, for quite a long time.
Have the scientists actually been "chased out"? Dan Ariely is still at Duke. My recollection from Andrew Gelman following a lot of this stuff is that Ariely's co-author Gino is the exception for being kicked out.
It's remarkable to me how many different groups of people in positions of great trust and authority--police, priests, doctors, researchers, teachers, military officers, etc.--follow the same instinct to close ranks and protect their own from consequences of even really terrible behavior.
This defense of the field of psychology bears the characteristics of a motte-and-bailey fallacy. You have established a motte of mostly neuropsychological findings, and you are using this motte to defend a bailey consisting of trash research that has a far larger on people's daily lives than the neuropsychological part does. Furthermore, these neuropsychological findings rest on different research methods from those employed in the rest of the field of psychology. I say we abandon the bailey altogether; it has proven to be barren ground in these past decennia.
Abandon what? Any attempts to understand how people and groups behave, which as you say has a large on people's daily lives? It would have to be restarted somehow in a better form which is easier said than done
I think there is sidficient and meaningful evidence that social psychology is about as meaningful phrenology or zodiac signs. It must no longer be understood as a science. Instead this kind of research should be understood as a literary endeavour.
Dostoevsky foresaw this: “the whole work of man really seems to consist in nothing but proving to himself every minute that he is a man and not a (machine)”
It is impossible to stop the self-reflexive loops of consciousness without destroying a mind. The attempt of psychologists to formulate rules for the functioning of the mind is in its final reduction nothing else than an attempt to destroy our souls. If any mind reads and understands the ‘scientific laws of psychology’, this mind has definitionally superceded these laws by placing them inside a larger epistemological framework, making the laws meaningless.
We should not stop attempting to understand ourselves, but we should stop pretending that introspection has anything to do with the scientific method and falsificationism. ‘Psychology’ is much more akin to an art form or a dance, and it is all the more valuable and beautiful for it.
This seems to be a consistent blind spot for Scott. He sees science (true/false) issues where others see people (honest/liar) issues.
A good example is his claim here that people cannot distinguish good psychology from "dumb woke psychology". But to many, the folks who decide what "settled science" is, in psychology, lost credibility. The "dumb woke psychology" was literally "settled science" not long ago.
Same thing with covid arguments. Scott says "here's the most recent evidence, let's evaluate it". And I say "that person lied before, why should I trust their evidence this time?" People even invoked "settled science" to shut down inconvenient information during covid, so it's odd to see the term used unironically here!
That's also part of the appeal of populist politicians. "I may not be fully on board with the [DSA|MAGA|gold standard|canine personhood] folks, but they have not lied to me (yet)." Conversely, nobody gets worked up about string theory, except physicists, since people behind it aren't trying to sell me on anything.
I think there're interesting things to be learned in this area, e.g. it's entirely possible that the people-based evaluation method is not optimal. But the "people are either stupid or trying to discredit psychology" is not it (see "it’s worth examining the trains of thought..." section).
To get out of the politics & culture war area, the opioid crisis is another example. For a while, it became "settled science" that opioids weren't particularly addictive, at least among doctors. American Medical Association and American Society of Anaesthesiologists were more than happy to jump on the marketing push from Purdue Pharma and Johnson & Johnson.
I don't know how to operationalise that information, it's not like whenever I hear of medical advise I do the opposite. But it does make me a little less trusting of "settled science" and a little more likely to "do my own research".
Yeah, it's worth remembering that there's a constant pendulum swing on opiates/opioids for pain control, because there is a genuine godawful tradeoff between turning people into addicts and leaving people in unbearable pain.
Part of the argument made there is that people who aren't taking the drugs recreationally really are much less likely to become addicted. Of course, once the drugs existed, people WERE going to take them recreationally (defeating some mechanisms the pharmaceutical companies used to prevent getting a short intense high vs long lasting smaller effect).
>This seems to be a consistent blind spot for Scott. He sees science (true/false) issues where others see people (honest/liar) issues.
I thought a similar blind spot was evident in Scott's comments in the recent links post about AI being better at persuasion than the most persuasive people. I can believe AI presents a fuller, clearer, better-reasoned argument, and for that reason gets more votes from subjects for the case it was arguing. But full, clear written arguments are only one form of persuasion, and votes for good argument quality are only one measure of persuasion. It's clear that in real life a great deal of being persuaded happens via social identification, peer group pressure, being swayed by recurring unnoticed messages, vibes, seduction, direct and indirect rewards for having certain beliefs, etc.
MAGA lies all the time, even more often than other politicians. I will also say that I doubt that you actually can show that the sources Scott cited on Covid lied.
I'm generally in agreement with the post, but one potential nitpick regarding example 4 on heritability:
"but the further you go from this basic picture, the gnarlier the chains of indirect causation you require."
Isn't that put the wrong way around? To my understanding, other methods of estimating heritability at least partially get lower estimates by trying to control for anything but direct heritability, therefore removing gnarly chains of indirect causation. It's twin heritability that includes them all.
> Psychotherapy works with moderate effect size, but (with a few exceptions) seems to be more about the relationship with the provider than the details of the therapy itself.
This is a common but IMO misleading conclusion. Common factors research (which includes work on the therapeutic alliance/relationship) is almost entirely correlational (so it is confounded and many studies cannot even rule out reverse causation, where remission causes you to evaluate the relationship to the therapist positively), whereas the effect sizes we have on therapy effectiveness come from RCTs. The differences between therapies research comes from mainly underpowered RCTs which cannot realistically detect the differences we would expect. Presumably, differences between therapies that you can find “in the wild” are larger than what you can expect in RCTs, which have more quality control in all arms.
When randomized to patients, therapist effects explain only small percentages of variance.
So, I think the “alliance/relationship” stuff is mostly confounding and I would never confidently say that the details of therapy don’t matter (which also would require you to hold other beliefs included here, like that we understood reinforcement, which is core to CBT but not analysis, very lightly). IMO we have good evidence that CBT is at least more efficient, probably also more effective, I think this is even more true in the wild, when we are far removed from the quality control of RCTs
A little nitpick to 5: I think the term "subpopulation of neurons" is misleading, because it sounds like a permanent division. My understanding is rather that neurons spontaneously assemble into synchronized groups through brain waves. But the set of neurons which participate in a group or even generally which respond to a specific rhythm changes all the time.
I have never looked that deep into brain rhythms, but a quick GPT check (that I tried to phrase neutrally) seems to back me up.
But yes, brain waves exist, are important, and are linked to states of consciousness.
>"Most personality traits and general capabilities are about 50% genetic and 50% random, with little role for family environment or childhood history, with a few notable exceptions like partisan preference."
Sorry…what? Is this meant to imply that abusing or neglecting your children has no impact on their adult personality?
This statement is heavily simplified and easily misleading. I believe the source is that twin studies show 40%-50% of the difference in personality is "explained by" heritability (genetic and non-genetic), but heritability is a very confusing thing to measure, because e.g. most twins that participate in adoption studies end up in "safe" families (due to adoption requirements/patterns) so they have similar environments and therefore less of the twins' differences can be explained by environmental differences. The studies naturally include very few abusive households. In a world where all children are genetic clones raised in perfect families, you'll see studies saying that personality traits are "0% genetic/heritable and 100% random", but obviously if one child has a severe mutation or gets orphaned that child will be very different.
Yeah, the concept of "all else being equal" or "at the margin" is important here.
As I understand it, the best evidence is that among adoptive families (which are screened to be relatively well-off and stable), we mostly don't see big impacts on life outcomes from differences in upbringing.
That doesn't tell you that having a kid raised in a cave by a biker gang won't affect his life outcomes, nor that having him raised on a space station surrounded by geniuses won't affect his life outcomes. Just that *within the range of variation of the people in the study* we don't see a lot of those effects.
Very little in reality has cutoffs like that so I’m sceptical that there arent effects in the normal range given there are effects at the margins, even if the effects are smaller I the normal range.
But anyway - whether you have loving parents,’good role models, people who talk to you and read you stories, older siblings who torment you and make you feel ugly or small—-you really telling these don’t affect adult personalities?
I'm saying that the studies did not find big effects of upbringing on the measured outcomes. That doesn't mean there aren't other outcomes that changed a lot, or that there weren't either small or rare effects that didn't show up in the statistics.
There was a French adoption study where they tried to find 40 adoptees, ten each in four quadrants: born rich/raised rich, born rich/raised poor, born poor raised rich, and born poor/raised poor. But they could only find 8 kids for one of the Prince and the Pauper segments.
Anyway, they found a more sizable effect of Nurture than is normally found. This could just be the tiny sample size, or it could be that most adoptees during the golden age of adoption agencies (roughly 1945-1975) got pretty decent upbringings so we don't know much about what really bad parenting can do. For example, even as critical-minded of a personality as Steve Jobs was wholly satisfied with his blue-collar but loving, stable, and honest adoptive parents.
"The gamma rhythm corresponds surprisingly well to items in conscious awareness."
This is not settled science.
There is plenty of solid science on theta-band oscillations (especially in the medial temporal lobe in relation to memory function), alpha-band oscillations (especially in the occipital cortex and thalamus in relation to attentional allocation), and beta-band oscillations (especially in the motor cortex and basal ganglia in relation to the timing and organization of movement).
However, the gamma rhythm is measured with lower signal-to-noise than most other rhythmic processes, is more subject to various kinds of confounding processes (e.g. muscular artifacts), and also is subject to extensive terminological and methodological confusion (gamma band vs high-gamma band processes, band-limited rhythmic shift vs broadband/arrhythmic shift, sustained oscillations vs transient bursts). I'm not saying there is nothing there, but it's amongst the least settled parts of the work on brain rhythms, especially in humans. And that's before we get into the potential connections to consciousness.
Echoing other commenters that most of your examples of settled "psychology" seem drawn from neurology, neuroscience and psychopharmacology. These disciplines are correctly viewed as part of the hard sciences, and even a lay person who isn't very well-versed in the distinctions between them understands that not just anyone can interpret a brain MRI or conduct a neurosurgical procedure (per the common joke about a job which isn't challenging "it's not exactly brain surgery, is it?"). Even psychiatry seems meaningfully "harder" than clinical psychology, and most people intuitively understand that it's more of an accolade to qualify as a psychiatrist than a clinical psychologist. The criticisms of psychology (which predate the replication crisis by decades) tend to take the form that it's a soft cargo-cult "science" which makes claims which range from trite to unfalsifiable and produces no useful predictions.
I do understand that there's meaningful overlap between psychology and the other disciplines, but I think it's a bit of a shell game (dare I say a motte-and-bailey argument?) to insist that psychology is in rude health and then tout a bunch of robust findings from neuroscience, psychopharmacology etc. Where are the robust findings from psychology as practised by academics with no background in neuroscience, pharmacology etc.?
A funny thing is that some of the most controversial fields of psychology, like IQ, are actually among the most settled portions of psychological science. Some questions -- such as how much is due to Nature, how much to Nurture, and how much to Noise -- remain unsettled, but the basic facts are beyond dispute after 120 years of studies replicating so much that they've gotten boring.
For example, that there are racial differences in terms of average IQ in the U.S. is completely settled science (why these differences exist is not, however). Yet, a large fraction of college graduates get outraged at anybody who mentions this fact, which they can't believe is anything other than pseudoscience of the most laughable kind.
You know damn well what he means and there are obviously parts of science which are more or less provisionary and whose probability of truth is higher or lower.
Not denying science can provide higher probabilities of truth revealing, which is not equivalent to claims of it having been settled. My point is that scientific discovery, replication, advancement is a never ending enterprise.
Many areas of science are "settled" in the sense that a prudent well-informed person would base a billion-dollar project on them, or stake their job or even their life on them being true. These are of course not beyond revision--perhaps tomorrow we'll discover that we were wrong about something we thought was totally nailed down. But they are solid enough to rely on in practice.
By contrast, we've also sometimes relied on very shaky and politicized scientific findings for large-scale government programs, changes to the laws, etc. That's the thing we want to avoid doing. Nobody wants to forbid you from trying to find some ways that general relativity isn't true, we just don't want to end up designing GPS based on indigenous ways of knowing about time and distance because a bunch of serious-looking scholars with PhDs said this was the right way to proceed.
Large parts of psychology are NOT settled. I mean, you can't really sit down and pull out good psychological research papers that you'd base x265 on, right? I'm pretty damn sure the people who wrote x265 didn't spend a year reading research articles. They spent that year doing their own research, and didn't put it in the psych research journals neither.
And you can count that as billion dollar psychology.
Yes, settled science is where the value is at -- the problem in psychology and many other highly empirical areas of science -- is that the incentives to weed through the many studies that suggest a new phenomenon and turn that into settled science isn't there.
If you replicate that work and it turns out to be valid you get very little of the credit because you weren't the first to publish and if you debunk it as a false positive you don't get much credit either.
It's not that there is anything wrong with the things that are being done it's just that no one is doing the hard work to actually turn new results into robust settled science. This encourages people to misinterpret things that aren't settled science yet as settled science because we haven't created a higher bar for them to look at.
Physics faced this problem to an extent and answered it with the 5 sigma rule -- it's not really a discovery until you hit 5 sigma. Psychology needs something similar and incentives for people to do the extra tests to turn promising early studies into settled science.
"The key distinction isn’t “recent research in social psychology” vs. “recent research in cognitive/bio/behavioral psychology”, as my essay last week unfortunately implied"
I'm not sure settled science vs novel science really explains what parts of psychology are reliable or not. Many of the findings within psychology which catastrophically failed to replicate are things which ex ante people would have said are "settled science", with scores of apparently confirming studies over decades (e.g. ego depletion which had hundreds of supporting studies).
At the same time I think you are underselling the extent to which this does map on to something like "social psychology" (and studies sharing the method of social psychology) vs biopsychology/neuropsychology studies not sharing the method of social psychology.
Here are some of the specific features of "social psychology" style studies that beget poor validity / reliability:
- Use of stimuli, often vignettes to try to manipulate general categories without attending to whether the stimuli are representative (e.g. through stimulus sampling) or unconfounded (https://pubmed.ncbi.nlm.nih.gov/40372824/), e.g. you try to test whether people respond better to "scientific" or "prosocial" arguments by presenting people one of two paragraphs arguing for something. But there's a whole universe of possible stimuli you could have used to represent your intended constructs, and the particular stimuli you used could produce different effects for a wide variety of different reasons.
- Use of measures with poor validity: you test how people respond to your vignettes by measuring their "endorsement of vegetable consumption" score (constructed by averaging likert items you made up like "I think of myself as a person who likes to eat salads") based on tests of reliability (like cronbach's alpha) or model fit measures which do little on their own to confirm whether the measures are measuring what you claim they measure.
A number of these points are made (I think persuasively) in Schimmack's Validation Crisis (https://open.lnu.se/index.php/metapsychology/article/view/1645) which explains why much psychology suffers from validity issues, not just a replication crisis solvable by pre-registration and better sample sizes. Many post-replication crisis studies have dutifully pre-registered well-powered studies which replicate, with designs that mean they are completely invalid.
I think it's clear that and clear why many of these features don't apply to the kinds of studies you highlight as being most robust (e.g. GABA receptors behave such and such way).
The The Beauty Of Settled Science ... PUBLISHED 16 HOURS AGO 73 LIKES 2 RESTACKS 7 COMMENTS framing is the part I keep coming back to. Most takes on this skip the underlying mechanism and try to win on vibes.
W. V. O. Quine's philosophy of science was that it's all subject to revision, but we typically rely on a stable "core" while modifying the periphery. He used a metaphor of modifying a boat while you sail on it (I don't recall where I came across that one, though I know Robin Hanson cited his metaphor of a "fabric" of many threads https://www.overcomingbias.com/p/ignorance-about-intuitionshtml ).
Almost every single one of the examples you cite either comes from cognitive science or behavioral psychology. It's a bit of a stretch to defend the whole umbrella of "psychology" by pointing to its uniquely rigorous and politically neutral subfields.
Idk: if they want to act like a propaganda mouthpiece they can be treated like one.
When it’s technical enough that the writer probably isn’t thinking with their woke brain, sure you can just treat it like honesty.
But anything getting closer to actual people or society and they’re gonna go right back to their old habits.
it is free to just selectively ignore them. And there is little difference between me having to police and scrutinize them constantly vs me just believing my old priors anyways
> the main way that Psychology affected their lives was through bad clickbait studies used to buttress dumb policies; some p = 0.049 study says that exposure to microaggressions makes minorities’ brains shrivel into raisins, and now their company has to implement punitive DEI or be sued for hazardous work conditions.
Is that an actual thing that happened? I'm sure people have occasionally handwaved towards some study or another for justification, but I think the causal claim is very dubious.
I think DEI (commercial DEI, at least) has been largely sold on the basis that if you don't make people from minority groups feel uncomfortable, you'll be able to hire from a larger pool of competent candidates. Which is fundamentally true, and much like with psychology, the problem was in not drawing boundaries between the true stuff and various questionable add-ons; and in many cases in mixing up actual DEI (self-interested or ethically motivated HR concerns) with PR (the internet is angry at one of our employees at some reason, and the minimum-effort solution is firing them).
So this is the key question here, IMO: How should outsiders distinguish between settled science and stuff that's still uncertain and ideology dressed as science and high-profile scientists speaking based on their politics/ideology rather than their science and institutions taking political positions and resting on their credibility as scientists and all that stuff?
Like, apparently serious people with PhDs and impressive-looking publication records tell me about both stereotype threat and regions of the brain being associated with particular mental functions. If I am not in a position to dig into the field myself and understand methods and evidence, how should I tell these guys apart?
For that matter, apparently serious people with PhDs and impressive-looking publication records tell me that IQ is racist pseudoscience, and also that IQ is one of the more solid areas of psychology and has a bunch of replicated results showing that it predicts work and school performance. I know enough to distinguish, but only because I have spent some time trying to understand the issue--otherwise, how should an outsider proceed?
For the public, few of whom are ever going to read a scientific paper or evaluate the statistical methods in some research study, what guidance can they have to distinguish real settled science from "settled science" claimed to win a political argument despite being very shaky or even just plain wrong?
“My framing emphasized that social priming studies had failed to replicate” actually replication crisis seems to be in fact that metanalysis protocol is non complete. It checks the correlation effects, while in many fields, especially in psychology effect could be modulation, moderation.
In example, if we have modulation effects, then trying to measure the meta consistent correlation effect would lead to replication crisis every time.
It is not the study bad, but the point of view is not exhaustive
"Instead, you can say the thing you mean - that woke partisans shouldn’t try to occupy and exploit the otherwise-noble science of psychology for their own political gain - and wage the same campaign with a halo rather than a comic-book-villain goatee."
This assumes that "just surgically cut out the bad tissue and leave everything else" is workable. Claim: it is not workable. Evidence: an entire legion of scientists, scholars, "lovers of truth", tried this in the late 2010s/early 2020s, and experienced the epistemic equivalent of ignominious slaughter with no remembrance.
Where do you think the beta-amyloid research in connection to Alzheimer's falls in the spectrum of settled science vs novel research?
For those unaware, my understanding is as follows:
(1) in the 80s brain 'plaques' associated with Alzheimer's were found to be filled with amino-acids (beta-amyloid).
(2) throughout the 90s and early 2000s there was a lot of research into this connection, but it all basically led to basically nothing
(3) In 2006 there was a paper with a huge breakthrough that showed a particular beta-amyloid would cause memory issues when injected into mice. https://pubmed.ncbi.nlm.nih.gov/16541076/
(4) a huge effort follows for the next 15+ years where tons of federal research money and an enormous amount of pharmaceutical money is poured into beta-amyloid research (while not in this field, it has been described to me by peers as an era of "if you don't mention beta-amyloid, you won't get your Alzheimer's research funding".
(5) Since 2006, and partly because people struggled replicating the original results, other researchers began to raise allegations of image/data manipulation against newer work of the 2006 papers lead author. This eventually leads back to the 2006 paper where people find...it too contained fake data and image/data manipulation. The 2006 paper is eventually retracted
The original (now retracted) article from 2006 has a bunch of p<0.01, 0.001, etc. results. But maybe the meta-beta-amyloid hypothesis should have been thought of as nearing or going over p~0.05 once other studies failed to replicate? I.e. this always should be been considered novel and not settled.
I still have lots of questions about this era of work (i'm not an expert on any of these, just my curiosity), like:
(1) why did funding agencies so heavily prioritize amyloid-beta after seemingly one paper?
(2) why did drug companies rush to get beta-amyloid treatments out and spend so much money on this? Obviously they are profit driven by trying to be the first to do it and capture the market (this is bad imo), but did they not have their own labs trying to replicate the research?
(3) how did Aducanumab eventually get FDA approval? (this seems like a whole other controversy rabbit hole)
Implicit in Scott's description is an assumption of widespread good faith commitment to research and truth. Some bad apples, but they eventually get discovered and discredited. But this assumption seems unwarranted in a field that is (1) highly politically activist, (2) highly ideological, (3) almost entirely monoculture. The replicability crisis has hit many fields, but the most politicized and monocultural (like social psych) seem much more likely to lack the impetus to correct themselves. Once a field picks a team, hard, public faith in internal correction, updating, and paradigm shifts plummets, and justifiably so. See also most of academia, increasingly. Social psychology isn't singled out just for being particularly error-prone, it's singled out for being particularly activist, tribal, and incurious. Which also leads to errors, but the errors aren't the only issue.
I don't know specifically. Anthropology certainly has that reputation. History has gained more of that reputation as well, so it wouldn't be surprising if archaeology did too.
I enjoyed that Yudkowsky quote, and would now like a subscription to a magazine that only reports on scientific advances that are at least ten years old and haven't yet been disproved.
Most readers probably don't realise just how poor education research is. Your 90% bad, 10% good estimate is probably generous.
There is a growing movement trying to do better. The Education Endowment Foundation (EEF) in the UK is one example, commissioning more rigorous trials and producing evidence summaries for teachers.
The one paper that is most worth reading is Kirschner, Sweller and Clark's Why Minimal Guidance During Instruction Does Not Work. It is a devastating critique of a whole paradigm in education, built on a foundation of cognitive psychology rather than educational theory.
The broader Cognitive Load Theory literature is interesting because it sits at the intersection of cognitive science and education. I would be interested in where someone with a psychiatry background thinks it is strongest, weakest, or most likely to have been misunderstood.
'studies fail to replicate' is no crisis, it is merely science as it is normally done
'practitioners fail to update on studies failing to replicate' is a problem, and like other problems can rise to the level of crisis if it continues long enough
'media sensationalizes new study' is unfortunately just media-as-commerce, and probably inescapable in free societies
I think you're overestimating how easy it is to know which of “Microaggressions makes minorities’ brains shrivel up like raisins” and “the hippocampal theta rhythm seems to be associated with both memory and pathfinding, suggesting an unexpected yet surprisingly compelling relationship between the two” is settled vs p = 0.049.
You frame it as detecting "woke" vs "not-woke", which I admit might be an easier distinction for a layperson to make (one of them has the keyword "minorities"), but it's not necessarily clear to a layperson that there's any correlation between wokeness and truthfulness, or wokeness and settledness.
So sure, perhaps “Microaggressions makes minorities’ brains shrivel up like raisins” is "woke", but is it "true"? Is it "settled"? A lay person would have no idea.
Nice clarification, Scott. Though you missed the most obvious «train of thought»:
The dumb woke psychology was wrong and bad, and woke people (or normies who blindly accept media interpretation of woke psychology because it’s «science») have terrible consequences for society, and most people don’t have either the expertise or the energy to rebut every woke study trumpeted by media/academia, therefore the rational response is a negative view of the field in general, until it proves itself capable of avoiding these bad societal externalities.
In item #7, you wrote "The exact substrate of memory is unclear but seems to involve new synaptic connections between neurons in at least some way", but we know much more than that. Long term memories in the hippocampus are well-modeled as a self-recurrent connectivity matrix in the CA3 region, directly analogous to a Hopfield network, an artificial neural network which implements long-term content-addressed memory.
The APA advised the Supreme Court, based on the strongest scientific evidence, that adolescents are as mature as adults and can decide perfectly well to have an abortion; and, also, that adolescents are not as mature as adults, and therefore shouldn't be given the same criminal penalties as adults. Hodgson v Minnesota and Roper v Simmons. This isn't the worst example of corrupt APA policy influence, just the most legible.
The stakes of this debate are very high! The Supreme Court writes citations to APA amicus briefs into US law! And (although regarding the adolescent maturity issue, a few Justices called out the hypocrisy) they don't always seem to be able to tell the difference, despite how easy Scott thinks it is, between corrupt and not-corrupt psychology. Or they pretend so. There are some Justices who take the raisin brain hypothesis very seriously.
The American Psychological Association represents psychology as a whole. That's what they're for, that's their purpose. And they're corrupt. It's not that we can't distinguish between good and bad psychology; and it's not that it's useful to discredit psychology as a whole just to fight the bad psychologists. It's that psychology, as a whole, to the extent that such a thing exists, acts corruptly. The psychologists got together and formed an organization to represent themselves; that organization acts in a way that discredits them.
Your contrition is barely perceptible. I definitely chalk the original post to the 'debunks of decay narratives are always more wrong than the original claims' flavour.
> Memory divides somewhat fuzzily into bins of immediate-sensory, short-term, and long-term, with separate retrieval and storage functions, and separate systems for declarative vs. procedural learning. These can be harmed selectively, usually by hippocampal injury, so that some patients can remember their life stories but not form new long-term memories, or learn new skills like bike-riding but not new facts. The exact substrate of memory is unclear but seems to involve new synaptic connections between neurons in at least some way. <
I drowned at 13 and lost all my childhood memories. Except the emotional content. I can recognize people by how I feel when I look at them. I can re-watch a movie and know how I felt about it the first time, whether I liked it or not.
You say there's suggestive evidence marijuana makes schizophrenia risk worse, but schizophrenia risk seems to be the same between pairs of identical twins where only one smokes weed. See doi 10.1037/abn0000867
I think that population-level correlations between marijuana use and schizophrenia are just confounding, and I think the procedures for controlling for confounders in a regression are rather sloppy in general, not just in studies of the mental health risks of marijuana use. But it seems your settled science list doesn't have other conclusions based on population-level correlations and this one is just there as "suggestive evidence" so I guess this doesn't undermine your conclusion.
My count for this "settled science" is 4 out of 10.
1) Reward functions and the nucleus accumbens are linked, but tieing this to predictive coding is still speculative. Low prediction error in itself is certainly not the reward, otherwise I wouldn't be very happy to see a Ferrari appear in my garage. More subtle connections need to be made (and to be fair are being made, but I would not call it settled). Also it's unsure if metaphors like "algorithm" really apply to the brain—there's no centralized, time-locked set of rules that govern the behavior and brain function may emerge from decentralized behavior of individual neurons. Brain as computer metaphors driving popularity of theories like predictive coding may be severely limited.
2) Psychedelic drugs do often act on those receptors, but again the prior/new evidence stuff is still speculative. There's no clear way to say that tripping people are more bottom up or top down. Jumbled, sure. But then it's unclear if predictive coding is a main role for these receptors or if alterations in prediction are concomitant /emergent with other brain changes– changes in regional connectivity or in emotional modulation for instance.
3) No to the Broca / Wernicke seperation. I've talked to a person who treats aphasic patients (speech difficulties) who says she purposefuly does not look at brain scans before treating patients because the conventional knowledge is often wrong. And if this seems anecdotal, there are studies of direct brain probes during awake brain surgery that indicate the same thing.
4) oof yes wow you put a number on nature / nurture, you figured it out you tricky devil. The field of study that relies on hunting out twins seperated at birth, and if you can't find these elusive creatures then settling for the next best thing (regular twins?) and sprinkling in some shaky statistics to untangle all the confounding variables. (I read your writeup of this and didn't see you address the possibility of feedback loops where small genetic difference may drastically change environment, leading us to overestimate the genetic effect)
5) Brain waves is mostly right to my understanding. Ofc it gets complicated the more you try to map out specific functions to rhythms.
8) ? The SSRI meta-studies I've looked at often have it working no better than placebo. Modern psychiatry is plagued by division of probably heterogenous populations into fixed categories; driven by methodological errors like lack of proper observation and construct validity errors—for instance assuming that those with similar behaviors and similar answers on questionnaires share fundamental similarities in mental function, not to mention lived experience. (all of this exacerbated by looping effects, where diagnosis itself can shape behavior).
9) bipolar and schizo might be the same disease? Not sure where you're getting this. We don't have clear neural biomarkers for either disease. Maybe some genetic links? From my study of the inner experience of these diseases, they are not the same. For schizophrenia we find mostly low-level perceptual alterations, mostly visual (e.g. floating glasses), sometimes auditory (e.g. voices of the gods). For bipolar, it's unclear, perhaps a heterogenous group— sometimes multiple experience, or lack of clear emotion, or lack of clear mental content altogether. But usually not perceptual alterations/ hallucinations.
10. again with the algorithms. But yes, classical and operant conditioning, sure.
I'd say there is some settled pschychology, but mostly on the lower level: how sensory input maps from retina to occipital visual areas, how genes encode various neuroreceptors, how your ears transfer wavelengths to neural firing patterns, how neural polarity leads to firing, stuff like that.
The solid subfields are ones led by direct inductive observation. The shaky ones are those overinfluenced by theory and premature hypothesis testing. Especially where we don't understand the divisions we've made between various population groups (much of psychiatry), aren't properly controlling for multiple testing (most fMRI studies), assume that tasks in a lab environment mimic ecology and can induce certain predefined states, can't separate confounding variables (heritability studies), etc. And that's before we get to publication bias and other issues in an academic system that incentivizes many other things besides truth.
On a broader epistemic note, much of psychology now encourages large sample sizes to find averages. Its unclear if this is the best route for many problems in psychology, where individual variety may be more important and than such generalizations. How much would you learn from averaging every person's fingerprint? Why is this the default we apply to studying the mind?
Re. "Psychedelic drugs exert their effects through 5-HT2A receptors, pointing to a role for these receptors in some sort of construct like the balance between priors and new evidence."
That suggests that psychedelics disrupt a brain mechanism for perceiving the noumena behind phenomena. It seems reasonable to suppose that many different kinds of optical or neural noise create systematic, patterned aberrations, like rainbows, or chromatic aberration in microscopes. Psychedelics may, like Impressionist paintings, let us see something closer to the raw phenomena. They aren't providing deeper insight into reality; they're providing shallower views of it.
If so, this provides evidence for the argument that we don't need to worry about differences between noumena and phenomena because evolution guarantees that our brains work hard to eliminate any discrepancies of that kind which could possibly be relevant to us.
Aren’t almost all the “good settled science findings” neuroscience, rather than psychology? The only one I’d classify as psychology is the one about mental illness treatment and, arguably, the ones about heritability and schizophrenia
It seems to me that whenever this topic comes up scott mostly mentions examples on the "psychology is great" side that are neuroscience, pharmacology, and basically the more STEM leaning subfields of psychology.
All of these for example:
"Psychology is centrally ideas like brain waves, GABA receptors, reinforcement learning, twin studies, and connectionism"
are things I studied in a neurosceince bachelors, not a psych one. People belittling psychology mostly talk about the subfields that are in the difference set of "things you'd study in a psych degree" minus "thins you'd study on a neuro/cognitive science" degree... such as psychotherapy concepts, personality types, organizational psychology, social psychology, educational psychology and so on.
I'm pretty sure the precentage of psych critics that think there's no such thing as a broca's area is very, very low. But you can argue that that's neurosceince or some other title that's not what people mean then they say "psychology". On the other hand, the precentage of these people who think that a randomly chosen concept in personality studies is not well founded and kinda bogus is probably pretty high.
chatgpt suggested:
"One useful way to simplify this enormous list is to think of psychology as having roughly six big trunks:
Brain → biological psychology / neuropsychology
Mind → cognitive psychology
Individual → personality psychology
Other people → social psychology
Change over time → developmental psychology
Mental health → clinical/counseling psychology
Then fields like forensic, educational, organizational, health, sport, etc. are largely applications and combinations of those foundational areas."
I think its not that hard to claim that psychology is very scientific if you mainly stick to the Brain and Mind branches, the sceince there is basically chemistry and biology. People are mostly ok with those. The crux of the debate would be about the more conceptual later beanches.
Yeah. To the degree that Scott is a psychiatrist so most of his favorite examples will inevitably come from there this makes sense, but if it's genuinely hard to find this level of solid science results in fields closer to what most laypeople think of as psychology it really might be close to the education system bar
Yup, I'm reminded of his old article that he chose to become psychiatrist instead of psychologist because it makes him feel that he's really doing something. Feels like everything he mentioned here is in the psychiatrist side instead of psychologist side. Which is okay, my priors have been like that all along.
I don't think it is hard to find this level of solid science results in fields closer to what most laypeople think of as psychology. Think about claims like how intermittent reinforcement tends to produce stronger learning than 100% reinforcement. Or that you aren't sensitive to color in your peripheral vision but it is part of your experience there anyway. Or that people are slower at Stroop tests when counting backwards (or whatever).
I also think it's a bit misleading for Scott to use the word "woke" so much in the original post. It's true that woke versions of these claims were more likely to be propagated through the media than non-woke ones, but the priming stuff was mostly independent of wokeness, and I suspect there's a non-trivial amount of woke stuff that is actually just fine.
There are some. Pavlovian conditioning is such an example.
However, it is a very old finding. Anything newer on that scale, that is completely black-boxing the behavior, that is newer?
Also, Most new Psychology being garbage could have the same output:
Well, 100 years ago we had serious psychologists, but they are gone now.
Let's defund the psychology departments everywhere.
Scott has a different post that does in fact list a bunch of social-psychology-ish examples of settled science: https://www.astralcodexten.com/p/heres-why-automaticity-is-real-actually
Thanks, I forgot about that post
a lot of cognitive psychology is very settled but not mentioned here! spreading activation theory and other theories of activation or cognitive load are similarly settled
Conversely, there’s probably a lot of genuinely interesting controversy in cognitive psychology. (I’m just guessing.) E.g. how does memory work, how do we map 3d spaces when we aren’t looking at them, etc.
There are plenty of examples from the more "mind-content" side of things:
1. Big 5 personality traits
2. EMDR for treating PTSD
3. Exposure therapy for treating anxiety and OCD
4. Regular exercise has a positive effect on mental well-being
5. News media coverage of suicide increases suicide rates
Here's a column I wrote on a study of how the Big 5 personality traits differ between people who say that Kubrick's "2001" is one of their favorite movies and people who say Julia Roberts' "Pretty Woman" is one of their favorites.
I'd say that Big 5 looks pretty sensible and useful for applications like predicting your favorite movies:
https://www.takimag.com/article/are-we-what-we-watch/
I kind of went down a rabbit hole after reading your article, and the linked study, but had Claude code up a nice visualizer for the data. If any wants to check it out, here's a link: https://claude.ai/public/artifacts/f97936aa-1945-41d0-9a30-9b74a1ea5dbe It lets you pick a movie, and see what other movies are liked by people with similar big 5 traits, or input your own big 5 traits, and see what movies it predicts you'd like.
Thanks.
Big 5 is impossible to do cross cultural/groups. That is an issue
It could be that 50 years from now, psych majors will be taught a Big 3 or Big 7 model of personality. There's no absolute proof that the Big 5 model is the best, it just seems to work fairly okayish so far.
But who knows what the future might bring?
The Dark Triad/Tetrad stands as a nice counter to the whole Big 5 idea (yes, you can find some correlation to Big 5, but these are mostly orthogonal -- you don't say someone who is Highly Open is likely to be a Sadist).
>"Psychology is a robust science!"
>Neuroscience example
>Psychopharmacology example
>Neuroanatomy example
>"No real effect from factors typically considered 'psychological'"
>Electrophysiology example
>Psychopharmacology example again
...
Maybe the "psych" in neuropsychopharmacology is enough that we should lump it in as a subcategory of psychology—certainly all the above fields are intellectual descendants of psychology.
But as each of these subfields has grown rigorous and interesting in its own right, they've left home—taking that rigor with them. So most of what's left in "true psychology" is, as jumpingjacksplash says, the mind-content stuff.
How are they intellectual descendants of psychology? They are descendants of anatomy and physiology and pharmacology. (Yes, I understand there isn’t a clear practical distinction in many cases).
You have four grandparents, don't you?
Actually they aren’t even ”descendants”, eg neuropharmacology is just pharmacology and neuroanatomy is just anatomy.
In that case it's all just natural philosophy :-)
And computer science is all mathematics.
It is a bit as if astronomy and astrology never separated.
"Hey, I know that horoscopes are entirely useless and yet used as a reason to push pro-Capricorn quotas on the entire economy. But the same science is giving us spectrums of stars, which is cool!"
And then I look at general relativity, and I say "what's the point in all of this work, if none of the work gets referenced in Real World Money Making Enterprises?"
(x265, for "visual interpolation and compression" as claimed by OP as part of psychology.).
i think youre probably overestimating the extent to which most laypeople conceive of a difference between neurology, psychology, and psychiatry
The median layperson maybe, but I bet the median ACX reader knows the difference
I don't think it's fair to say priming failed to replicate
“Fairness” itself is worth bringing up here. Why do so many non-psychologists, non-scientists, heck, non-STEM-degreed people have so much interest in the Replication Crisis that I can put it in caps and be assured most people here will know what I’m talking about?
Is any of this really about science, or one level down is it really about culture war?
Well, I suppose if the culture war is fueled by bad or good science, it is relevant.
Not if the culture war itself is irrelevant. Which I know it isn’t to all people, but my experience at least has been that the more you get out and “touch grass” as the kids say, the less relevant you discover it to be.
Coincidentally, I did a bit of lawn maintenance earlier today. Grass well and truly touched.
But I was once driven out of the community that was my online-and-occasional-meetups home before this one, by culture warriors. And I was here when the culture war fragmented this community in a fortunately-unsuccessful attempt to destroy it. So, yeah, the culture war is quite relevant to me.
I think that on this Substack, it's a lot more about culture war. But in academic contexts, it's really about science. I don't think many people in the general public, other than maybe those with degrees related to psychology in the past decade or so, would recognize the phrase "the Replication Crisis".
I’d be curious to know, if it is about science to some people, do they think the term is apt? Is there really a crisis, or does cutting-edge research in any field often yield low replication rates, and this one just happened to earn a catchy label that was widely circulated because it played into existing culture war tropes?
I've heard slightly different views on this - social psychology has been the field most commonly said to be affected, but I've also seen claims that nutrition and cancer biology are too, and sometimes some aspects of economics. I think it matters that all of these are fields that work with humans, though it's interesting that some of them are experimental and it's not all observational.
I read a blog post making the rounds about a decade ago, that I unfortunately can't find now, that put forth the theory that the fields vulnerable to the replication crisis are the ones that both (1) rely on controlled experiments, and (2) don't have a strong theory known to govern the phenomenon they study. The explanation for (1) was something like, if your field is entirely observational studies then that forces it to develop better metis about confounders, or something like that.
I was actually thinking exactly the same thing (about having read a blog post about a decade ago making something like that point), but you added a few more details I hadn't recalled, so I copy-pasted your comment into ChatGPT and it found it:
https://simplystatistics.org/posts/2016-08-24-replication-crisis/
Paid research is often nothing more than advertising. End of story. When you need to have research to make $$$, then it becomes incumbent on the profiteers to find the most efficient, best ways to make $$$ -- if that's finding folks willing to lie in order to publish... well...
It's all very controversial, of course. Some believe that the whole thing is overhyped, there's nothing to see, it's business as usual. Others claim that the standards are too low, it's feasible to improve them, and it's a coordination problem/entrenched interests that block improvements.
My answer would be essentially yes, the term replication crisis was embraced by many scientists and was accurate, but I think that "is there really a crisis" simply uses the wrong tense.
In the early 2010s, when the news came up, people in science were in a huge genuine shock, and the term replication crisis was spot on. This was a crisis, people all over science re-thought hard about what they can change about it. I think that the general sentiment in science was that this was a good term.
But a crisis, by the very definition is something acute, not something that lasts for decades. The replication crisis WAS a crisis of science around 2010-2015.
This does not mean that everything has been solved. Certainly a lot of things have improved. A lot of journals have changed their policies in some ways to address the problems better. Perhaps the most lasting effect is a general awareness in science that even a "settled field" can break, and that it is more important to test it. On the other hand, crappy science continues to exist, and many problems have not been overcome. But the term crisis for the term of science nowadays is inadequate. Scientists have changed what they managed to change, and have otherwise moved on. For the state now, more adequate terms would be "good" or "bad", but not "crisis".
To clarify then, is it fair to say the replication crisis referred to stuff like “oops, we realized that a lot of our findings were based on observations of grad students who are W.E.I.R.D. and not representative of the general population” (which they’ve since better accounted for and fixed) but in the popular imagination it was mainly about “look at these silly ideologically-captured journals publishing papers about icebergs being literal instantiations of oppressive patriarchy”?
https://jacobfer.substack.com/p/does-priming-work?utm_source=share&utm_medium=android&r=52htm0
Here
https://jacobfer.substack.com/p/does-priming-work?utm_source=share&utm_medium=android&r=52htm0
Hm, it was definitely not about the latter, but I don't think the former captures it either.
The problem was not that the findings were only for W.E.I.R.D. grad students. The problem was that when a second person tried to do exactly the same experiment for exactly the same type of W.E.I.R.D. grad students, they would not get the same results.
I agree that there has been a tendency in the last two decades or so to go beyond W.E.I.R.D. grad students, or to test drugs on women, too. Which is a good thing. But I wouldn't link that to the replication crisis. The replication crisis has not much to do with culture wars or woke topics, except that it was picked up later in culture war battles. (Perhaps because social priming was a popular concept that failed to replicate and that was also used in woke arguments. But there were many other examples, too.)
The difference is that if someone told pre-crisis scientists that the outcome would be different for women and for men, or different for people of different age, they would probably have said "Huh, interesting. I never thought about this aspect, but sure, could be." But the actual replication attempts really tried to replicate the original experiment as exactly as possible, and they still failed. And the reaction was more "What? No way! This can't be!". (Most) scientists genuinely didn't expect that to happen, and it was a big shock that raised fundamental questions of how we could even continue to do science in those fields.
It you want to get a feeling of this shock, I recommend an old article by our host: https://slatestarcodex.com/2019/05/07/5-httlpr-a-pointed-review/ . Scott was clearly shocked by this, and this is very similar to how the replication crisis hit many fields a few years earlier.
The big story in academic publishing I'm reading about on Twitter nowadays is Jason Arday's extensive plagiarism, fraud & obvious incompetence being covered up & rewarded. Of course, he's not a psychologist though. As Greg Cochran quipped, some fields didn't have a replication crisis because, unlike social psychologists, they don't care about being wrong or feel embarrassed when others notice https://westhunt.wordpress.com/2017/02/17/the-replication-crisis-in-sociology/
EDIT: Andrew Gelman's thoughts on why social psychology was the sweet spot for the replication crisis are also worth reading https://statmodeling.stat.columbia.edu/2018/05/07/replication-crisis-centered-social-psychology/
I think you guys are underestimating the general public, or at least the more literate segments of it. There have been mentions of the Replication Crisis and similar academic scandals in Vox, The Atlantic, and NYT over the last several years.
I suspect that, on this Substack, culture war and science are very tightly integrated.
It's extremely about culture war, in my opinion.
The discussion about replicability was of interest and important to a relatively small population of active researcher and journal editors. Reforms to the process were worth discussing within that small community.
But the spread into a wider cultural phenomenon is, to my mind, entirely about giving sides in the culture war yet another tool to justify dismissing any scientific findings or expert opinions that don't align with their own motivated beliefs.
Your description matches my suspicions, but I just don't know any of the specific details of the history to give it confirmation in my mind. I do think the vast majority of people are primarily motivated in their opinion-having by whatever they decide is their tribal identity, even if it seems (even to them!) like they got there by reasoning.
I directionally agree, but I'm also chronically charitable in some regards and fell the need to complicate that story.
I think that we overestimate the degree to which people accept whatever beliefs their tribal identity is 'supposed' to accept, and underestimate the degree to which people end up with a tribal identity based on the best epistemics available to them from their individual position and knowledge, and from their get stuck in filter bubbles and trusting sources in a way that traps their priors despite proper Bayesian reasoning.
Like, it's definitely a problem that most people end up believing whatever their tribal identity says they should believe, but I think in some sense it's trivializing the problem to conclude that people ignore logic and reason and just accept beliefs for social reasons. I think the much more daunting truth is that it's just really really hard to end up believing only true things through logic and reason, and the shape of our culture and politics and economy is designed to drive certain people away from certain truths even if they are trying to be rational and skeptical.
You can get a proper Bayesian to believe wrong things if you control the evidence they see well enough, and a lot of what we see is very very controlled by various motivated sources, starting in infancy with our parents and proceeding from there.
I appreciate what you’re saying, and there are indeed people who are resistant to tribal identity and its various tugs, but I think broadly I am correct and that the sort of charity you describe is unwarranted.
The problem isn’t that it’s hard to believe only true things (though I agree that the second clause is true: it is hard to believe only true things); the *problem* is that people feel compelled to have strong beliefs about lots of things they know little or nothing about, and to ignore or not pursue knowledge of counter-beliefs that might temper them, and they do this—they feel compelled in this way—because they have internalized a tribal identity.
It is difficult, I theorize, not to internalize a tribal identity, because of how humans are wired by evolution. Even I, a freak, a natural contrarian, a Groucho Marxist who “wouldn’t join any club that would have me as a member”, still have to stop and be intentional about my epistemology from time to time to make sure I’m not starting to reason from after having picked a side.
Ground Penetrating Radar is a fun one. You'd have to actually have studied the physics to know how easy it was to mistake tree roots for "buried children"....
https://nypost.com/2023/08/31/still-no-evidence-of-mass-graves-of-indigenous-children-in-canada/
The Americans were laughing about this the entire time, of course, because they knew the failure modes of the GPR, and this was an entirely obvious "RTFM" -- or listen to the Americans telling you how to use the system.
... but, sitting there on the outside, not being part of the GPR physics folks, I wouldn't have known boo. And the scare of "mass graves everywhere" does sound implausibleish...
Wrong culture war, I think. The STEM vs soft science beef is its own kind of culture war, and its the stemlords who turned the replication crisis into a wider cultural phenomenon. Of course people did then use it to reject findings that they didnt like *from specific disciplines*, but thats basically in line with the intent.
“The STEM vs soft science beef is its own kind of culture war”
Among whom? I would say it’s a theater in “The” culture war. In real life i know actual scientists/technologists/engineers/mathematicians as well as psychologists/sociologists/anthropologists/archeologists/etc., and they don’t seem to have this beef with each other’s fields at all; it’s only in the lay peanut gallery where you find this supposed confrontation. I might even say the STEM vs. soft science beef is primarily a plaything of the Very Online. (Everyone else is busy working, they ain’t got time for dat noise.)
In fact even there, the confrontation mostly seems to be one-sided, people motivated by a certain tribal identity who scowl at non-STEM unrequitedly.
>I would say it’s a theater in “The” culture war
There are points of contact, but I dont think it correlates all that well with red/blue affiliation
>In real life i know actual...
Most people arent active in The culture war either, and its also very much an online-skewed activity.
Most don't have outright beefs, but many are finely attuned to the perceptions of their own fields and the fields of others and develop various psychological complexes around them of varying severity.
For instance, quite a few economists have "math envy" and have an instinctive need to feel mathematically sophisticated, at once feeling superior to the "softer" social sciences and insecure around hardcore physicists or mathematicians. This is a whole thing (I'm pretty sure there was a famous essay written about it), and of course the better economists understand the point is just to do good economics.
Every field has an analogous "story" of how it relates socially/psychologically to other fields. Mathematicians and physicists mostly have a cordial relationship but mathematicians feel physicists write down equations without knowing what they mean while physicists feel mathematicians get into the weeds about really pedantic things that don't matter or reflect reality. Mathematicians and humanities professors tend to respect each other at a distance as "pure fields", but not know much of what each one does. These are all stereotypes of course, but I think you get the idea. These days because of the larger culture war against the academy as a whole, there is a strong force to unite all the disciplines and faculty often identify strongly with each other as being simply an academic or scholar. But there certainly are complicated (and not all positive) feelings between the disciplines.
I've heard economics nobel prize winners say, "But, it was just economics!" to mean that they didn't actually have to do anything "real world" like an experiment proving that reality was XYZ.
Well, this blog has always been partly about psychology and psychiatry, and science in general, so it's not that weird that people here would recognize the term. It's also something I learned about just by reading fairly mainstream publications like The Atlantic or something, years and years ago. It's not really esoteric knowledge. Something like, say, the Sokal Hoax is more niche and political, and that's something I've heard about primarily through a culture war context, but the Replication Crisis is a pretty normal thing for a semi-educated adult to have some familiarity with.
I agree.
People built a lot of sandcastles on top of the baseline priming results, and file-drawer problem meant that a lot of obscure/hyper-specific things which probably weren't real managed to get a few articles deep before petering out, but the central phenomenon is pretty solid.
A bunch of consultants and companies latched onto priming ideas to try to make products and sell services and they promulgated a lot of junk ideas, but that's a problem with capitalism, not psychology research.
As I wrote in 2013:
Social Psychology & Priming: Art Wears Off
Steve Sailer • February 5, 2013 • 2,700 Words • 80 Comments
One of the most popular social psychology studies of the Malcolm Gladwell Era has been Yale professor John Bargh’s paper on how you can “prime” students to walk more slowly by first having them do word puzzles that contain a hidden theme of old age by the inclusion of words like “wrinkle” and “bingo.” The primed subjects then took one second longer on average to walk down the hall than the unprimed control group. Isn’t that amazing!
This finding has electrified the Airport Book industry for years: Science proves you can manipulate people into doing what you want them to! Why you’d want college students to walk slower is unexplained, but that’s not the point. The point is that Science proves that people are manipulable.
Now, a large fraction of the buyers of Airport Books like Blink are marketing and advertising professionals, who are paid handsomely to manipulate people, and to manipulate them into not just walking slower, but into shelling out real money to buy the clients’ products.
Moreover, everybody notices that entertainment can prime you in various ways. For instance, well-made movies prime how I walk down the street afterwards. For two nights after seeing the Coen Brothers’ No Country for Old Men, I walked the quiet streets swiveling my head, half-certain that an unstoppable killing machine was tailing me. When I came out of Christopher Nolan’s amnesia thriller Memento, I was convinced I’d never remember where I parked my car. (As it turned out, I quickly found my car. Why? Because I needed to. But it was fun for thirty seconds to act like, and maybe even believe, that the movie had primed me into amnesia.)
Now, you could say, “That’s art, not marketing,” but the distinction isn’t that obvious to talented directors. Not surprisingly, directors between feature projects often tide themselves over directing commercials. For example, Ridley Scott made Blade Runner in 1982 and then the landmark 1984 ad introducing the Apple Mac at the 1984 Super Bowl.
So, in an industry in which it’s possible, if you have a big enough budget, to hire Sir Ridley to direct your next TV commercial, why the fascination with Bargh’s dopey little experiment?
One reason is that there’s a lot of uncertainty in the marketing and advertising game. Nineteenth Century department store mogul John Wanamaker famously said that half his advertising budget was wasted, he just didn’t know which half.
Worse, things change. A TV commercial that excited viewers a few years ago often strikes them as dull and unfashionable today. Today, Scott’s 1984 ad might remind people subliminally, from picking up on certain stylistic commonalities, of how dopey Scott’s Prometheus was last summer, or how lame the Wachowski Siblings 1984-imitation V for Vendetta was, and Apple doesn’t need their computers associated with that stuff.
Naturally, social psychologists want to get in on a little of the big money action of marketing. Gladwell makes a bundle speaking to sales conventions, and maybe they can get some gigs themselves. And even if their motivations are wholly academic, it’s nice to have your brother-in-law, the one who makes so much more money than you do doing something boring in the corporate world, excitedly forward you an article he read that mentions your work.
(“Priming” theory is also the basis for the beloved concept of “stereotype threat,” which seems to offer a simple way to close those pesky Gaps that beset society: just get everybody to stop noticing stereotypes, and the Gaps will go away!)
But why do the marketers love hearing about these weak tea little academic experiments, even though they do much more powerful priming on the job? I suspect one reason is because these studies are classified as Science, and Science is permanent. As some egghead in Europe pointed out, Science is Replicable. Once the principles of Scientific Manipulation are uncovered, then they can just do their marketing jobs on autopilot. No more need to worry about trends and fads.
But, how replicable are these priming experiments?
I did the research
https://jacobfer.substack.com/p/does-priming-work?utm_source=share&utm_medium=android&r=52htm0
I remember my wife driving us home after watching Tron. We made it OK, but a 1980's car was very poor at 90 degree turns!
Indeed: https://www.astralcodexten.com/p/heres-why-automaticity-is-real-actually#§most-priming-replicates-and-is-real
Idk, the one of these I understand the most of is the heritability of psychological attributes, and this seems sketchy to me (in both "directions" - heritability underestimated due to measurement error, biological determinism overestimated due to multiple factors). Can I assume the others are equally sketchy?
The difficulty is that if your discipline makes claims to lots of 'settled science' - as mine, ecology, does - then stuff gets accepted that shouldn't. And if a discipline is taught to undergraduates it will have a textbook...and you have to put something in the textbook. Everything put in will be treated as settled science unless you say different. One example, we teach logistic population growth in 1st year ecology. Logistic population growth makes quantitative predictions if you have estimates of r (intrinsic growth rate) and K (carrying capacity). But if we take 30 years of data, estimate r and K with the first 29 years of data then predict year 30 - a random walk usually makes better predictions. So, it's very questionable when or where this population growth model we teach in first year actually applies - if anywhere at any time. It wouldn't surprise me if social psychology in a similar predica(t)ment.
Calling something "settled science" is useful for winning arguments, which means it becomes overused.
Goodhart wins again.
Logistic growth is one of those things like frictionless planes in physics, or the tradeoff between inflation and unemployment in economics, where it idealizes away a lot of real-world complications to get at real causal understandings, but isn't very useful for empirical prediction (unless you learn all the factors to add to get rid of the idealization).
Hi Kenny, sorry it took me so long to get back to this. I agree that it ignores lots of other causal factors, but shouldn’t an idealization still make better predictions thana random walk?
do you have a recommendation for something i could read about why logistic population growth is a poor predictor?
Hey NLRG, a grad student of mine published a paper showing this to be true. But that doesn't address why it's true. I expect the logistic population model is a poor predictor because it doesn't include the factors that are the important drivers of population fluctuations. Maybe it's predators or disease or resource availability...but we don't know as yet. But here's the paper that showed the logistic model made poor predictions. https://pubmed.ncbi.nlm.nih.gov/38424456/
I think loss aversion is settled science. Is it? In any case, is there anything known about a physiological basis?
Very short term memory-- I don't think I'm the only one who can replay a second or so or sound that didn't register so I can understand it.
I'm bitter about science journalism. So much focus on the latest and possibly most hypothetical thing.
I am under the same impression about loss aversion so, same question.
Here's Scott's previous look at the evidence for loss aversion: https://www.astralcodexten.com/p/on-hreha-on-behavioral-economics#:~:text=III.%20Okay%2C%20But%20Is%20Loss%20Aversion%20Real%3F
When she was doing her master's degree, I helped my wife comb through the education literature. I was surprised how terrible the quality of the arguments and experimental designs were.
I remember one study on tracking (separating kids into gifted+slow learning classrooms). The authors compared the untracked classroom to the various tracked classrooms. By the end of the school year, they "found" average scores for "high track" (gifted) students were better than for the untracked class, but that the average scores of "low track" students were worse. They then attributed this to tracking as a practice ... not to the fact that their study design would have showed that no matter what.
I know what you're thinking, "sure, but what was the change from baseline?" LoL, this wasn't a rigorous study. You thought they'd have a baseline! Meanwhile the people on the other side of the debate had equally terrible studies.
Education researchers are, as best I can tell, worse at statistics than psychology researchers and are mostly dealing with stuff with more political and social salience, so it would be really amazing if educational research was in a better state than psychology research.
I don't want to paint with a broad brush in a field that is not my own. Especially since it's been a decade since I really looked into it. All I know is that what I saw in the late 2010's was shockingly bad.
I think there is both the problem that they have among the lowest GRE scores of any field, and also that they're so politicized they don't care about things like replication.
Out of curiosity, do you remember which study that was well enough to link it?
Education research is always hard because you can't really do controlled experiments, since you're dealing with real children who you have to care about and who live in the real world, and also because many of the effects you'd care about take longer to emerge than the publish-or-die pace of academia allows you to wait.
An additional problem, specifically in the U.S., is that a large fraction of students attending education schools are public school teachers who are there primarily because their collective bargaining agreements entitle them to an automatic pay raise if they get a postgraduate degree in education. This probably doesn't foster a healthy epistemic environment.
Don't remember, though I could go look up her master's thesis and try to find it ... but I won't. I was just chatting with Claude about the state of the evidence today, and it does not seem any better.
To be fair to the teachers, I think these master's programs are often poorly targeted. They don't teach future teachers skills for the classroom, and this exercise was no exception for my wife. I think there are plenty of teachers who would consult the research literature if it were useful and reliable. But since it's neither, I'm not surprised they don't take the exercise more seriously than a hoop to jump through so they can get back to their real work.
I am a bit suspicious whether this isn't just about a particularly bad sub-field of education research.
For example, the first "settled" question of education research that came to my mind is spaced repetition. I would assume that people remember facts better if they don't get presented them once, but with spacing, for example repeat it once after a day, once after a week, and once after a month. (Even if the total exposure time is the same.)
My naive assumption is that this is established by high standards. Like, that it is reproduced dozens of times with good study designs that vary all kind of details. And that it is also studied in practical classroom settings and holds up there. Is that not the case?
The second settled question that came to my mind is that you learn interconnected facts easier than isolated facts. The third one is that you learn better if you are emotionally engaged than if you are emotionless/bored. The fourth one that students learn better if they are activated. All of these are very relevant for practical teaching. Also here, I was assuming that all these are supported by high-quality studies. So what is actually the quality of studies for these questions?
the first three of these were discovered by psychologists, not education researchers:
https://en.wikipedia.org/wiki/Spaced_repetition
https://en.wikipedia.org/wiki/Schema_(psychology)
https://en.wikipedia.org/wiki/Emotion_and_memory (notice the names of all the journals being cited)
they're obviously relevant to education, but they're not from the education literature.
im unsure what you mean when you say a student is activated so im unable to comment on that
Ok, I don't know what exactly counts to which fields. But surely these things have been tested in classrooms as well, so it is also studied by education researchers, or not? Or are the lab studies by psychologists good and the classroom studies bad?
Activation is the concept that the students should actively do something during lessons. For example, make them think for themselves instead of just listening to you, or have them explain the concept to their neighbor in their own words, or let them vote on multiple-choice questions. In the broadest sense it could even be something unrelated to the material, like have all students stand up that understood the last 5 minutes. Whatever prevents them from just passively sitting in their chairs.
Though GPT is skeptical of the last type of activation that is unrelated to the material. I think it is a common concept, but it does not seem well supported.
These is a subtle difference between the well established research on effective learning techniques, and much of the dubious research on effective teaching techniques - especially as they relate to K-12 education.
The difference is whether the question you're asking is, "How do I learn and retain this information?" which is different from, "How do I get a diverse group of reluctant children to learn and retain this information?"
I heard about a university research that tried to experimentally verify Growth Mindset.
They split students to two groups. One of them educated traditionally, examined at the end of semester. Another group got an extra lecture on the importance of growth mindset, and also they did the same exam three times, so that they could see how they are improving (to support the idea of growth mindset by personal evidence), but only the third attempt at the end of the semester actually counted for the final grade.
Amazingly, the second group did much better. Growth mindset proved, checkmate IQ believers!
I asked: "Wait a moment! Is it possible that the second group did better simply because they had two extra opportunities to *practice* their final test?"
Response: I guess that might have helped a little... but most of the effect came from the growth mindset. (No evidence for this statement provided.)
> Most supplements are stuck in Poor Quality Study Replication Crisis Hell, but melatonin and l-theanine stand out as especially promising
Also on supplements, most nootropics studies that I've seen seem to understandably focus on the metrics that are the easiest to measure, like reaction time or N-back performance, but these are only tangentially related with what someone taking nootropics wants.
"The dumb woke psychology was wrong and bad; it seems useful to punish/discredit psychology as a whole to assist my campaign against it, so for propaganda purposes I will try to discredit psychology as a whole"
I agree that doing this is bad, and that people who reason this way have the problems you describe.
However, 'I ought to punish bad actors who peddle partisan nonsense using my money' is at least reasonable. Cambridge delanda est, etcétera etcétera. I think it is reasonable to hold partisan actors to partisan standards. But with this must be drawn with a line protecting non-bad-actors, and, ah, certain figures who are making anti-Harvard decisions have not been doing a perfect job of it. I prefer their route, to put my cards on the table, but for other reasons. On the limited topic of science, I think it is useful to split 'these researchers signed on to statements saying x y z, either lying for power / their own gain to do so or genuinely believing it, and this is disqualifying' from '[insert field I dislike] is inherently bad'.
It's hard to disentangle those. A field may not be "inherently bad", but it may be in practice bad.
There's also the problem, not specifically of money, but that the dumb woke psychology is being used as an excuse for drastic social and political changes. Using my tax money is bad, but using policies based on bad psychology as an excuse for forcing your political beliefs on me is worse. Not a lot of people are demanding policies based on "oh, we found the receptor for psychedelics", so pointing out that that is genuine knowledge is beside the point.
True. Different kinds of science (and 'science') are likely more or less vulnerable to this sort of thing. At the same time, I think that nonsense will find a way (take, idk, rent control or anthropogenic climate change denialism), though coming from sources other than universities will greatly improve a country's epistemology.
But with AGW, you have denialists from one side saying it's all made up, but also fabulists from the other side saying global warming is going to destroy the world or extinct mankind or something. But it sure seems (from my outsider perspective, with many problems with my filters and information sources) like only one side of that gets a lot of organized feedback from climate scientists.
I think this is in part because the denialists are saying things that are confidently known to be false, while the alarmists are mostly stating as established fact things that are possibly true but uncertain. I agree that I would like experts to do a better job correcting the latter kind of misinformation, but it's harder to do since people are entitled to their own opinions about ongoing scientific controversies.
(Also, it takes time for things to be corrected; today I think most climate scientists are pretty sure that the most catastrophic climate-change scenarios aren't going to happen, but that wasn't true 15 years ago.)
Climate scientists aren't neutral truth-lovers, as a class. They generally believe that AGW must be mitigated, and if it takes a few white lies to move the clueless masses in that direction, then those are amply justified.
That logic does not hold when the "white lies" are demonstrably pushing the public in the *opposite* direction
Climate scientist here. One important factor is that the false statements from denialists and fabulists tend to be different in kind. For example, a common talking point for many denialists is that carbon dioxide can't possibly be having a significant effect on present-day climate change. That's incorrect as a matter of physics. But you won't hear many fabulists saying microplastics are having a significant effect on present-day climate change, which would be similarly incorrect as a matter of physics.
The false (or at least unlikely-to-be-true) statements from fabulists come much farther downstream and are often a matter of interpretation. For example, a quick google turned up this reasonably nuanced discussion on whether climate change is an existential threat to humanity: https://climate.mit.edu/ask-mit/why-do-some-people-call-climate-change-existential-threat
While I agree with the general argument, I think that comparing the solidity of those facts to change of paradigm in astronomy is maybe a bit too much. To just single out one instance over which there is very good evidence that the mainstream interpretation is probably wrong, most of the simplified picture of cortical reorganisation that is included in the list actually seems to come mostly from the wrong notion of brain function localisation (ie, many areas being already partially involved in the functions they help recovering). For a very good exposition I can recommend https://elifesciences.org/articles/84716
re astrology: I've met a number of people who I believe would benefit from learning the first rule of astrology - different people are different - that doesn't make it a science, though, just entertaining and perhaps useful.
One difficulty is that it is hard for outsiders to distinguish between the parts of a scientific field that are pretty solid, the parts that are probably more-or-less right but still under construction, and the parts that are extremely shaky. This is obviously made worse by media outlets that report on the latest n=5 study showing that too much social media exposure turns frogs gay or something.
But also, a lot of high-prestige scientific organizations, publications, and universities have visibly and loudly make pretty nonsensical public statements as part of various culture war battles or current hot political issues. And again, from the outside, it's not so easy to tell that in this case, that's Dr Schmoe, whom everyone knows is kind-of a twit who will say whatever gets her in the papers, or that in this other case, the organization is making some public political statement and most experts in the field are kind of wincing and remaining silent rather than get dragged on social media.
A nice example of this was that infamous open letter by public health experts about how after George Floyd was killed by the cops, it was okay to protest because racism is a public health crisis, too. As I understand it, the actual people who signed the letter were mostly not very important or prominent people in the field, but of course as an outsider, I had no way to know that without doing a bunch of extra research.
But also, a bunch of gender ideology and race ideology stuff got put out during the Great Awokening that was obviously someone reading the room and saying the current thing without any concern for being factually accurate. And mostly that seemed to go without pushback from actual experts in those fields, though it's not so clear if that's a problem with who was heard or who was willing to speak. When I see a respectable institutions spread obvious nonsense for political/social reasons, I am much less willing to give their other statements much weight.
As an outsider, how should I evaluate what findings/expertise from anthropologists or sociologists or public health experts or psychologists is actually very solid, what is shaky, and what is just them saying what some donor or an angry Twitter mob wants them to say?
Wasn't there quite a bit of work on stereotype threat? And a lot on implicit bias tests? As I understand things, neither of those have held up too well....
Important note on implicit association tests: This refers to multiple different findings. "People have faster reaction times pairing socially privileged groups with positive-valence concepts, and socially oppressed groups with negative-valence concepts, than vice versa" has held up fine. "People who have larger reaction-time differences on those associations are more likely to behave in prejudiced ways" has not.
Yeah, that fits my understanding too. As I understand it, IATs repeatedly give consistent results, peoples' scores are pretty stable, etc., but they aren't useful for predicting behavior. Which makes them oversold (I think people were offering IAT testing/training to police departments to reduce racial bias), but not false.
I'll never forget about The Smithsonian infographic claiming that punctuality was an aspect of white supremacy.
Zvi's exhaustive covid coverage let me build a model with enough gears that I generally feel comfortable assessing public health claims...no idea about other fields though. Scott used to do this for psychology and various other science things, but that's been fewer and farther between as the years go by, and it feels epistemically shaky to keep referring back to SSC posts that are sometimes 10+ years old. Surely the consensus has changed since then! (There's a lot of good bonus content on his Lorien Psych site, but it's nowhere near as fun to read, and I always forget it exists.)
And again, it's hard to tell from the outside how much stuff like the Smithsonian infographic (which was embarrassingly bad) represents actual elite thought vs some dimwit given a bigger platform than they deserved. Though this also illustrates the problem with institutions giving a pass to low-quality scholars/contents with the right partisan or culture-war valence.
If I read some internet article about a published result that makes some culture-war arguments and clothes them as science, it's pretty easy for me to know that I shouldn't give that a ton of weight--the writer probably wasn't trying to find truth, they were trying to find clicks.
If I read an op-ed from a professor at Harvard making some culture war argument and clothing it in science, or see some display put up at the Smithsonian, or read an editorial in Nature doing that, the "it's just some nobody clown getting a megaphone to make you mad" defense doesn't work anymore. Those are prominent institutions putting their prestige on the line. If many Harvard scholars show up in public to speak nonsense, I'm going to lower my opinion of Harvard.
The Smithsonian infographic was actually a reasonably good and concise summary of the cultural norms of modern Western civilization. It just missed two critical details. First, Western culture *works*, and for empirical metrics like peace, prosperity, scientific and technological achievement, it works spectacularly well. And second. while invented by white people, it has long since adopted an open-admissions policy such that e.g. pretty much all the black people I know are full members.
I'll never forget going to the Smithsonian, and looking at exhibits shortly before they were reported (and removed) for wrongthink (Asbestos endorsing doctors, cigarette endorsing doctors, very bad things to be mentioning in a covid19 era where "trust the doctors" was the watchword).
Mostly because this occurred circa 2020, and around 2025, the Smithsonian decided to make a big stink about the government wanting to decide what it got to put up (as if this had never happened before).
One of the things you want from museums and historians and reference works is a long view that isn't just tuned to the burning issue of the day.
That's why the "doctors endorse cigarettes" was such a thought-provoking exhibit. And also why it had to go (notably, without the Smithsonian complaining about the expungement of wrongthink).
Exactly. If I'm not able to assess the claim's trustworthiness, assessing claimant's trustworthiness seems like a reasonable fallback.
https://www.bbc.com/news/articles/ce86jzgxxy4o
https://quillette.com/2026/02/05/no-bodies-no-accountability/
You should assume that most findings are false, of course. Some are even planted by enemies. And sometimes, the idiots just don't read the directions (The Americans that had used the GPR first started laughing when they saw the Canadian conclusions).
What is connectionism?
https://en.wikipedia.org/wiki/Connectionism
"Instead, you can say the thing you mean - that woke partisans shouldn’t try to occupy and exploit the otherwise-noble science of psychology for their own political gain"
I think part of what's driving the Camp 2 people is that academic research psychologists -- of whom Scott, notably, is not one -- didn't say this. When the microaggression industry called itself science, did we see a meaningful movement of workaday academic psychologists denouncing them? We saw the opposite; that seems important.
>We saw the opposite
Did we? What makes you say that?
Well the APA did their thing, just as you'd expect, and social media denizens did theirs. But the background professional sentiment was that microaggression work was good:
https://www.siop.org/tip-article/holding-the-line-strategies-for-sustaining-dei-amid-the-rising-tide-of-opposition/
I seem to recall that some fraudulent studies involving people being committed to mental institutions (it was all made up, IIRC) were a big part of the movement to deinstitutionalize. This seems kind-of relevant. If the good science is there and known to the practitioners, but the bad science is used by congressmen and judges to make policy, we have a big problem!
>If the good science is there and known to the practitioners, but the bad science is used by congressmen and judges to make policy, we have a big problem!
Eh. My impression is generally that congressmen and judges are going to do whatever they want, and will use whatever they can find to justify it post-facto.
If there's some fringe science they can misinterpret to justify the thing they already wanted to do, great. If not, who cares, they'll appeal to traditions or christian values or public safety or whatever-the-fuck works in the moment.
I don't think the science has any casual role here most of the time, so I'm not worried about it existing for this reason.
Congressmen and judges follow public opinion, and it's possible for studies to shift public opinion. I suspect this happened with the Rosenhan experiment.
Surely sometimes people are just looking for a study to support what they want to do anyway, but also, I think that people who read about this result when it came out probably thought it was telling them something true about the world, and their subsequent decisions were bad partly because their model of the world was wrong.
"To people in the second camp, my response is: deliberately exaggerated propaganda campaigns are bad and this is a confession, not an excuse. Inability to view science through anything but a political lens was what got us into this problem; doubling down on it is a temporary solution but a recipe for permanent disaster. If you pronounce entire fields of science to be acceptable collateral damage in your campaign against bad political actors, you will alienate not just scientists but patients, clinicians, and general lovers of truth. Instead, you can say the thing you mean - that woke partisans shouldn’t try to occupy and exploit the otherwise-noble science of psychology for their own political gain - and wage the same campaign with a halo rather than a comic-book-villain goatee." I agree in theory--science should not be politicized but it always has been. There's a reason Gaileo was persecuted, his discoveries challenged the authority of the Church. Scientific findings and truth(and truth in general) are challenges to power of incumbents, the legitimacy of society and traditions etc etc. At some level bad actors will always use scientific findings to push their agenda or alternatively disregard science when it goes against their political interest. The only way out is for good scientific actors to have enough legitimacy in a political sense(democratically popular not just correct on merits or actually factually correct) to be able to counter bad actors. Science will always be politicized/weaponized for those who have or want power to enact a particular agenda--only actors who have both political and scientific legitimacy at the same time will be able to stop that.
> There's a reason Gaileo was persecuted, his discoveries challenged the authority of the Church.
That's not the actual reason Galileo was persecuted.
https://tofspot.blogspot.com/2013/08/the-great-ptolemaic-smackdown.html
The Church claimed authority over theology, rather than astronomy, and didn't approve of his use of theological arguments in support of his astronomical ones (which happened to be incorrect in the case of explaining the tides).
Also, his book not-so-subtly insulted the pope, which turns out not to be a good idea when you live under the authority of the Church.
I agree that it's a disaster that science has become so politicized, and that the likely outcome of that is people making worse decisions and convincing themselves/others of more nonsense. This is a big problem that needs to be fixed!
I think most of the work here has to come from within the individual fields. If people are speaking politically-convenient nonsense in the name of your field, you need to try to oppose that, or the public (including very science-positive people like me--I'm a working researcher FFS) will discount your field's findings as being mainly about politics and not getting the best available picture of reality. If the people at the top of your field are making public statements about how activism is the core of your field, okay, I'm probably going to believe them. If it turns out that being known to have the wrong politics or publishing results / doing research in areas with the wrong political valence blights your career in that field, and I learn of it, then again, I'm going to assume that I can't take the findings of your field at face value. The main people in your field have made sure of that.
One of the many ways this goes wrong is that it's hard for outsiders to know which fields are real vs BS fields. A math professor may have silly political or social views, but his actual field is serious. Is the same true for a gender studies professor? That looks like a field that's mostly about ideology and politics, but maybe I'm being unfair--how would I tell?
Gender Studies is mostly about ideology, but so is Climate Change. And you wouldn't believe how much trouble you can get in for nominating someone for a Nobel Prize in Physics (again, see Climate Change -- notable nobel prizewinners have commented on the relative bunkness of the field).
As a Social Psychologist by training, I really appreciated the recent article. I also agree with this update. Relating to the microagression example, I would add, as someone whose graduate research involved aggression that the whole concept of microagressions isn't actually from social psychology. It originated in the 1970s, but wasn't really studied until the 2007. Those researchers come from clinical and counseling psychology. In my experience, social psychologists don't cite that research and it's considered pretty unconvincing among actual aggression researchers. Scott Lilienfeld wrote a great critique of the microagression literature that captures my thoughts on it pretty well: https://homepages.se.edu/cvonbergen/files/2018/05/Microaggressions_Strong-Claims-Inadequate-Evidence.pdf
This seems like a combination of (a) not psychology, (b) meaningless tautological statements that are unfalisifiable, (c) things already known or suspected to be false
(1) just says "there's a prediction-error-based learning algorithm in the brain". I never understood what predictions that makes and what it would mean for it to be false
(2) is not really psychology
(3) is false I think? Your links don't demonstrate that the brain can use neuroplasticity to recover from damage to Broca's area
(4) is controversial but seems likely to be false, as new molecular heritability studies place certain heritabilities closer to 30%, while old twin studies (and old SSC articles!) place them at 80%. Unless you mean that everything from 20%-80% is "about 50%", I guess. As for "environment does nothing", see birth order effects on SSC readership!
(5) is nonsense; your link doesn't actually give any evidence that brain waves have to do with consciousness, it's just pure woo and "woah, waves, isn't it cool" level stuff
some of the later ones are a bit better, I'll grant.
Isn't the heredity issue basically the difference between "we observe this heritability based on looking at identical vs fraternal twins" and "we found a set of genes that account for this level of heritability?" I wouldn't take GWAS studies as the final word on how heritable things are, just as a lower-bound on stuff we can pretty-much know.
No, the modern molecular studies are not GWAS. They look at random variation in how much DNA is shared by siblings (on average it's 50%, but it is sometimes 40% and sometimes 60%), then check whether more generically related sibling pairs are more correlated in phenotype; there's no need to find the actual set of genes responsible.
Thanks. I'll try to find an overview of the research by hitting up Claude, but if you have time to suggest one, I'll read it.
Scott himself wrote up a review: https://www.astralcodexten.com/p/missing-heritability-much-more-than
I will simply repeat my point from the previous post: we shouldn’t care about results before they have been replicated three or four times.
I suggest we in this community who value truth memorize and deploy this stock phrase: "yeah but has it been replicated?" And simply ignore any new studies or findings that have not yet been sufficiently replicated (no matter if it confirms our biases and hopes and dreams or not!)
This is a reasonable stance for the type of social psych experiments that can be done on any group of 50 undergraduates, but generalizes very badly to other fields.
I don’t think the excuse "it’s hard to do" makes it any better. There is an argument that if the effect size is large and also N is large in the original study, maybe you don’t need so many replications, but I will hold firm that you do need at least one independent replication in absolutely all cases. That should be table stakes for something we call "science".
It generalizes very well to fMRI studies! They're generally light on the N. Do ten of them, and you get a decent (replicated) result. Expensive studies mean you're trying to get the most out of the least amount of people (it's even worse if you need to get people who have already been admitted, and are as yet untreated.)
As well as the all-important phrase to put after like 95% of media-reported health science papers: "...IN MICE!"
> > Freud was “settled science” for a long time ... Astrology? Phrenology? These seem different in that their entire orientation was flawed from the start...
I could be wrong, but from what I understand, Freud's conclusions rested on the kind of methodology that wasn't very much stronger than the methods behind astrology and phrenology. If that's what passes for "settled science" in psychology today, then maybe the entire field *should* be discarded.
Physical medicine was worse than useless for most of its history https://westhunt.wordpress.com/2016/03/31/medicine-as-a-pseudoscience/ But we shouldn't discard the entire field NOW just because it was consistently terrible before.
I agree, but at the same time, modern medicine had discarded e.g. Humorism. If modern doctors still considered Humorism to be "settled science", then yeah, I'd argue that modern medicine is due for a serious scrubbing if not a total rebuild.
You mean, throw out the baby with the bathwater?
Yes, absolutely, if it's 99% bathwater and 1% baby.
You're quite sure of that?
I did say "if".
Note to self: Do not give Bugmaster the job of cleaning out my swimming pool if there's any chance anyone will be swimming in it!
You probably occupy more than 1% of your swimming pool, so you should be safe :-)
Clearly I need to up my dose of Ozempic!
When you sit around the pool, you sit *around* the pool ! ... ... I'll see myself out.
4.) How do you separate random form the randomness of genetics. My kids (sample size of two.) seem to be a mixture.
Traditionally, twin studies. In more recent decades, genotyping. (Lately the two have given seemingly contradictory results, which has created a spicy scientific controversy that Scott has blogged about a few times.)
>This isn’t a claim that settled science is always right - Freud was “settled science” for a long time
I always want to stand up for Freud a little bit. Today we hear about his wackiest bits of psychoanalytic theory as color commentary about how weird the field used to be. But his most substantive contributions were ideas like 'maybe significant events during your childhood affect your personality and behaviors as an adult' and 'maybe you are doing a lot of mental processes outside of direct conscious experience that still affect you in profound ways'.
These paradigms are so core to our understanding of everyday experience that they seem 'obvious' to us, like something no one would have had to 'invent' because obviously everyone would know them just from living their life. But AFAIK these were really not obvious and accepted truisms before Freud popularized them.
This is sort of a 'reading philosophy backwards' thing, it's hard to appreciate the contributions of people who popularized ideas that are now so widespread that they seem 'obvious' and sourceless.
To the extent that Freud was correct about these things, was this due to his robust methodology, or was it because even a stopped clock is right twice a day ? My impression is that the latter might be more accurate...
I think that the "stopped clock" framing is a bit unfair. Fields of study with strong methodologies don't emerge already fully formed - somebody has to obtain the initial vague good ideas through some other means. I'd say that it's fair to credit Freud with doing just that, and at the same time to blame him for constructing an enormous edifice of nonsense atop that meager foundation, although everybody else deserves much more blame for taking that nonsense seriously.
I hear what you're saying, but Freud wasn't born in Ancient Sumer. He was (roughly speaking) a contemporary of Maxwell. By that time the scientific method had already advanced beyound merely thinking really hard about aesthetically pleasing ideas, and toward rigorous measurements and hypothesis testing via probability theory. So Freud IMO has no excuse -- but you're right in saying this also applies to any people who took him seriously.
Some of this is likely hindsight bias - Popper's falsifiability was still a few decades away (largely in response to Freud & Co, as it happens), and it's inherently more difficult to be rigorous in "squishier" domains (something that's still clearly relevant, considering that "maybe psychology is all bunk" is a very topical idea).
The problem here is that psychology has a moving target; people and cultures change, and what might be true in one time is not so true in another.
That is a separate problem. Climate also changes over time, yet it is possible to study climate quantitatively, as opposed to just going on vibes, as psychology appears to be doing.
Well, I don't think comparing the climate to people's personalities is a very good comparison. However, I'm not entirely sure what you're throwing into the basket of psychology here, and I'm not trained in the field. I have my own experiences with it and my own opinions about it, but that's my own personal weather, isn't it?
How much of the good commonsense stuff did Freud actually originate, though? The idea of a subconscious/unconscious mind dates back to Plato, or at least that's what my own intro-psych textbook said.
Right, and I doubt Freud was the first person who thought childhood affects adult personality. Anyone who thought about education would fall into that category.
> maybe significant events during your childhood affect your personality and behaviors as an adult
Having read Judith Harris, I'm still skeptical of that. I still think Freud was a pseudoscientist.
I don’t think you are interpreting Harris work correctly. I think her point was that peer groups are much more important than parents in one’s behavior. It was not that significant events had no effect. Of course, the whole thing exists on a Bell curve in that more normative upbringings, shall we say, tend to resolve themselves without any great influence, but psychology only really gets interesting when you’re dealing with the tails
One of her points was that even the Holocaust didn't seem to affect survivors that much, and that surely qualifies as a "significant event". And there's theory behind this: Trivers pointed out that there's a genetic conflict of interest between children and parents, so that if children could be molded by parents, the parental interest could win out. But if we've evolved to be resilient, then we'll continue to be whatever we were going to be regardless of "significant events".
Well, I suppose it depends on the survivor. There’s lots of accounts of people who were affected very deeply. I would say we are going to be what we will be including our significant events, not in spite of them. A lot of it depends on at what age a significant event occurred. I don’t think they had much patience for small children in concentration camps.
There's a common belief that there's a "formative age" that matters, but per Judith Harris the reason not to abuse the child is the same as the reason not to abuse an adult. Steve Sailer likes to quip that the relevant time keeps getting pushed back further as studies/interventions fall prey to Rossi's Iron Law, to the point that it can be nine months prior to birth BUT NOT MORE!
>the reason not to abuse the child is the same as the reason not to abuse an adult.
no doubt but the reason is not the issue.
I believe brain development in humans has been well studied and divided into stages. Things land very differently depending on age and circumstance. Or does Steve Sailer believe that a young child can be subjected to physical, sexual or extreme emotional abuse by their caregivers and it ain't no big deal?
This discussion feels like it's turning into a circular firing squad. I don't have a very high opinion of Harris and too much depends on the specifics for generalized conclusions. As I said before, it's the tails of the curve where things get interesting. Even "a significant event" is highly contextual.
https://www.lesswrong.com/posts/ZbApyaLw4PeamCtEF?commentId=4HDNjxZvnJn8XiBs4
> To put things in context, to show how diseased psychology was in Freud's era, let me just say that the most famous Freud's student and then competitor, Carl Gustav Jung, rejected much of Freud's teachings and replaced them with astrology / religion / magic, and this was considered by many people an improvement compared with the horribly offensive ideas that people could be predictably irrational, motivated by sexual desires, and generally frustrated with the modern society based on farmers' values. (Then there was also the completely different school of Vulcan psychologists who said: Thoughts and emotions cannot be measured, therefore they don't exist, and anyone who says otherwise is unscientific.)
> I think the bad PR comes from combination of two facts: 1) some of Freud's ideas were wrong, and 2) all of his ideas were controversial, including those which were correct. So, first we have this "Freud is stupid" meme most people agree with, however, mostly for wrong reasons. Then, the society gradually changes, and those Freud's ideas which happened to be correct become common sense and are no longer attributed to him; they are further developed by other people whom we remember as their authors. Only the wrong ideas are remembered as his legacy.
A lot of the parts of Freud that weren't crazy just sound like common sense today, but they mostly weren't at the time he was writing.
He was also one of the first people to say "hey, maybe we can study people's minds scientifically just like everything else" and actually have it catch on.
>I won’t dispute this as an accurate description of how the sausage gets made,
I would dispute it inasmuch as I don't think the science is the driving factor in most of these cases. Rather, I think the typical story is that there's a political movement that's looking for any rhetorical tools it can find to justify its own necessity and increasing power, and skimming journal articles for any finding it can appropriate or misrepresent to its own ends is just one of the dozens of avenues it uses to do this.
No matter what the scientists believe or prefer, I don't think there's much they can do to stop political movements from using them this way. I feel like I very often see the scientists in question going 'no, stop it, that's not what my research means, those claims aren't justified by my findings' and getting ignored as the political movement keeps on citing them.
Which isn't to say there aren't also cases where the scientist in question sees an opportunity for fame and fortune, and joins the political movement as a willing stooge. But even then, I still see the political movement as the driving factor; the existence of the science is not itself the causal force creating the political movement people are complaining about.
My perspective is undoubtedly skewed, as I wouldn't have a way to know that a scientist protested against misuse of their research. Given that, most examples I do see involve people jumping on the bandwagon. In fact, it's not just individual authors. Plenty of examples where major governing bodies (AMA, APA) and respectable publications were happy to do the same.
An example the other way from many years ago was the APA statement on IQ and intelligence, published during the controversy after _The Bell Curve_ was published. My impression (it's not my field) is that this was in fact a sober and honest snapshot of what was known at the time about intelligence and IQ.
Right. IQ is a field of which tens of millions of people with college degrees have 180 degree wrong notions of what is the settled science.
There are few easier ways to make yourself subject to rage and name-calling than to mention the settled science regarding IQ and race, even while carefully explaining that the fundamental causes of these endlessly replicated racial differences in average IQ are not at present settled science.
Settled science impacts ideology and goes splat. Rinse, wash repeat.
Someone on DSL had some line along the lines of "politics+science=politics," which I think is basically true.
Science is hard, and social science is so damned hard we can just barely do it. Adding in current political battles, ideological concerns, and threats of outrage campaigns is like hitting a delicate piece of equipment with a sledgehammer.
I don't think camp 1 is as simple as you make it out to be. Psychology as a field has been used for decades now to advance progressive social causes, primarily by medicalizing bad behavior in order to garner sympathy for the practicioners (the poster child is alcoholism). Was "woke psychology" meaningfully different or did the movement just get out in front of its skis a little bit? Apparently the scientific mechanisms can't be relied on to distinguish between the hypotheses.
When she sucking your nuts but you a gangsta
Well said.
I guess the biggest blow to popular psychology is, despite the frequent confirmation of genetic inheritance you have mentioned, it's still fought tooth and nail in every step, like you have covered. So not only popular psychology not have too much replicable results, they reject ones they do have.
Like some other people I was wondering whether you make any distinction between psychology and neurobiology/neurophysiology.
Any suggestions for members of the general public who want to know if a particular claim is part of settled science rather than an unsettled part of the research frontier? In the absence of enough time to read through a textbook (which may or may not cover the particular claim...), and the absence of a professional in the field with time to spare to answer such questions, are SOTA LLMs fairly accurate or not on "Is <claim X> settled science?" questions?
I think they're pretty good. I don't ask whether something in psychology is settled science, though. I write something like "where I left off the prevailing view was this [and I describe it in a paragraph]. Is that still the prevailing view, or has more research and thought cast doubt on the it, or has some new view replaced it? Please search the last 20 years of studies and articles on the topic using only higher-quality sources such as juried journals, paying particular attention to well-done meta-analyses, then summarize the result, giving links to the most relevant studies."
Many Thanks! Your query is much more precise than what I'd been considering using. Great approach! I'll use it too.
I think there's a "the barman told the man in the labcoat with the 'I p-hack to achieve social justice aims' button to leave because he was a bad scientist, then took a funnel plot with nails in it from the back room and chased him away" story going on here, in that if you let bad scientists hang out in your bar then it's now a bad science bar.
Ironically, all the bad scientists getting chased out of the psychology bar has caused the general public to notice they were in there in the first place, uninterrupted, for quite a long time.
Have the scientists actually been "chased out"? Dan Ariely is still at Duke. My recollection from Andrew Gelman following a lot of this stuff is that Ariely's co-author Gino is the exception for being kicked out.
It's remarkable to me how many different groups of people in positions of great trust and authority--police, priests, doctors, researchers, teachers, military officers, etc.--follow the same instinct to close ranks and protect their own from consequences of even really terrible behavior.
This defense of the field of psychology bears the characteristics of a motte-and-bailey fallacy. You have established a motte of mostly neuropsychological findings, and you are using this motte to defend a bailey consisting of trash research that has a far larger on people's daily lives than the neuropsychological part does. Furthermore, these neuropsychological findings rest on different research methods from those employed in the rest of the field of psychology. I say we abandon the bailey altogether; it has proven to be barren ground in these past decennia.
Abandon what? Any attempts to understand how people and groups behave, which as you say has a large on people's daily lives? It would have to be restarted somehow in a better form which is easier said than done
I think there is sidficient and meaningful evidence that social psychology is about as meaningful phrenology or zodiac signs. It must no longer be understood as a science. Instead this kind of research should be understood as a literary endeavour.
Dostoevsky foresaw this: “the whole work of man really seems to consist in nothing but proving to himself every minute that he is a man and not a (machine)”
It is impossible to stop the self-reflexive loops of consciousness without destroying a mind. The attempt of psychologists to formulate rules for the functioning of the mind is in its final reduction nothing else than an attempt to destroy our souls. If any mind reads and understands the ‘scientific laws of psychology’, this mind has definitionally superceded these laws by placing them inside a larger epistemological framework, making the laws meaningless.
We should not stop attempting to understand ourselves, but we should stop pretending that introspection has anything to do with the scientific method and falsificationism. ‘Psychology’ is much more akin to an art form or a dance, and it is all the more valuable and beautiful for it.
This seems to be a consistent blind spot for Scott. He sees science (true/false) issues where others see people (honest/liar) issues.
A good example is his claim here that people cannot distinguish good psychology from "dumb woke psychology". But to many, the folks who decide what "settled science" is, in psychology, lost credibility. The "dumb woke psychology" was literally "settled science" not long ago.
Same thing with covid arguments. Scott says "here's the most recent evidence, let's evaluate it". And I say "that person lied before, why should I trust their evidence this time?" People even invoked "settled science" to shut down inconvenient information during covid, so it's odd to see the term used unironically here!
That's also part of the appeal of populist politicians. "I may not be fully on board with the [DSA|MAGA|gold standard|canine personhood] folks, but they have not lied to me (yet)." Conversely, nobody gets worked up about string theory, except physicists, since people behind it aren't trying to sell me on anything.
I think there're interesting things to be learned in this area, e.g. it's entirely possible that the people-based evaluation method is not optimal. But the "people are either stupid or trying to discredit psychology" is not it (see "it’s worth examining the trains of thought..." section).
To get out of the politics & culture war area, the opioid crisis is another example. For a while, it became "settled science" that opioids weren't particularly addictive, at least among doctors. American Medical Association and American Society of Anaesthesiologists were more than happy to jump on the marketing push from Purdue Pharma and Johnson & Johnson.
I don't know how to operationalise that information, it's not like whenever I hear of medical advise I do the opposite. But it does make me a little less trusting of "settled science" and a little more likely to "do my own research".
It's worth reading a contrary view on opioids
https://entitledtoanopinion.wordpress.com/2020/08/11/a-contrary-perspective-on-the-opioid-epidemic/
I note there that I'm not entirely persuaded, but a lot of discussion on this exhibits the flaws of populism also found in other domains.
Yeah, it's worth remembering that there's a constant pendulum swing on opiates/opioids for pain control, because there is a genuine godawful tradeoff between turning people into addicts and leaving people in unbearable pain.
Part of the argument made there is that people who aren't taking the drugs recreationally really are much less likely to become addicted. Of course, once the drugs existed, people WERE going to take them recreationally (defeating some mechanisms the pharmaceutical companies used to prevent getting a short intense high vs long lasting smaller effect).
>This seems to be a consistent blind spot for Scott. He sees science (true/false) issues where others see people (honest/liar) issues.
I thought a similar blind spot was evident in Scott's comments in the recent links post about AI being better at persuasion than the most persuasive people. I can believe AI presents a fuller, clearer, better-reasoned argument, and for that reason gets more votes from subjects for the case it was arguing. But full, clear written arguments are only one form of persuasion, and votes for good argument quality are only one measure of persuasion. It's clear that in real life a great deal of being persuaded happens via social identification, peer group pressure, being swayed by recurring unnoticed messages, vibes, seduction, direct and indirect rewards for having certain beliefs, etc.
MAGA lies all the time, even more often than other politicians. I will also say that I doubt that you actually can show that the sources Scott cited on Covid lied.
Suggested image modification to improve resonance with a title including the word"beauty": https://imgur.com/a/BDjc6W0
I'm generally in agreement with the post, but one potential nitpick regarding example 4 on heritability:
"but the further you go from this basic picture, the gnarlier the chains of indirect causation you require."
Isn't that put the wrong way around? To my understanding, other methods of estimating heritability at least partially get lower estimates by trying to control for anything but direct heritability, therefore removing gnarly chains of indirect causation. It's twin heritability that includes them all.
> Psychotherapy works with moderate effect size, but (with a few exceptions) seems to be more about the relationship with the provider than the details of the therapy itself.
This is a common but IMO misleading conclusion. Common factors research (which includes work on the therapeutic alliance/relationship) is almost entirely correlational (so it is confounded and many studies cannot even rule out reverse causation, where remission causes you to evaluate the relationship to the therapist positively), whereas the effect sizes we have on therapy effectiveness come from RCTs. The differences between therapies research comes from mainly underpowered RCTs which cannot realistically detect the differences we would expect. Presumably, differences between therapies that you can find “in the wild” are larger than what you can expect in RCTs, which have more quality control in all arms.
When randomized to patients, therapist effects explain only small percentages of variance.
So, I think the “alliance/relationship” stuff is mostly confounding and I would never confidently say that the details of therapy don’t matter (which also would require you to hold other beliefs included here, like that we understood reinforcement, which is core to CBT but not analysis, very lightly). IMO we have good evidence that CBT is at least more efficient, probably also more effective, I think this is even more true in the wild, when we are far removed from the quality control of RCTs
https://research.vu.nl/files/105340005/annurev_clinpsy_050718_095424_The_Role_of_Common_Factors_in_Psychotherapy_Outcomes.pdf
A little nitpick to 5: I think the term "subpopulation of neurons" is misleading, because it sounds like a permanent division. My understanding is rather that neurons spontaneously assemble into synchronized groups through brain waves. But the set of neurons which participate in a group or even generally which respond to a specific rhythm changes all the time.
I have never looked that deep into brain rhythms, but a quick GPT check (that I tried to phrase neutrally) seems to back me up.
But yes, brain waves exist, are important, and are linked to states of consciousness.
>"Most personality traits and general capabilities are about 50% genetic and 50% random, with little role for family environment or childhood history, with a few notable exceptions like partisan preference."
Sorry…what? Is this meant to imply that abusing or neglecting your children has no impact on their adult personality?
This statement is heavily simplified and easily misleading. I believe the source is that twin studies show 40%-50% of the difference in personality is "explained by" heritability (genetic and non-genetic), but heritability is a very confusing thing to measure, because e.g. most twins that participate in adoption studies end up in "safe" families (due to adoption requirements/patterns) so they have similar environments and therefore less of the twins' differences can be explained by environmental differences. The studies naturally include very few abusive households. In a world where all children are genetic clones raised in perfect families, you'll see studies saying that personality traits are "0% genetic/heritable and 100% random", but obviously if one child has a severe mutation or gets orphaned that child will be very different.
Yeah, the concept of "all else being equal" or "at the margin" is important here.
As I understand it, the best evidence is that among adoptive families (which are screened to be relatively well-off and stable), we mostly don't see big impacts on life outcomes from differences in upbringing.
That doesn't tell you that having a kid raised in a cave by a biker gang won't affect his life outcomes, nor that having him raised on a space station surrounded by geniuses won't affect his life outcomes. Just that *within the range of variation of the people in the study* we don't see a lot of those effects.
Very little in reality has cutoffs like that so I’m sceptical that there arent effects in the normal range given there are effects at the margins, even if the effects are smaller I the normal range.
But anyway - whether you have loving parents,’good role models, people who talk to you and read you stories, older siblings who torment you and make you feel ugly or small—-you really telling these don’t affect adult personalities?
I'm saying that the studies did not find big effects of upbringing on the measured outcomes. That doesn't mean there aren't other outcomes that changed a lot, or that there weren't either small or rare effects that didn't show up in the statistics.
There was a French adoption study where they tried to find 40 adoptees, ten each in four quadrants: born rich/raised rich, born rich/raised poor, born poor raised rich, and born poor/raised poor. But they could only find 8 kids for one of the Prince and the Pauper segments.
Anyway, they found a more sizable effect of Nurture than is normally found. This could just be the tiny sample size, or it could be that most adoptees during the golden age of adoption agencies (roughly 1945-1975) got pretty decent upbringings so we don't know much about what really bad parenting can do. For example, even as critical-minded of a personality as Steve Jobs was wholly satisfied with his blue-collar but loving, stable, and honest adoptive parents.
"The gamma rhythm corresponds surprisingly well to items in conscious awareness."
This is not settled science.
There is plenty of solid science on theta-band oscillations (especially in the medial temporal lobe in relation to memory function), alpha-band oscillations (especially in the occipital cortex and thalamus in relation to attentional allocation), and beta-band oscillations (especially in the motor cortex and basal ganglia in relation to the timing and organization of movement).
However, the gamma rhythm is measured with lower signal-to-noise than most other rhythmic processes, is more subject to various kinds of confounding processes (e.g. muscular artifacts), and also is subject to extensive terminological and methodological confusion (gamma band vs high-gamma band processes, band-limited rhythmic shift vs broadband/arrhythmic shift, sustained oscillations vs transient bursts). I'm not saying there is nothing there, but it's amongst the least settled parts of the work on brain rhythms, especially in humans. And that's before we get into the potential connections to consciousness.
Echoing other commenters that most of your examples of settled "psychology" seem drawn from neurology, neuroscience and psychopharmacology. These disciplines are correctly viewed as part of the hard sciences, and even a lay person who isn't very well-versed in the distinctions between them understands that not just anyone can interpret a brain MRI or conduct a neurosurgical procedure (per the common joke about a job which isn't challenging "it's not exactly brain surgery, is it?"). Even psychiatry seems meaningfully "harder" than clinical psychology, and most people intuitively understand that it's more of an accolade to qualify as a psychiatrist than a clinical psychologist. The criticisms of psychology (which predate the replication crisis by decades) tend to take the form that it's a soft cargo-cult "science" which makes claims which range from trite to unfalsifiable and produces no useful predictions.
I do understand that there's meaningful overlap between psychology and the other disciplines, but I think it's a bit of a shell game (dare I say a motte-and-bailey argument?) to insist that psychology is in rude health and then tout a bunch of robust findings from neuroscience, psychopharmacology etc. Where are the robust findings from psychology as practised by academics with no background in neuroscience, pharmacology etc.?
As I recall, a ton of very low-quality fMRI papers got a lot of press a few years back, so hard/soft isn't quite enough distinction.
A funny thing is that some of the most controversial fields of psychology, like IQ, are actually among the most settled portions of psychological science. Some questions -- such as how much is due to Nature, how much to Nurture, and how much to Noise -- remain unsettled, but the basic facts are beyond dispute after 120 years of studies replicating so much that they've gotten boring.
For example, that there are racial differences in terms of average IQ in the U.S. is completely settled science (why these differences exist is not, however). Yet, a large fraction of college graduates get outraged at anybody who mentions this fact, which they can't believe is anything other than pseudoscience of the most laughable kind.
I recall Steve Pinker pointing that out long ago.
Note that priming studies tended to appeal most to the Malcolm Gladwell-reading center-right marketing majors.
No science is ever “settled”; the notion that it even could be should offend scientists.
You know damn well what he means and there are obviously parts of science which are more or less provisionary and whose probability of truth is higher or lower.
Not denying science can provide higher probabilities of truth revealing, which is not equivalent to claims of it having been settled. My point is that scientific discovery, replication, advancement is a never ending enterprise.
Yes, of course but do you think Scott meant to deny that by using the term 'settled'?
But sorry if my comment was a big aggro. I just don't think taking Scott to deny that is a fair interpretation of him.
Many areas of science are "settled" in the sense that a prudent well-informed person would base a billion-dollar project on them, or stake their job or even their life on them being true. These are of course not beyond revision--perhaps tomorrow we'll discover that we were wrong about something we thought was totally nailed down. But they are solid enough to rely on in practice.
By contrast, we've also sometimes relied on very shaky and politicized scientific findings for large-scale government programs, changes to the laws, etc. That's the thing we want to avoid doing. Nobody wants to forbid you from trying to find some ways that general relativity isn't true, we just don't want to end up designing GPS based on indigenous ways of knowing about time and distance because a bunch of serious-looking scholars with PhDs said this was the right way to proceed.
Large parts of psychology are NOT settled. I mean, you can't really sit down and pull out good psychological research papers that you'd base x265 on, right? I'm pretty damn sure the people who wrote x265 didn't spend a year reading research articles. They spent that year doing their own research, and didn't put it in the psych research journals neither.
And you can count that as billion dollar psychology.
Yes, settled science is where the value is at -- the problem in psychology and many other highly empirical areas of science -- is that the incentives to weed through the many studies that suggest a new phenomenon and turn that into settled science isn't there.
If you replicate that work and it turns out to be valid you get very little of the credit because you weren't the first to publish and if you debunk it as a false positive you don't get much credit either.
It's not that there is anything wrong with the things that are being done it's just that no one is doing the hard work to actually turn new results into robust settled science. This encourages people to misinterpret things that aren't settled science yet as settled science because we haven't created a higher bar for them to look at.
Physics faced this problem to an extent and answered it with the 5 sigma rule -- it's not really a discovery until you hit 5 sigma. Psychology needs something similar and incentives for people to do the extra tests to turn promising early studies into settled science.
"The key distinction isn’t “recent research in social psychology” vs. “recent research in cognitive/bio/behavioral psychology”, as my essay last week unfortunately implied"
I'm not sure settled science vs novel science really explains what parts of psychology are reliable or not. Many of the findings within psychology which catastrophically failed to replicate are things which ex ante people would have said are "settled science", with scores of apparently confirming studies over decades (e.g. ego depletion which had hundreds of supporting studies).
At the same time I think you are underselling the extent to which this does map on to something like "social psychology" (and studies sharing the method of social psychology) vs biopsychology/neuropsychology studies not sharing the method of social psychology.
Here are some of the specific features of "social psychology" style studies that beget poor validity / reliability:
- Use of stimuli, often vignettes to try to manipulate general categories without attending to whether the stimuli are representative (e.g. through stimulus sampling) or unconfounded (https://pubmed.ncbi.nlm.nih.gov/40372824/), e.g. you try to test whether people respond better to "scientific" or "prosocial" arguments by presenting people one of two paragraphs arguing for something. But there's a whole universe of possible stimuli you could have used to represent your intended constructs, and the particular stimuli you used could produce different effects for a wide variety of different reasons.
- Use of measures with poor validity: you test how people respond to your vignettes by measuring their "endorsement of vegetable consumption" score (constructed by averaging likert items you made up like "I think of myself as a person who likes to eat salads") based on tests of reliability (like cronbach's alpha) or model fit measures which do little on their own to confirm whether the measures are measuring what you claim they measure.
A number of these points are made (I think persuasively) in Schimmack's Validation Crisis (https://open.lnu.se/index.php/metapsychology/article/view/1645) which explains why much psychology suffers from validity issues, not just a replication crisis solvable by pre-registration and better sample sizes. Many post-replication crisis studies have dutifully pre-registered well-powered studies which replicate, with designs that mean they are completely invalid.
I think it's clear that and clear why many of these features don't apply to the kinds of studies you highlight as being most robust (e.g. GABA receptors behave such and such way).
The The Beauty Of Settled Science ... PUBLISHED 16 HOURS AGO 73 LIKES 2 RESTACKS 7 COMMENTS framing is the part I keep coming back to. Most takes on this skip the underlying mechanism and try to win on vibes.
W. V. O. Quine's philosophy of science was that it's all subject to revision, but we typically rely on a stable "core" while modifying the periphery. He used a metaphor of modifying a boat while you sail on it (I don't recall where I came across that one, though I know Robin Hanson cited his metaphor of a "fabric" of many threads https://www.overcomingbias.com/p/ignorance-about-intuitionshtml ).
Almost every single one of the examples you cite either comes from cognitive science or behavioral psychology. It's a bit of a stretch to defend the whole umbrella of "psychology" by pointing to its uniquely rigorous and politically neutral subfields.
Re: education research, I recommend Carl Hendrick's substack.
Idk: if they want to act like a propaganda mouthpiece they can be treated like one.
When it’s technical enough that the writer probably isn’t thinking with their woke brain, sure you can just treat it like honesty.
But anything getting closer to actual people or society and they’re gonna go right back to their old habits.
it is free to just selectively ignore them. And there is little difference between me having to police and scrutinize them constantly vs me just believing my old priors anyways
> the main way that Psychology affected their lives was through bad clickbait studies used to buttress dumb policies; some p = 0.049 study says that exposure to microaggressions makes minorities’ brains shrivel into raisins, and now their company has to implement punitive DEI or be sued for hazardous work conditions.
Is that an actual thing that happened? I'm sure people have occasionally handwaved towards some study or another for justification, but I think the causal claim is very dubious.
I think DEI (commercial DEI, at least) has been largely sold on the basis that if you don't make people from minority groups feel uncomfortable, you'll be able to hire from a larger pool of competent candidates. Which is fundamentally true, and much like with psychology, the problem was in not drawing boundaries between the true stuff and various questionable add-ons; and in many cases in mixing up actual DEI (self-interested or ethically motivated HR concerns) with PR (the internet is angry at one of our employees at some reason, and the minimum-effort solution is firing them).
The best way to understand most DEI/diversity training and other initiatives is as a kind of rational astrology. https://www.interfluidity.com/v2/3513.html
So this is the key question here, IMO: How should outsiders distinguish between settled science and stuff that's still uncertain and ideology dressed as science and high-profile scientists speaking based on their politics/ideology rather than their science and institutions taking political positions and resting on their credibility as scientists and all that stuff?
Like, apparently serious people with PhDs and impressive-looking publication records tell me about both stereotype threat and regions of the brain being associated with particular mental functions. If I am not in a position to dig into the field myself and understand methods and evidence, how should I tell these guys apart?
For that matter, apparently serious people with PhDs and impressive-looking publication records tell me that IQ is racist pseudoscience, and also that IQ is one of the more solid areas of psychology and has a bunch of replicated results showing that it predicts work and school performance. I know enough to distinguish, but only because I have spent some time trying to understand the issue--otherwise, how should an outsider proceed?
For the public, few of whom are ever going to read a scientific paper or evaluate the statistical methods in some research study, what guidance can they have to distinguish real settled science from "settled science" claimed to win a political argument despite being very shaky or even just plain wrong?
“My framing emphasized that social priming studies had failed to replicate” actually replication crisis seems to be in fact that metanalysis protocol is non complete. It checks the correlation effects, while in many fields, especially in psychology effect could be modulation, moderation.
In example, if we have modulation effects, then trying to measure the meta consistent correlation effect would lead to replication crisis every time.
It is not the study bad, but the point of view is not exhaustive
"Instead, you can say the thing you mean - that woke partisans shouldn’t try to occupy and exploit the otherwise-noble science of psychology for their own political gain - and wage the same campaign with a halo rather than a comic-book-villain goatee."
This assumes that "just surgically cut out the bad tissue and leave everything else" is workable. Claim: it is not workable. Evidence: an entire legion of scientists, scholars, "lovers of truth", tried this in the late 2010s/early 2020s, and experienced the epistemic equivalent of ignominious slaughter with no remembrance.
Sometimes you need chemotherapy.
Where do you think the beta-amyloid research in connection to Alzheimer's falls in the spectrum of settled science vs novel research?
For those unaware, my understanding is as follows:
(1) in the 80s brain 'plaques' associated with Alzheimer's were found to be filled with amino-acids (beta-amyloid).
(2) throughout the 90s and early 2000s there was a lot of research into this connection, but it all basically led to basically nothing
(3) In 2006 there was a paper with a huge breakthrough that showed a particular beta-amyloid would cause memory issues when injected into mice. https://pubmed.ncbi.nlm.nih.gov/16541076/
(4) a huge effort follows for the next 15+ years where tons of federal research money and an enormous amount of pharmaceutical money is poured into beta-amyloid research (while not in this field, it has been described to me by peers as an era of "if you don't mention beta-amyloid, you won't get your Alzheimer's research funding".
(5) Since 2006, and partly because people struggled replicating the original results, other researchers began to raise allegations of image/data manipulation against newer work of the 2006 papers lead author. This eventually leads back to the 2006 paper where people find...it too contained fake data and image/data manipulation. The 2006 paper is eventually retracted
The original (now retracted) article from 2006 has a bunch of p<0.01, 0.001, etc. results. But maybe the meta-beta-amyloid hypothesis should have been thought of as nearing or going over p~0.05 once other studies failed to replicate? I.e. this always should be been considered novel and not settled.
I still have lots of questions about this era of work (i'm not an expert on any of these, just my curiosity), like:
(1) why did funding agencies so heavily prioritize amyloid-beta after seemingly one paper?
(2) why did drug companies rush to get beta-amyloid treatments out and spend so much money on this? Obviously they are profit driven by trying to be the first to do it and capture the market (this is bad imo), but did they not have their own labs trying to replicate the research?
(3) how did Aducanumab eventually get FDA approval? (this seems like a whole other controversy rabbit hole)
More reading:
https://www.science.org/content/blog-post/faked-beta-amyloid-data-what-does-it-mean
https://www.michiganmedicine.org/health-lab/everything-you-know-about-alzheimers-wrong
https://www.alzheimers.org.uk/what-we-do/researchers/news/explaining-amyloid-research-study-controversy
Implicit in Scott's description is an assumption of widespread good faith commitment to research and truth. Some bad apples, but they eventually get discovered and discredited. But this assumption seems unwarranted in a field that is (1) highly politically activist, (2) highly ideological, (3) almost entirely monoculture. The replicability crisis has hit many fields, but the most politicized and monocultural (like social psych) seem much more likely to lack the impetus to correct themselves. Once a field picks a team, hard, public faith in internal correction, updating, and paradigm shifts plummets, and justifiably so. See also most of academia, increasingly. Social psychology isn't singled out just for being particularly error-prone, it's singled out for being particularly activist, tribal, and incurious. Which also leads to errors, but the errors aren't the only issue.
Hasn't archaeology succumbed to the same fate?
I don't know specifically. Anthropology certainly has that reputation. History has gained more of that reputation as well, so it wouldn't be surprising if archaeology did too.
I enjoyed that Yudkowsky quote, and would now like a subscription to a magazine that only reports on scientific advances that are at least ten years old and haven't yet been disproved.
Most readers probably don't realise just how poor education research is. Your 90% bad, 10% good estimate is probably generous.
There is a growing movement trying to do better. The Education Endowment Foundation (EEF) in the UK is one example, commissioning more rigorous trials and producing evidence summaries for teachers.
The one paper that is most worth reading is Kirschner, Sweller and Clark's Why Minimal Guidance During Instruction Does Not Work. It is a devastating critique of a whole paradigm in education, built on a foundation of cognitive psychology rather than educational theory.
The broader Cognitive Load Theory literature is interesting because it sits at the intersection of cognitive science and education. I would be interested in where someone with a psychiatry background thinks it is strongest, weakest, or most likely to have been misunderstood.
'studies fail to replicate' is no crisis, it is merely science as it is normally done
'practitioners fail to update on studies failing to replicate' is a problem, and like other problems can rise to the level of crisis if it continues long enough
'media sensationalizes new study' is unfortunately just media-as-commerce, and probably inescapable in free societies
https://xkcd.com/2501/
I think you're overestimating how easy it is to know which of “Microaggressions makes minorities’ brains shrivel up like raisins” and “the hippocampal theta rhythm seems to be associated with both memory and pathfinding, suggesting an unexpected yet surprisingly compelling relationship between the two” is settled vs p = 0.049.
You frame it as detecting "woke" vs "not-woke", which I admit might be an easier distinction for a layperson to make (one of them has the keyword "minorities"), but it's not necessarily clear to a layperson that there's any correlation between wokeness and truthfulness, or wokeness and settledness.
So sure, perhaps “Microaggressions makes minorities’ brains shrivel up like raisins” is "woke", but is it "true"? Is it "settled"? A lay person would have no idea.
Nice clarification, Scott. Though you missed the most obvious «train of thought»:
The dumb woke psychology was wrong and bad, and woke people (or normies who blindly accept media interpretation of woke psychology because it’s «science») have terrible consequences for society, and most people don’t have either the expertise or the energy to rebut every woke study trumpeted by media/academia, therefore the rational response is a negative view of the field in general, until it proves itself capable of avoiding these bad societal externalities.
In item #7, you wrote "The exact substrate of memory is unclear but seems to involve new synaptic connections between neurons in at least some way", but we know much more than that. Long term memories in the hippocampus are well-modeled as a self-recurrent connectivity matrix in the CA3 region, directly analogous to a Hopfield network, an artificial neural network which implements long-term content-addressed memory.
For point 1 isn't the habenula also involved? It's the stick side of the equation
The APA advised the Supreme Court, based on the strongest scientific evidence, that adolescents are as mature as adults and can decide perfectly well to have an abortion; and, also, that adolescents are not as mature as adults, and therefore shouldn't be given the same criminal penalties as adults. Hodgson v Minnesota and Roper v Simmons. This isn't the worst example of corrupt APA policy influence, just the most legible.
The stakes of this debate are very high! The Supreme Court writes citations to APA amicus briefs into US law! And (although regarding the adolescent maturity issue, a few Justices called out the hypocrisy) they don't always seem to be able to tell the difference, despite how easy Scott thinks it is, between corrupt and not-corrupt psychology. Or they pretend so. There are some Justices who take the raisin brain hypothesis very seriously.
The American Psychological Association represents psychology as a whole. That's what they're for, that's their purpose. And they're corrupt. It's not that we can't distinguish between good and bad psychology; and it's not that it's useful to discredit psychology as a whole just to fight the bad psychologists. It's that psychology, as a whole, to the extent that such a thing exists, acts corruptly. The psychologists got together and formed an organization to represent themselves; that organization acts in a way that discredits them.
Your contrition is barely perceptible. I definitely chalk the original post to the 'debunks of decay narratives are always more wrong than the original claims' flavour.
> Memory divides somewhat fuzzily into bins of immediate-sensory, short-term, and long-term, with separate retrieval and storage functions, and separate systems for declarative vs. procedural learning. These can be harmed selectively, usually by hippocampal injury, so that some patients can remember their life stories but not form new long-term memories, or learn new skills like bike-riding but not new facts. The exact substrate of memory is unclear but seems to involve new synaptic connections between neurons in at least some way. <
I drowned at 13 and lost all my childhood memories. Except the emotional content. I can recognize people by how I feel when I look at them. I can re-watch a movie and know how I felt about it the first time, whether I liked it or not.
The specificity is pretty weird to experience.
You say there's suggestive evidence marijuana makes schizophrenia risk worse, but schizophrenia risk seems to be the same between pairs of identical twins where only one smokes weed. See doi 10.1037/abn0000867
I think that population-level correlations between marijuana use and schizophrenia are just confounding, and I think the procedures for controlling for confounders in a regression are rather sloppy in general, not just in studies of the mental health risks of marijuana use. But it seems your settled science list doesn't have other conclusions based on population-level correlations and this one is just there as "suggestive evidence" so I guess this doesn't undermine your conclusion.
My count for this "settled science" is 4 out of 10.
1) Reward functions and the nucleus accumbens are linked, but tieing this to predictive coding is still speculative. Low prediction error in itself is certainly not the reward, otherwise I wouldn't be very happy to see a Ferrari appear in my garage. More subtle connections need to be made (and to be fair are being made, but I would not call it settled). Also it's unsure if metaphors like "algorithm" really apply to the brain—there's no centralized, time-locked set of rules that govern the behavior and brain function may emerge from decentralized behavior of individual neurons. Brain as computer metaphors driving popularity of theories like predictive coding may be severely limited.
2) Psychedelic drugs do often act on those receptors, but again the prior/new evidence stuff is still speculative. There's no clear way to say that tripping people are more bottom up or top down. Jumbled, sure. But then it's unclear if predictive coding is a main role for these receptors or if alterations in prediction are concomitant /emergent with other brain changes– changes in regional connectivity or in emotional modulation for instance.
3) No to the Broca / Wernicke seperation. I've talked to a person who treats aphasic patients (speech difficulties) who says she purposefuly does not look at brain scans before treating patients because the conventional knowledge is often wrong. And if this seems anecdotal, there are studies of direct brain probes during awake brain surgery that indicate the same thing.
4) oof yes wow you put a number on nature / nurture, you figured it out you tricky devil. The field of study that relies on hunting out twins seperated at birth, and if you can't find these elusive creatures then settling for the next best thing (regular twins?) and sprinkling in some shaky statistics to untangle all the confounding variables. (I read your writeup of this and didn't see you address the possibility of feedback loops where small genetic difference may drastically change environment, leading us to overestimate the genetic effect)
5) Brain waves is mostly right to my understanding. Ofc it gets complicated the more you try to map out specific functions to rhythms.
6) GABA stuff seems mostly right. Inhibitory function.
7) What you say about memory seems right.
8) ? The SSRI meta-studies I've looked at often have it working no better than placebo. Modern psychiatry is plagued by division of probably heterogenous populations into fixed categories; driven by methodological errors like lack of proper observation and construct validity errors—for instance assuming that those with similar behaviors and similar answers on questionnaires share fundamental similarities in mental function, not to mention lived experience. (all of this exacerbated by looping effects, where diagnosis itself can shape behavior).
9) bipolar and schizo might be the same disease? Not sure where you're getting this. We don't have clear neural biomarkers for either disease. Maybe some genetic links? From my study of the inner experience of these diseases, they are not the same. For schizophrenia we find mostly low-level perceptual alterations, mostly visual (e.g. floating glasses), sometimes auditory (e.g. voices of the gods). For bipolar, it's unclear, perhaps a heterogenous group— sometimes multiple experience, or lack of clear emotion, or lack of clear mental content altogether. But usually not perceptual alterations/ hallucinations.
10. again with the algorithms. But yes, classical and operant conditioning, sure.
I'd say there is some settled pschychology, but mostly on the lower level: how sensory input maps from retina to occipital visual areas, how genes encode various neuroreceptors, how your ears transfer wavelengths to neural firing patterns, how neural polarity leads to firing, stuff like that.
The solid subfields are ones led by direct inductive observation. The shaky ones are those overinfluenced by theory and premature hypothesis testing. Especially where we don't understand the divisions we've made between various population groups (much of psychiatry), aren't properly controlling for multiple testing (most fMRI studies), assume that tasks in a lab environment mimic ecology and can induce certain predefined states, can't separate confounding variables (heritability studies), etc. And that's before we get to publication bias and other issues in an academic system that incentivizes many other things besides truth.
On a broader epistemic note, much of psychology now encourages large sample sizes to find averages. Its unclear if this is the best route for many problems in psychology, where individual variety may be more important and than such generalizations. How much would you learn from averaging every person's fingerprint? Why is this the default we apply to studying the mind?
Re. "Psychedelic drugs exert their effects through 5-HT2A receptors, pointing to a role for these receptors in some sort of construct like the balance between priors and new evidence."
That suggests that psychedelics disrupt a brain mechanism for perceiving the noumena behind phenomena. It seems reasonable to suppose that many different kinds of optical or neural noise create systematic, patterned aberrations, like rainbows, or chromatic aberration in microscopes. Psychedelics may, like Impressionist paintings, let us see something closer to the raw phenomena. They aren't providing deeper insight into reality; they're providing shallower views of it.
If so, this provides evidence for the argument that we don't need to worry about differences between noumena and phenomena because evolution guarantees that our brains work hard to eliminate any discrepancies of that kind which could possibly be relevant to us.
Aren’t almost all the “good settled science findings” neuroscience, rather than psychology? The only one I’d classify as psychology is the one about mental illness treatment and, arguably, the ones about heritability and schizophrenia
(this has probably been said, but still)
It seems to me that whenever this topic comes up scott mostly mentions examples on the "psychology is great" side that are neuroscience, pharmacology, and basically the more STEM leaning subfields of psychology.
All of these for example:
"Psychology is centrally ideas like brain waves, GABA receptors, reinforcement learning, twin studies, and connectionism"
are things I studied in a neurosceince bachelors, not a psych one. People belittling psychology mostly talk about the subfields that are in the difference set of "things you'd study in a psych degree" minus "thins you'd study on a neuro/cognitive science" degree... such as psychotherapy concepts, personality types, organizational psychology, social psychology, educational psychology and so on.
I'm pretty sure the precentage of psych critics that think there's no such thing as a broca's area is very, very low. But you can argue that that's neurosceince or some other title that's not what people mean then they say "psychology". On the other hand, the precentage of these people who think that a randomly chosen concept in personality studies is not well founded and kinda bogus is probably pretty high.
chatgpt suggested:
"One useful way to simplify this enormous list is to think of psychology as having roughly six big trunks:
Brain → biological psychology / neuropsychology
Mind → cognitive psychology
Individual → personality psychology
Other people → social psychology
Change over time → developmental psychology
Mental health → clinical/counseling psychology
Then fields like forensic, educational, organizational, health, sport, etc. are largely applications and combinations of those foundational areas."
I think its not that hard to claim that psychology is very scientific if you mainly stick to the Brain and Mind branches, the sceince there is basically chemistry and biology. People are mostly ok with those. The crux of the debate would be about the more conceptual later beanches.