"he found no effect. I take some of the blame for this"
What blame is there to take? The whole point is that no-effect results are super valuable! I suggest that the stigma of reporting no-effect results is a huge driver of whatever replication issues exist.
We need more people paid to replicate bold claims being proud of reporting no results, not fewer!
are you sure about this? I saw some people speculate that it was humour, but did Scott confirm it was a joke? like, it's kinda lousy, unintelligible humour if that's the case... it's played completely straight, in a non-fiction article, which is deliberately documenting real events, to make a real point... so was all that about it being the first big discussion with his wife also false? Like in a Louis CK skit you'd know something can be false, because it's in a context where things are made up for humour, but something coming from a mostly non-fiction author, in a non-fiction article, telling it as part of other events expected to be true, seems like really, really bad communication.
Same thing here... are we supposed to assume it's humour, because someone familiar with the topic would know it makes no sense to "take the blame" for this, and that it'd make no sense for Scott to think it was bad? Same as it makes no sense to convert to catholicism on the spot if seeing a miracle? Like I don't know, the beliefs in the rationalist community are out there enough that it seems likely that he'd just actually have nonsensical notions like converting to catholicism because of a miracle, because of some galaxy brain argument he thought up... and the readers would learn to spot these kinds of attempts at humour based on how consistent he was, but if he plays it completely straight like this, it's extremely likely a tonne of people will just take it at face value, it can even end up coming across as a bad faith attempt to get plausible deniability for his out there beliefs by claiming it was some weird attempt at humour.
And further, if it really is humour, then are we supposed to assume that what he claimed in his "My AI Opinions" article about the simulation argument could also be humour? it's a short section, he plays it straight as usual, but it's also something I'd consider nonsensical and bad epistemology to believe, but again, I know some rationalist beliefs are out there, so I assume it's meant in earnest. Having these weird attempts at completely straight humour in between just gives extremely mixed signals
I don't know, if these are genuinely meant as humour, I think it's bad form
He wrote he never jokes. And i am his true believer.
Also, there are several post with a *strong* Christian vibe. And a good friend got very Catholic. Also his history of being fascinated by ... I'd call it 'magical history BS' as Atlantis. (He learned diving to check out some underwater structures.)
Looking forward to pray a whole rosary with him. Ubi caritas et amor, deus ibi est.
"because someone familiar with the topic would know it makes no sense to "take the blame" for this, and that it'd make no sense for Scott to think it was bad? Same as it makes no sense to convert to catholicism on the spot if seeing a miracle? "
Yes, pretty much. You can assume that if Scott professes something completely stupid, like "blaming" himself for a replication failure, this is dry humor and he's counting on you to get that, UNLESS, the thing in question is a commonly held rationalist idea, like AI doom or something, which you may or may not consider stupid, but it is an exception to the above principle nonetheless. Maybe he was a little serious about Catholicism, but also overstating it for fun.
This post demonstrates why it is in fact important to be ambiguously humorous sometimes, especially if you suspect that people may care about what you think!
okay, why? hopefully not to laugh at the people that don't get it. I really don't see what it provides other than plausible deniability for the more idiotic claims made, and confusing people about what you actually believe. You can be witty without muddying the points you're making.
If whatever points are being made need thinking about, the listener should evaluate them for themselves, and in my opinion it shouldn't matter what the maker thinks.
Sure, if you were writing a chemistry textbook it would be out of line to invent new elements. But we're not talking about a textbook, just somebody describing their thoughts in the moment.
Is it? Mea culpa if so. The rest of the paragraph read like an explanation of why Scott chose to give that grant in spite of all the red flags that in retrospect should have made it obvious the result would be no-effect; hence my response of no, please, hindsight is always 20/20, we need more grants/research like this, not less. But maybe I misinterpreted.
We need more "replication research" sure, but if all we're doing is doing the "sexy" replication research, there's going to be a lot of people with head trauma (or maybe that's just German Sleep Researchers...).
[Yes, this is in jest. A slap to the head is part of the protocol!]
If I remember correctly the original study had some flaws that were obvious in retrospect*, so the no-effect results could have been predictable without spending money on the study.
(*"Obvious in retrospect" is rarely obvious, and I agree that no-effect results are valuable, and admitting that the study was unneeded in retrospect was also holding themselves to a very high standard for science, which I appreciate).
Well, it's valuable to the extent that the study would have been taken seriously otherwise. It can be a waste of time if there were obvious flaws to begin with. (Edit: or even non-obvious flaws that are nevertheless disqualifying and are deductible from their data or methods)
That's true, but unfortunately bad papers still get through. And the likelihood that a paper that gets published was actually rigorous is some function of both the journal's impact factor and its tendency to weigh quality over flashiness of the results.
To be cynical, the publication review process is for shaping your papers to flatter the reviewers, and possibly for checking to see if your English is parseable by a non-expert (don't laugh, it's quite possible to be able to math perfectly well, but not English good).
Cynic notes that large scale submission of entirely faked AI papers is ongoing, and largely undetected. Peer Review is just as dead as it's ever been (Millikan's oil drop experiment was peer reviewed, wasn't it? Implausible research continues to get published to this day.)
It's fair but I know the guy and Sasha himself said basically that the replication could have been avoided because a deeper read of the studies ended up leaving him with the conclusion that the whole thing was shoddy enough it wasn't even really worth bothering replicating. You have to pick what to use resources on and if a study claims something already unexpected and was also conducted in a clearly poor manner the chances of it being bunk skyrocket.
Like, we do we even give creed to studies that have not been replicated yet? Any study that has zero replications so far should just be considered proivisonal and not even worth reporting on. Or maybe we should always use the word "allegedly" like you do in a court until it's been proved (by an independt second or third study).
"New study allegedly shows X, but is yet to be confirmed independently."
If Psychology held itself to this standard I'd be onboard calling it a science.
I would add two things pointing in different directions.
1. fMRI experiments in psych are also pretty bad on average
2. Having spent a lot of time mucking around in machine learning lately, I am not convinced social psychology is uniquely bad. Much of biology likewise. Parts of ML strike me as closer to actively dishonest.
I propose the conjecture that most research is just flawed in general (or at least should be suspected by default) because it's hard to get things right even if you're competent and professional, and there are silent failures that won't yell at you or your reviewers "hey, this invalidates the whole thing" etc, and because the existing filters for good vs. bad science are just not selective enough.
I know this is a crude first approximation and fields are likely different in that respect etc
Saying “most research is flawed” sounds like a damning judgment, but on reflection it’s probably exactly right - most of everything is flawed, even the best things. Not so flawed as to be crap, but it has flaws.
It doesn't matter how you excuse it. The question is: is everything mostly fine except that one subfield of psychology, or is everything "flawed" enough to be riddled with false/unreplicable results.
My bet is on the latter: most fields, perhaps all fields, are have plenty of results with subtle flaws that invalidate some important points/takeaways of the paper. And it's mostly not due to lack of competence and diligence on behalf of the authors: it's just that the process is really difficult and the filters are inadequate, so it's the business of the entire community to double- and triple-check each other's results, but attention to replication is a scarce resource and some results don't get it.
Lots of ML people don't know their stats, so they publish results that are just noise (eg a 1.1 pp improvement on a benchmark with ~50-100 questions) or they aren't rigorous about giving their "baseline" comparator a fair shot -- e.g. they tune hyperparameters for Our New Genius Method but don't tune them for Boring Baseline Method.
ML is also not really peer-reviewed so it does suffer more from oversights - my sense is that more of the knowledge about "oh the stuff from this paper or this group is mostly bunk" just lives in the unwritten knowledge within the field to a greater extent than other fields of science.
There's also just a huge amount of ML research, so per Sturgeon's law, there's a lot of bad ML research out there.
This is sort of a side effect of computer scientists presenting stuff at conferences and moving on, leaving the conference proceedings as the closest thing to a written record of what they did, without ever bothering to make something good enough to get into a journal.
Because you can just post a paper to arXiv whether or not it's accepted at a conference. Preprint servers carry a lot more weight in ML than in other disciplines.
My impressions are somewhat different -- that preprint servers play a significant role in many disciplines now -- but I don't see an easy way to resolve it.
When you write papers that are "known" lies, and do it over and over again, including paying people to write MORE lies (by faking research)... this is NOT progress.
Well, I guess one measure would be how often researchers in a field display a reputation for honesty that earns them the dreaded "You should run for public office." epithet. :-)
The public-facing part is a real difficulty. Psych has a problem that "harder" sciences do not, which is that you can fairly easily do shoddy work that generates widespread public interest.
Economics has a similar problem, but they have a much more combative culture and will gladly fight each other in the public eye. Psych has a very agreeable, very fragmented "let a thousand flowers bloom" type culture, so dubious public-facing communication goes largely unchallenged.
Well, sure, if you count the disciplines where the research gets cancelled by political pressure. (This isn't physics/chemistry and mostly not biology either). Psychology is about what you'd expect, given that post-World War II Western Society is built on a bunch of lies. Many lies are promulgated in "Psychology 101" -- some of them quite famous experiments...
There are a few fields where "burn it all to the ground" might be a good idea.
Marketing psychology's the obvious "the government should not only not fund distributing this, but should actively sin-tax it" where the field is of negative societal value.
There are other fields where the field is conceivably of positive value but could arguably benefit from tabula rasa - the history looked at by "grievance studies" is potentially interesting, but the people currently running and manning those programs are mostly useless for that and outright worse than useless if they're allowed to select their replacements (and many of them will not co-operate with training replacements they do *not* approve of), so it's plausibly better to just declare all credentials in the area void and try to rebuild that wing of academia from scratch in 20 years or so.
I know that ML academics can grind out low-quality/questionable papers, and sometimes they will massage the conclusions to make their results look deceptively "SOTA" (some people rightly push back against this "everything's SOTA" standard of publication), but ML research has an incredibly powerful test in the industrial applications. It's much harder to fake that your algorithm is working IRL and doing what the business needs. (Not impossible, of course, but if my models aren't providing anything useful, my bosses will notice.)
If your models are telling them what they want to hear and help them justify doing what's in the interest of their own careers (whether or not it's in the interest of the shareholders), it might not matter whether or not the models are accurate.
When I was doing this at big corps and for financial companies, we very much measured the financial impact of the models our teams built, and this was communicated regularly and at high levels, and audited a couple of ways.
I feel like if it grounds out in actual revenue and / or profit, you can't be faking that much.
I think you can’t blame people for not making the distinction. Maybe you can blame the media for sharing the flawed attention-grabbing studies.
This “social priming“/social psychology/sociology subfield spread misinformation like no other field of academia and probably influenced politics to a significant degree. There are still midwits in 2026 citing the Stanford prison experiment as if this is a reliable result.
Those other “fine“ subfields of psychology just seem so minor in comparison.
Also many people are distrusting academia as a whole, not just psychology.
This is a good point. Social Psychology may have been 0.5% of all psychology papers but it had a massively outsized effect on the politics of the Second Social Justice Era (2012-2022) being the veneer of scientific respectability for many things.
The rest of psychology should be enormously angry at the few researchers who delighted in the attention and enthusiastically rode the tiger of feeling important in politics.
No different from how some small fraction of climate scientists abused their technical credibility to say that science says we need to do rapid decarbonization via X, Y pathways. It’s heady stuff to be on the national morning shows. But it undermines the entire field.
> There are still midwits in 2026 citing the Stanford prison experiment as if this is a reliable result.
The problem with the Stanford Prison Experiment is not that it fails to replicate, it's just that it was never clear what it was supposed to show. The main flaws aren't things that we only found out later, the flaws were on the page from day one.
What's the supposed conclusion of the Stanford Prison Experiment? That if you get a bunch of undergraduates and manipulate them into acting like dicks then they'll act like dicks? We knew that already! We have lots of evidence of people acting like dicks.
There's nothing in the experimental design that can tell us anything about why people act like dicks, except that they can at least sometimes be manipulated into acting like dicks under one particular set of circumstances.
The supposed conclusion of the Stanford prison experiment, as it was sold to the public, was that if you give ordinary people a bit of authority and freedom, they quickly turn into evil tyrants.
And obviously it was fraudulent since it was closer to some roleplay session organized by the experimenters since they were much more “hands-on“ than they initially let the public know.
Well, people can be manipulated to act like dicks is an important result for me, as the other hypothesis, "some fraction of people are dicks, no matter the influence" seems to be the default assumption.
Gino and Ariely were both professors in business schools, and I don't know what their standing was in psychology proper. I would have liked to see the interdisciplinary aspect teased out a bit more - I think there's a lot of bad interdisciplinary research, though you hear less emphasis of it now than 10 years ago.
When I taught a 'research fundamentals' course to business students, one of their projects was to analyse the original 'power pose' study. Most of them could see through it right away without any particular training in physiology or statistics. I primed them, of course. But that could not have gone viral without the complicity of the media.
I half agree. I think the emphasis on the heterogeneity of the discipline and the likely risks of extrapolating from one subfield. My sense is that the problem was, although maybe particularly bad in social priming, likely due to much broader norms around experimental practice, which likely apply more widely (indeed, not just in psychology).
Another point you didn’t make that I think is important here - social psychologists (not all of them obviously, but as a field) deserve a lot of credit for facing up to these issues and working to change some of these norms and practices. I would be more confident in the field now than I would be in some adjacent disciplines where no such reckoning occurred.
That jumped out at me, too. I have long been annoyed at "MTBI bad, Big 5 good" discourse on the grounds that two of the Big 5 dimensions are fairly well correlated with MTBI dimensions, and two more correlate enough to suggest they're at least sort of measuring the same thing: introversion/extraversion is about the same in both (R^2 = 0.72), Openness is sensing/intuition, (0.77). Agreeableness correlates somewhat with thinking/feeling (-0.44) and consciousness somewhat with judging/perceiving (-0.49). Those correlations seem strong enough that if one of them is worthless nonsense then the other one probably is, too. The anti-MTBI discourse probably has a point that reporting the scores as a binary type on each axis provides a lossy signal compared to reporting the raw scores, but the scales themselves are either both nonsense or both have at least something to then even if they're different enough that one might be more precise or more applicable than the other. Or they might both be measuring different real.things that are proxies for one another, like how height and weight are moderately well correlated.
One thing that strikes me that I haven't heard talked about much is that Myers Briggs frames the dimensions in a relatively value-neutral way while Big 5 dimensions aref framed more to be good or bad. Especially neuroticism and consciousness.
The means matter as well as the ends in research: MBTI was developed by a mother-and-daughter team based on Jung, if I remember right, while big 5 used a regression on data collected (probably from first year psych undergrads). They're certainly measuring something, as is IQ. It's not clear that it's well-defined with cardinal directions. It's probably more like a surface in a high-dimensional space which can be projected down and measured in lots of ways and none is really much better or worse than the others.
Personally, I'm of the opinion that tools like this (when I'm provocative I list them as Big 5, astrology, Myers-Briggs and Harry Potter's sorting hat) can be useful for self reflection and analysis but they're not 'scientific', whatever that means.
Women seem more likely to sympathize with thieves and other "bad people" than men do (at least as seen in the fMRI studies) -- men see thieves and bad people getting punished as "karma" and women just get upset when anyone gets punished, regardless of whether or not they deserve it.
I think this is a good correlative study to "thinking" versus "feeling" (which are only going to be fun if you want to say "women don't think, they feel arguments, and make decisions based on said feelings." I've been talking to advertisers again, haven't i?)
Big Five is an empirical finding, not a hypothesis or theory, and so the very idea that it would be falsifiable is kind of weird and missing the point. You can't falsify a finding: you can fail to replicate it, fail to find it useful, or find exceptions and deviations which lead you to a better model. But to even think that falsification is the goal means you have misunderstood the nature of what researchers are doing.
There are many different theories people have proposed to explain why these factors continue to pop up, and none have gained consensus. One such hypothesis is that the traits are somehow "real." Another hypothesis is that they are just one particular way of representing the data. Currently most researchers lean towards the latter view, and that is also where I would put my money. But even for those who claim they are real, none are claiming the Big Five are exhaustive for what personality is. They are merely claiming it is a pragmatic tool.
Having misunderstood this, Banana goes on to cherry pick studies to critique the model. For example, I've never once heard "inspiration" discussed in the context of the Big Five, and yet he is treating it as if it were representative. But more representative is things like educational achievement, income, number of friends, how often you smile, mental illness, sensitivity to disgust. These are not nearly as vague as "inspiration." Meanwhile, other behaviors are cultural or political (mask wearing), and yet he ignores that Big Five would actually predict a negative correlation between mask wearing and conscientiousness for rightwing individuals. He also has a weirdly unempirical critique of surveys where he thinks they have no correlation to behavior, which is not a critique that even critics of the Big Five take seriously.
He also claims ALL the correlations are trivial, which is simply not true. Compare Big Five to any lay personality test, and you will find large differences; artistic appreciation is not correlated with introversion but with intelligence, there is no "fiery" personality type and the whole idea of types has little scientific validity, happiness is not the opposite of neuroticism but instead better correlated with extroversion, unstable emotions and sadness correlate very highly, motivation and disgust correlate highly, highly agreeable people lie a lot. These are genuine findings, and not intuitive. Even the correlations which are intuitive to people are often downstream of psychology popularizing the correlation.
The best part of his essay is his criticism of how general the factors are (though his point about non-weird populations seems overstated). But it is important to note that his points are well known limitations of the Big Five mode made by Big Five researchers themselves, not just their critics. Just because a model has limitations does not mean it is not useful. Here is a good summary of similar points: https://blog.arturnilsson.se/2022/09/01/five-myths-about-the-big-five-personality-model/
And it's not like I am a big defender of Big Five! I vastly prefer Mischel and Shoda's CAPS Theory to Big Five because of issues around the limitations or surveys, and the lack of theoretical foundation for the five traits. Typically, I am the core audience for critiques like this. But Banana's critiques are just too bad to take seriously. He claims Big Five is simultaneously trivial while also not correlating to anything in the real world, which is contradictory on its face. But more importantly, he seems to genuinely not understand the very thing he is trying to critique, the whole essay stenches of a hit job, not a genuine review of the strongest evidence.
The psychology replication crisis did much more to me. Before, “science” was on this pedestal, similar to how HPMOR did it. A way to find truth. Something to be awed by. A path toward moving humanity forward.
Unfortunately, while I knew that was an idealized version, I didn’t realize how strongly the industry of scientific research tarnished that vision. Now, post replication crisis, even when “science suggests something” (covid opinions, flu vaccine suggestions, responses to spiking AQI in the Midwest, etc) I am immediately suspicious.
While engineering in the world of capitalism, or pharmaceutical companies may use science to develop something new, I’m no longer optimistic about any technology or knowledge to be obtained from universities or government funded research.
I’d love a few bullet points of government funded/university funded research that has been released in the last 5 years that isn’t just a headline. As far as I can tell, that large expenditure is vaguely worthless as the low hanging fruit of research has mostly been completed and if it were valuable for humanity, humans would pay (capitalist companies) to do it.
To be clear, the replication crisis on only social priming psychology shined light on the POTENTIAL fallibility of all other research. One bad apple. So while you might suggest it’s only 20% of psychology effected, I would argue that all research went from 90% accurate in my head-model to 60% accurate.
>In theory, science should get credibility because of its ability to discard former mistakes.
In the abstract, yes.
In the particular, it's difficult to tell whether any individual theory is correct or one of these mistakes which will be discarded later on, which is unfortunate because most "science-based" policy is based on individual theories, not on "science" as a whole.
Stolen credibility, in the case of psychology. An overall low-quality field with untrustworthy results that are not gatekept. And, unless I missed it, Scott didn't even mention its possibly best proven subfield, IQ. Amusingly, this is instead violently rejected by many psychologists. So much for truth seeking.
That is not to say only psychology is guilty of plentiful trash results. The big replication crisis was in pharma and related, as I recall. Beyond that, many such cases!
Do research psychologists, by and large, disapprove of IQ research? I had the impression that psychologists themselves were mostly favourable to it, and that most of the fundamental criticisms come from outside the field. I don't have good data on this, though.
IIUC, they accept IQ in the abstract, while acknowledging that every IQ test is culture-bound. So they're probably good with a properly standardized group of test takers.
Actual research psychologists note wryly that we really don't measure IQ except in children, so we miss the decay of IQ as people grow older and their brains become less cognitively flexible.
I think the sexy promise of science is to have something theories that's generally true but is wrong on the margins, and as we improve observations we can shrink those margins (even if it actually required massive foundational change in the background). This is how it works for physics. But this is certainly not true in something like nutritional science which keeps flipflopping between whether chocolate, wine, fat being good or not.
In other words, science theories should be converging. But some of them seem to be stagnant or even diverging.
Cocoa has tons of antioxidants. A friend of mine just got out of a single-blind single-participant study on cocoa's effects on bone regrowth.
Cocoa is very good for you. Too much fat is bad for you. These are easy things to prove. Wine studies are run by wine sellers, and do not show anywhere near the amount of antioxidants that cocoa has (cocoa is a serious outlier, compared to most other fruits and vegetables).
So, given this, psychology should really be among the top fields in replicating and verifying results?
Current Science as a social construct does not reward such activities much, as is well-known. This is of course a problem, for example when the field has a lot of trash results. If we assume the 20% bad value, every journal issue should include 1-2 retractions of previous contributions and there should in general be a decent wait before proclaiming a result accepted.
Ideally, failed replications would be published in the same journals as the original studies, and they would then subsequently be cited more than the original study. Unfortunately, reality is not like that.
There are thousands of research papers that psychologists do every single year, and none of them are getting written up because the psychologists don't have the time to write them (or analyze them, honestly).
"As far as I can tell, that large expenditure is vaguely worthless as the low hanging fruit of research has mostly been completed and if it were valuable for humanity, humans would pay (capitalist companies) to do it."
There's still a lot of low hanging fruit in rare-diseases.
Pharma companies fundamentally don't care about any disease that only affects a few hundreds families around the world unless one of the families is the royals of europe or similar.
But there's a lot of nasty diseases like that. it's a long long tail.
I've seen people go all the way from discovery to miracle-cure.
A few folks from the Clinic for Special Children, world's experts on genetic conditions common in Amish and Mennonite populations, have launched an effort in exactly that space. Got the gene therapy for MSUD working in cows and mice; humans next, dear FDA? But market is too small for pharma companies to be interested in carrying that ball across the finish line. Worldwide, ~700 babies a year are born with the condition. Therefore:
Capitalism is bad at doing long time horizon research. If we are just a little bit more research away from something valuable, and we might have a product in 5 years, capitalist companies can do that. (But often the insight comes from universities anyway)
The longer term and more abstract something is, the harder it is for any one company to get the benefit.
Pharma companies have longer horizons than that, haven't they? Something like 10-15 years from patent to medicine, I believe, with many failures.
And if we take our favorite field of AI, private venture capital has funded it munificently for at least 15 years now, so much that the public is getting scared and shouting about bubbles on a daily basis.
> I’d love a few bullet points of government funded/university funded research that has been released in the last 5 years that isn’t just a headline.
The GLP 1 drugs (Ozempic and similar) were developed based on government-funded research into the underlying biology. While that foundational research was generally done more than 5 years ago, the practical applications (medicines) from the last 5 years wouldn't have happened without it. A lot of government funded research is foundational with like that that takes a long time to reach practical applications, so it's too early to tell which recent government funded research will pay off.
> As far as I can tell, that large expenditure is vaguely worthless as the low hanging fruit of research has mostly been completed and if it were valuable for humanity, humans would pay (capitalist companies) to do it.
This only works for research with a clear path to monetization. Clinical trials are expensive, so companies will only fund them if they have the patent on the drug. For supplements and off-patent drugs, that incentive isn't there, so the research on which supplements are effective for what and new uses for old drugs mostly doesn't happen.
I've seen hundreds of million of dollars go past me to things that will never, ever, ever, benefit a single human. Only shareholders. Of the things I've seen that will benefit humans, they were mostly sidelined in favor of things that were easier to sell.
Remember: the thing the market loves the most is whatever is the most like opium. Cheap to make, easy to sell, Feels good the first time, and you live long enough to have all of your value extracted before dying. If a company does something other than try to become more opium like, they are either not traded or having a gun held on them by the state or being irresponsible in the fiduciary sense.
Something released in the last five years can’t be known to be much more than a headline. It takes more years than that for there to be enough progress in the noise of research to become confident that something is a really productive new direction.
One of the few things I can think of in the past few decades that was obviously a big deal in less than five years was CRISPR. Another was the breakthroughs in LIDAR-aided archeology in South America. There are probably a few others in some other fields.
But in general, the present cutting edge shouldn’t be what anyone outside the field is paying attention to - instead you should be paying attention to the building consensus based on the work of a decade or two ago.
It sounds a bit like you're "overswinging" the other way. It makes sense to be suspicious, but the conclusion "I’m no longer optimistic about any technology or knowledge to be obtained from universities or government funded research" seems like you're over-updating based on the data.
Science has always been imperfect. Philosophy of science has been mapping all these ways for decades, but also e.g. something like e.g. Ioannidis, J. P. (2005). Why most published research findings are false. PLoS medicine, 2(8), e124. (very influential, cited 15k+ times) shows how imperfect the whole thing can be. But imperfect =/= useless.
When it comes to research becoming valuable the term you're looking for is "academic spin-offs" - companies started out of academic research. See here for a report on them: https://content.dealroom.co/uploaded/2025/11/European-Spinouts-Report-2025.pdf?x94554 it's a market of 100s of billions which wouldn't exist without academic research.
Lastly, your statement "if it were valuable for humanity, humans would pay (capitalist companies) to do it." --> this isn't true. Companies will not take the risks that universities can. The famous story here is that e.g. a smartphone is filled to the brim with technology that would be impossible without fundamental research funded by governments (GPS, touch screen, the internet, etc.). Business is better at building applications than fundamental innovations/research.
The replication crisis is alive and well far outside of psychology. I work at an immunology CRO (animal models). Less than 20% of the results we get asked to replicate by clients (results from a single paper) actually replicate. Even after multiple attempts. (The number may be more like 5% - I haven't looked in a while, I'm trying to report a conservative percentage.)
We warn clients about that - sometimes they want to try anyway.
We are very good at this, and this is consistent with what other labs see. The problem is p-hacking and related bad scientific practice.
OTOH, if a result has been published as replicated, even once, usually it replicates when we try it. Results from some labs have MUCH better chance of replication than average (obviously good labs with careful practices; they are rare).
Just because something is wrong doesn’t mean it’s bad science! The point of science is to be wrong over and over again in ways that correct. 20% replication rate is still better than just choosing an idea that seems reasonable and testing it, even if it’s not nearly as good as you would like. But insisting on enough rounds of replication before even the first publication would slow down the sharing of knowledge.
Having worked and collaborated in multiple very different areas of science, I'm convinced the majority of science is essentially fraudulent. Maybe 70-90% of it.
Of that bad work, maybe 97% of the problem is insufficient rigor which is, in fact totally discernable beforehand, and 3% is outright dishonesty. I think this also varies enormously by field, psychology being one of the worst offenders that still qualifies as science. The variability by lab, institution, subfield, etc. is, I think what would most likely invalidate these estimates of mine, because one can only ever see a small corner of the world.
I'm sort of surprised by your responses in today's comments. You're a very smart guy, and philosophy of science is something you are professionally expert in, if I'm not mistaken. Do you mind if I ask how much experience scrutinizing actual scientists' work you have, from a correctness of empirical work perspective? I sort of expect the discrepancies between our opinions of science is based on in-the-weeds level, seeing-the-sausage-being-made impressions, so I'm curious.
In case it wasn't clear, as a professional scientist, I consider science to be in generally terrible shape, although of course I love the ideal.
I agree - the problem is mostly insufficient rigor (>90%), not outright dishonesty.
My experience is that the people who publish irreproducible results really believe that if they poke and prod at their data enough, they can turn noise into signal. They don't know they're p-hacking. They're just not smart enough to understand.
Which leads to my other hobbyhorse - we have too many scientists, esp. in biology. Being a good scientist takes a lot of intelligence. And intelligent people have lots of other, often more lucrative, fields to go into - finance, engineering, etc. There aren't enough smart-enough people to go around. But NIH and NSF funding ensures that lots of people who shouldn't really make the cut become scientists. I think this is why the *average* quality of science was higher before WW2 (before the funding). The total output is today much greater, and probably more *good* science is done now than then. But there's an awful lot more chaff to dig thru to find the wheat.
I believe what you're saying to be a problem. What would be a potential solution then? What about an increased knowledge of statistics and accountability for the "noise" in a lot of the research? But then it begs the question, who would pay for the accountability anyway? Lots to think about there. I'm not even sure if there is a legitimate, clear cut solution. I've heard a myriad of potential options put out in chemistry and biomedical science trade journals.
How does the NSF ensure that lots of people who don't make the cut become scientists? I know that universities and professors themselves are engaging in some sort of grade inflation and passing people on to the next course independently on the NSF, to my knowledge. Maybe the higher GPA and professors mentoring students to go on to the next level of education is the role the NSF plays.
Re solution, the best I've been able to come up with (been thinking about this for 15 years or so) are ways to make the reproducibility of published results affect authors' careers. One idea is a website where scientists can exchange info about published results and results of attempted replications, with careful tracking of credit/demerit. (I have a 25 page writeup from 2014).
Re how NSF (and even more NIH) funding does harm - by pumping money into the system without regard to how much money can usefully be absorbed. There are only so many good people (good in the sense of ethical and honest as well as intelligent) who want to do science/biology. If you pump more money into the system than needed to keep those people busy, you have to lower the bar for who gets in, to use up the money. This happens. The result is not just wasted money (that too tho) but the generation of *noise* which buries the signal of good scientific results. At some point more funding makes progress go slower, not faster.
For your first paragraph, isn't there already bioRxiv and the like? Granted arXiv is much more widely used in the physical sciences that don't overlap with biology (i.e. not so much for biophysics, biochem). Unless you mean registering clinical trials and the results, which is not such a bad idea.
W.r.t. to the second paragraph, how would we determine the ethical, honest, and intelligent people before they enter graduate programs? I guess programs already try to filter out based on GPA and letters of recommendation, but I'm not sure about moral character.
Maybe you're missing my point. The problem is not that most published results need 5 replication attempts (1 in 5 = 20%) before they replicate. It's that 80% of published results don't replicate at all no matter how many times you try, because the claimed effect is not real. If the authors had tried even a little to verify their own results they would have found that out, and not published.
But they didn't. This seems (here I'm offering the standard speculation) to be often because they know very well that most apparent new results aren't real, and that if they tried to replicate it, most of the time they wouldn't have a publishable result anymore. Grad students are routinely taught (in bad labs; i.e. most of them) to publish rather than replicate, lest they have nothing to publish.
There's every incentive to publish without verification, and every disincentive to scrutinize "interesting" results carefully for flaws before publishing. The result is that other labs waste astounding amounts of time and money trying to replicate spurious results. Interestingly, the probability of published results from commercial labs (pharma, etc.) replicating seems to be far above the average for academic labs - probably because they have less "publish or perish" pressure.
A related problem is p-hacking; you can Google on that.
Good labs (the few) don't publish until they're pretty sure they have something real.
I got the point that 80% of published claims are false. But you’re wrong to then say “if the authors had even tried a little to verify their own results they would have found that out”. Sometimes the experiment is so expensive and difficult that they don’t have funding to do another instance of it. And even if they did, sometimes the issue is some unrepresentative feature of the population they’re studying, or the equipment they have. It just isn’t valuable to have the first group do its own replication - you want it published so that someone else can do the replication, if the result matters to anyone (which it might not for a while, so it’s good to have it in the record for that future person).
What I said in a different thread is that there’s 1000 possible results that might be true, but only 50 of them work out once well enough to be published, and out of those only 10 are real enough to replicate. Publishing those first 50 is what you want, because insisting that nothing be published until it has that level of certainty means that everyone else has that much less to work on (because these 10 results aren’t available until years later, and the other 40 that don’t replicate often worked the first time because they’re adjacent to something real.)
Commercial labs have higher hit rates because they’re not working at the frontier of knowledge - they’re turning moderately well-understood things into commercializable products.
Most labs don’t publish things until they’re pretty sure it’s real, but that’s because you have to convince yourself to ignore the broader statistics to not get discouraged.
I think you're misreading the costs and benefits involved. My experience, as stated, is in rodent immunology work. Our lab runs hundreds of studies every year. We have an excellent idea of when a result is solid and when it's probably noise. So, I assume, does any experienced scientist in a competent academic lab (we have lots of people who came from such labs working for us). When an academic lab nonetheless publishes results they should know very well is unlikely to be correct, they waste vast time and effort in other labs attempting to replicate it. If you have a result you think is promising, I think it's your duty to check it out enough to at least think it's *probably* correct before publishing. But in fact that's rare, and it makes science incredibly inefficient. Yes, it takes some time and money, but it avoids wasting even more elsewhere.
I think you're defending psychology from the wrong direction. The replication crisis paper (https://www.science.org/doi/10.1126/science.aac4716) really only tested social psychology. But like you say, similar (also modern) replication attempts, even in other fields, e.g. biology, seem to have equally dire results, or worse. So psychology is not necessarily in worse shape than the rest of science, it just happened to have an unusually well done, broad replication attempt, and it was in social psychology.
I don't think the correct conclusion though is that we should trust most psychology experiments, we should just be skeptical of all non replicated work, until this problem is more broadly solved in science.
I'm actually not aware of a broad replication attempt that didn't find massive failure to replicate, though last I looked deeply into this was two years back.
So I don't think your implication that an introductory psychology textbook is broadly correct, is true. We can't say that just because of an absence of evidence on replication, when the best available base rate is really bad
But it is a credit to psychology that it is the field where such an impressively methodological and broad replication attempt was done
> I'm actually not aware of a broad replication attempt that *didn't* find massive failure to replicate, though last I looked deeply into this was two years back.
A new paper currently available as a preprint at PsyArXiv has tested [...] nine key findings from cognitive psychology, related to perception, memory and learning. [...] all the effects in question were replicated on all occasions [...] The nine cognitive effects that showed this robustness were:
* The Simon Effect: Participants had to press computer keys to indicate as fast as possible the colour of a on-screen circle. They were faster to respond if the appropriate keyboard key was on the same side of space as the circle they were responding to. This facilitatory effect of "stimulus-response" consistency is important for the design of human-machine interfaces.
* The Flanker Task: Participants had to press the correct keyboard key as fast as possible to indicate whether a target stimulus was a vowel or consonant. Participants were faster to respond if the target was surrounded by distracting letters associated with the same response (e.g. a target vowel surrounded by irrelevant, distracting vowels), as opposed to being surrounded by distractors associated with a different response (e.g. a vowel surrounded by consonants). The task shows how we can't help but process irrelevant information to a certain degree.
* Motor Priming: Participants had to press the appropriate keyboard key as fast as possible in response to left- or right-facing arrows flashed on-screen. Preceding arrows (known as a prime) gave advance warning of which way the target arrows would point: sometimes these primes were accurate, which led to faster performance, as you'd expect; if the primes pointed the wrong way, they slowed performance. Crucially, some of the primes were "masked" to make them subliminal (i.e. not consciously visible), in which case the effects were reversed, with primes pointing the wrong way leading to faster responses. The finding shows how information that's not consciously perceived can affect our behaviour, and that it can have an opposite effect when subliminal than when consciously perceived.
* Spacing Effect: Participants were presented with dozens of words in sequence, with some of the words shown more than once in quick succession (i.e. the repetitions appeared close together in the sequence, known as "massing"); the repetition of other words was spread out in the sequence with other words appearing in-between (known as "spacing"). Tested on their memory for the words later, participants showed superior memory for the spaced words than the massed words, a basic phenomenon of memory that has important implications for learning and study.
* False Memories: Participants were shown sequences of words of related meaning. Tested on their memory of the words later, participants were more likely to mistakenly say that a new word of similar meaning had been present in the earlier sequence than a new word with a meaning unrelated to the earlier list. This is a basic demonstration of the fallibility of memory and how easy it is to feel like we've experienced something before when we haven't.
* Serial Position Effect: Participants were challenged to remember lists of twenty words. They later showed better memory for words that appeared near the beginning and end of the lists.
* Associative Priming: Participants had to indicate as fast as possible whether target words flashed on-screen were real words or non-words. They were faster to identify target real words if they'd been preceded by a word (a "prime") of related meaning, consistent with the idea that activation spreads through networks of related information in the mind.
* Repetition Priming: Presented with lists of words, some more common than others, participants had to indicate as fast as possible whether each word was a real word or a non-word. Some lists were repeated. Participants were faster to make their judgments when re-tested on a word shown earlier, and this priming effect was more pronounced for less common words – a finding that shows an interesting interaction between short-term memory (based on a recent encounter of a word) and long-term memory (based on how common the word is).
* Shape simulation: Participants were shown successive pairs of sentences and pictures. For each picture they had to say whether the object it presented had featured in the preceding sentence. Participants were quicker to respond correctly to picture-sentence matches when the orientation of the object was similar in both cases – for example, if the sentence "The ranger saw the eagle in the sky" was followed by an image of a flying eagle as opposed to a perched eagle. The finding shows how we automatically represent the visual images implied by sentences we've read.
Any field of science where (a) one result is more exciting than the other and (b) data points are hard enough to come by that you need to worry about p-values, is probably problematic.
This is not all of science. If you find yourself measuring the genitals of moray eels then your statistics are probably robust. If you study the mechanical properties of jamming in grain silos then you're probably okay. The vast majority of science is fine because it doesn't deal with these sorts of issues.
But if you study whether x is related to y, and it would be much more interesting if it is than if it isn't, and if gathering each data point is expensive so you don't have a lot of them, and the trend you're looking for isn't incredibly strong, then you are in serious danger of doing lousy science. At best you have a file-drawer effect, and at worst you're tempted to massage your data in whatever way it takes to get an interesting rather than boring paper out of your very expensive experiment.
> That's a very good summary of the sorts of research that are more and less reliable.
I feel like Melvin's forte is precisely this - seeing to the crux or heart or dominant failure mode of something, and making a pithy comment highlighting it or summarizing it. He's high on my "ACX commenters to search for" list for precisely this reason.
The shame is also that the media and/or the rest of the academy don't properly communicate how unlikely it is.
In contrast, when there was the possibility that the EM drive worked (or the superluminal neutrinos) there was a lot of excitement that we might be on the verge of discovering a new understanding of the universe while at the same time there was a lot expectation tempering about how this is probably just some strange measurement error.
That we don't have the same happening is why we should trust stuff like this less.
I always joked to myself that it was way easier to get a good statistical analysis on the movement, trajectory, and potential collision course of some asteroid in a distant celestial body several hundreds of thousands of light years away then it is to get good data on the correlation and causation of whether a monkey consuming potentially carcinogenic food would get cancer and other diseases over a long period of time.
Even granting that many studies don't replicate, I think this is thinking about things on the wrong level.
Here are what I consider to be some of the most interesting few psych discoveries of the past few decades:
- Human behavior is sculpted by reinforcement learning controlled by a prediction-error-based algorithm in the nucleus accumbens.
- The spine, brain stem, and midbrain gate pain signals coming from the body based on attempts to predict their veracity
- Psychedelics work by overloading 5HT2A receptors, suggesting they have a role in some sort of construct like plasticity or rigidity
- Schizophrenia appears to be a neurodevelopmental disorder with a highly genetic component, possibly related to the immune system pruning synapses wrong and mediated by glutamate
- Depression seems to involve too little synaptic branching, especially in the hippocampus, probably downstream of BDNF in some way
- Most traits are about 50% genetic and 50% random, with seemingly little role of family environment
- There sure are lots of optical illusions, and that's because our brain makes complicated design decisions that balance speed vs. accuracy.
I think most of these will turn out to be true. Even the ones that turn out false won't fail just because one guy forgot to adjust for multiple comparisons and a replication attempt proves it wrong.
You could say that I'm cherry-picking especially robust or good results, but I think that most of what psychology teaches us, or what the average person should know about psychology, consists of things like these, and not of the latest n = 10, p = 0.0499 study.
I don’t think massive failure to replicate is a sign of a problem, depending on *how* massive. If there are thousands of promising ideas, and only 50 of them have a successful trial that can be published, and then only 15 of those can be replicated, then I’d say the field is doing pretty well, even though 70% of published studies fail to replicate. Insisting on waiting for the replication before you get any publication just means that other labs aren’t aware of the promising lead until much later, and you have a lot more undiscovered true effects, and you have less of a sense of who is doing productive work.
On the topic of intro psych textbooks, here is a recent study analyzing IP textbooks to see how they treat studies that have been replicated versus not replicated. In a nutshell, IP books talk about replication in the method section of course, and then largely ignore replication status of the studies that are cited and examined in the rest of the book.
And we should remind ourselves: who collected the data needed to critique the field? Who applied for funding, submitted the IRBs, and organized and ran the tedious studies to assess reproducibility and hectored everyone to improve their research practices?
Are the sub-portions of psychology captured by the two concluding chapters of your textbook - disorders, therapy and treatment- also "healthy" or is it fair for the skeptics to be concerned about those as well?
Depends on the level at which you're thinking about them.
I think we'll find that a lot of our categories aren't natural, or that a lot of medications work less well (or have more side effects) than expected. I think the best science has already priced a lot of this in (for example, the psychotherapy consensus is that no particular psychotherapy works any better than any other, and it all seems to be a sort of active placebo caused by good relations with the therapist, although I don't think they'd frame it in those exact terms)
I also think we've made a lot of progress since the time when our response to people behaving strangely was drilling holes in people's skulls to let the demons out. I wouldn't be a psychiatrist if I didn't think we could do good work.
I think I concur - "it all seems to be a sort of active placebo caused by good relations with the therapist"
But that's a pretty stark way to summarize a field. Oncology and hematology and pick-your-own-ology may be capable of understanding/treating a lot of associated ailments, but in psychology, our treatments largely fall to simply being a non-traditional-friend of sorts with the patient.
To the extent that's true, that seems a profound condemnation. In some ways, it's worse than mere replication. Replication is somewhat fixable in the near term if we want. And replication also rears its head in other fields. But this impasse on bold effectiveness is pretty unique.
Not that psychology isn't hard and thus force us into this situation, just that it lacks the solid-foundations that a lot of the rest of science does.
> In psychology, our treatments largely fall to simply being a non-traditional-friend of sorts with the patient.
>
> To the extent that's true, that seems a profound condemnation.
I don't think that should be considered a condemnation. If the right kind of social support is a necessary component of healthy functioning, and some people don't have that, then a sort of artificial substitute for that kind of social support might be exactly what a lot of people need. Conceptually, it seems to me like the same idea as an insulin pump being a "non-traditional-pancreas" for someone with Type 1 diabetes.
That's fine as far as it goes, but the field blithely continues with the charade that various therapies actually have non-trivial consequential content apart from the so-called "therapeutic alliance". Surely it's time to put up or shut up at some point?
I agree I am not very intellectually excited about psychotherapy. I used the "research psychology" term in this title to clarify that I meant psychology as the science of understanding the brain/mind, not psychotherapy the treatment modality.
I don't think failure to make intellectual progress in psychotherapy necessarily condemns our psychological knowledge. Wanting to make intellectual progress in psychotherapy is a crazy request! You can't cure cancer by talking about it, so our prior should be that you can't cure depression by talking about it either. The fact that you *sort of* can is quite impressive and I don't think we should demand further miracles from it.
I think psychology the science of understanding the brain is on a better intellectual foundation, which is what I try to describe here.
Psychologist here: I think it's true that the active ingredient in most general psychotherapy, i.e. psychotherapy that isn't targeting a circumscribed problem, is the relationship with the therapist. I wouldn't call that a placebo effect, though. Having a supportive person who likes you and is interested in you is genuinely helpful. Here's a simple example from my life: I am currently getting twice weekly physical therapy to remedy muscle weakness and muscle shortening that are left over from a spine problem I had surgically corrected. I often do my PT exercises at home on the off days, and I am sure I am more likely to do them because I think of my kind and friendly PT as a supporter. His presence in my mind helps keep me in a frame of mind where I am more likely to help myself. I often see similar things happening in the lives of patients of mine who feel understood and cared about. They take heart and start doing things that improve their situation.
And it is important to bear in mind that many people who come for psychotherapy are not at all easy to develop a warm, supportive relationship with because they are addicted to putting up a certain false front, or withdrawn, or extremely mistrustful, or reflexively nasty and sarcastic. And it doesn't help them much to fake-like them. You have to find a way past their bullshit in order to develop real rapport.
Also, there are some forms of psychotherapy where the active ingredient is a technique that targets a particular problem. Really thoroughgoing, thoughtful cognitive therapy is one example. Exposure treatment of phobias and OCD is another. Both of those treatments are hard to do right. They are genuine skills. And exposure treatment is also daunting because it is actually harmful if done wrong.
> Psychology is no different. It has flaws, but most of it consists of good ideas and trustworthy research
I'm broadly with you on the “Psychology is no different” part. But a commenter on the replication post, Jess Kerlin, pointed me to John Ioannidis's “Why Most Published Research Findings Are False” article: https://pmc.ncbi.nlm.nih.gov/articles/PMC1182327/
That article's title is rather bleak. Maybe psychology is no different, because other fields are also broken? What if we are more likely to notice brokenness in psychology because all of us are humans dealing with other humans all of the time? Very few of us pay enough attention to sedimentary rocks or sequencing bacterial genomes to complain about studies involving these thing.
That said, psychology does have specific incentives that make it less likely for studies to replicate. E.g. you doing hot Brain™ Science™, you can always say that a replication didn't draw from the same population as you did, and no one's going to sue you for a random badly run study the same way they might sue you for a badly run drug trial.
People do interact with other humans, but also, to replicate any psych study, you have to run an experiment about as large and involved as the one that found the finding. This isn't true in other disciplines (at least, the ones that don't mostly rely on experiments). Most econ papers can be replicated by checking the code and data, and if you want to make sure they hold up in another setting it's still easier usually to do something that other papers have already done. A bunch of linguistics research is much easier to verify (you can just ask a native speaker if something is correct) than to deduce the initial patterns, or document the vocabulary. I'm less familiar with the hard sciences but my impression is that the same is true there, like if you describe a rock as having some type of structure you can falsify it (by showing it doesn't do X thing characteristic of the structure) more easily than you can prove it (because there are other structures that don't do X thing, idk I don't know geology).
That being said, there are some other heavily experimental disciplines and subdisciplines like language acquisition, although I think compared to psychology it's actually much easier to notice if language acquisition predictions are true in the real world. I think they're probably most at danger for having a replication crisis, especially since they get less attention than big social psych studies.
Thank you - I was reading the comments specifically to see if anybody would bring up Ionnidis' work, and was surprised Scott didn't mention it in his article, in the sense that psychology's failure to replicate isn't terribly remarkable, because that's some huge percentage of studies across all science disciplines.
If you read the contents of "Why Most Published Research Findings Are False," it extends more broadly than psychology (although he's mostly talking about medicine,) but it also doesn't suggest that most of the *accepted knowledge* in most fields is false. We've gotten to a pretty dysfunctional place in terms of researchers' career incentives re: publishing, but "most published research findings are false" does not actually equate to most of the accepted knowledge in a field being incorrect.
I did read it and I do mostly agree re: accepted knowledge.
My main point is more like: what if other studies have their own equivalent of priming that we never hear about because very few of us have to deal with other fields the same way we have to deal with other humans — and acquire a model of their psychology one way or another through practice.
FWIW people often misinterpret the Ioannidis thesis (not saying you are); his point in that paper is that "false" includes a mix of (a) false positives caused by p-hacking, HARKing, etc, and also (b) false negatives because the study was too small to detect the effect, no appropriate power analysis beforehand, etc. Reading Ioannidis can get pretty doomy but on the whole I think medicine is at least marginally better than the average scientific field.
I think you’re misreading Ioannides if you think he’s saying most fields are broken. Most published findings being false is totally compatible with published findings still being our best bet for promising ideas. It just means you shouldn’t believe something on the basis of one study.
Your Ioannidis push is worth extending one more turn. The conditional "maybe psychology's no different because everything is broken" — taken seriously — doesn't just reverse Scott's defense direction, it shifts the shape of the discussion itself. The question "trust psychology?" may not be well-formed against a background of "everything is broken," because "trust" depends on an evaluator seated in some assumption base, and "everything is broken" removes the default seat. You semi-explicit this shift — worth articulating fully: not only "Scott's defense reverses" but "the question's well-formedness depends on tacit acceptance of a default assumption base." That tacit acceptance holds throughout this comment section — not because commenters don't realize, but because raising the point sits out-of-bounds in the discussion's game. Your "everything is broken" conditional has more force than its rhetorical function here typically permits.
"Social psychology fell into this trap especially hard because it naturally touches on hot topics of popular interest and political explosiveness"
On top of that, the cases where it fell hardest would have been far far more likely to be the kinds of cases where people **love** to quote the result at their political opponents. "CheckMATE republicans!" style.
So even if it makes up 0.5% of psychology it's likely to make up a vastly higher percentage of [the psychology that people throw in the faces of their opponents.]
So when their opponents watch a huge portion of the things they've had thrown in their faces for years crumble into dust it's not so unreasonable to also doubt the very foundation on which those were built and the departments that produced them.
As a psychologist reading this ... sees the title "oh good! :) " ... read the rest "oh he doesn't mean me (in social psych) :( " ... I am between social and cognitive so I will just keep doing what I am doing straddling the cognitive side as much as I can!
Not arguing the broader point of "some of this is probably fine", but I'll say that the rot extends far beyond psychology. A number of years ago, a friend asked on FB for help with "nonlinear curve fitting". At the time I was deep in the weeds of levenburg-marquardt analysis (follow-ups to this: https://danparshall.com/papers/parshall_2014_PRB89.pdf ), so I offered to help... and it was "I tried a linear fit, but it wasn't significant, so now I'm trying other things"... no mention, of course, of multiple-hypothesis corrections. This was the same person who also asked me to remind them how to take an average... as in literally, the arithmetic mean. They had a PhD in neuroscience :/
I've had to sit down with people and gently tell them they're p-hacking.
From the inside it often doesn't feel like it. When someone isn't confident with code and stats it feels more like they've fucked up doing the analysis somehow and need to find the correct test. A few hours or days later they've p-hacked their way to a test that yields a result without realising they've been multiple-testing the whole time.
That's understandable for a grad student. But if you're a postdoc, with multiple papers, and you're still doing that kind of thing, then your contribution towards actual Knowledge is very close to zero.
It's honestly amazing that we learn anything at all. But even p-hacked papers are probably right more often than not, and eventually the signal adds up.
I suspect that generally, people who have PhD outside STEM mostly suck at mathematics.
Following the ancient tradition of division of labor, perhaps each of them should have assigned a mathematician to check their equations? Seems like a more reasonable approach than expecting that people who on purpose chose "not math" as their field of study will magically get good at math. Many of them are quite aware that they suck at math, they just can't do much about it.
> Many of them are quite aware that they suck at math, they just can't do much about it.
Many would suggest they should not be doing science at all in these cases, and instead should be doing honest work like tilling the fields (or serving our coffees, I guess, in this age) instead of "publishing stuff I don't understand just because" and literally polluting the epistemic commons, *reducing* the sum total of knowledge in the world instead of adding to it.
But I've thought for a long time that 30% of people getting undergrad degrees is probably 2-3x too many, because it dilutes the signal to meaninglessness.
Sure, but I doubt many "top notch X's" are publishing absolute crap that they don't understand, and just running it through as many R packages as they can before finding significance.
Part of being "top notch" is understanding your epistemic limitations and the boundaries of your knowledge, and taking on collaborators on your papers to fill those, it's a time honored practice.
Manifestly, the people Viliam are talking about do not understand any epistemics, or the boundaries of their own knowledge, or take on co-authors, and are just polluting the informational commons because it's better than tilling the fields / serving the coffees for them personally (but not for the world, which was my point). It's the selfish / defect thing and polluting the commons I'm complaining about, not the fact that they didn't get a degree in stats.
"Erroneous conversion of gene names into other dates and other data types has been a frustration for computational biologists for years. ...
What is not funny though is having genomics spreadsheets suffer from auto-conversion of gene names like SEPT8, DEC1 and MARCH3 into dates, a problem first characterised in 2004. A 2016 article on this topic led the Human Gene Name Consortium to change many of these gene names to be less susceptible to autocorrect. Despite this, our work here shows that gene name autocorrect errors continue to accumulate in supplementary genomics spreadsheet files at a rapid pace."
yeah, spreadsheets should absolutely NOT be used for science. The famous Reinhart & Rogoff result about debt-to-GDP ratio was actually from a grad student who didn't drag the Excel formula down the full column (to their credit, they didn't throw the student under the bus).
But part of the reason this was found in the first place was because it's common practice in Econ for grad students to reproduce existing results. Would that other fields held the same standard!
People shouldn't be using gene names in big collections of data, full stop. Never mind the stupid Excel problem, the names themselves change too often to be useful. Gene IDs are the way to go.
More seriously, I'd say the field has a long-standing problem with clearly explaining how to do things properly (as well as not doing things improperly) to the less involved practitioners. Though in this brave new world, perhaps ChatGPT can do the proper experimental design etc for us.
Stats professors have a weird propensity for being amazingly capable people while being the absolute rock bottom worst teachers...
I remember during an intro seminar for incoming Phd students they had a stats professor come to give a talk that was *supposed* to revolve around laying out how to contact people in the stats department for consultation and collaboration.
What he *actually* did was to try to use his hour long slot to give a rushed stats lecture where he threw a huge amount up on a projector without explaining what he was showing.
This may depend on country and university, so not sure how much my experience generalizes, but... people seem to underestimate how horrible are math teachers outside the STEM bubble. And statistics is the most difficult part of math that still somehow concerns non-mathematicians.
I took lessons at three universities. When I studied computer science, the math lessons were great, in general. Only statistics was like learning a set of ad-hoc rules, without really explaining *why*. I was generally good at math, but when we got to the part about confidence intervals, it just completely stopped making sense to me. I was like: "wait, isn't this actually the other way round?" But no one in my class seemed to understand it; it was just like: memorize this, pass the exam, you will probably never see it again. -- Decades later, when I read on Less Wrong about the Bayes Theorem, it finally clicked: it indeed was the other way, that's why it didn't make sense. (It's like confusing "X implies Y" with "Y implies X", except here it is "X implies Y with probability P" and "Y implies X with probability P". People understand that this is about X, Y, P; but they misinterpret the connection.)
When I studied economy, the math teachers there didn't have any deep mathematical understanding, only a few memorized formulas they followed blindly; and that's what they trained their students to do. (For example, they taught you how to calculate the determinant of the matrix, but they had no idea that if two lines in the matrix are the same, the determinant must be zero. They insisted on "calculating it properly".)
When I studied psychology, the stat teachers just told us to put everything in SPSS.
Seriously, people outside of STEM have no chance to get proper math education. There just are not enough people who know statistics well and can explain it to someone who is not a mathematician at heart.
I'd say there's probably no solution to this, but there's definitely a solution to this now!
Claude knows statistics like the back of it's hand, and can spend hours of clock time talking anybody, from any inferential distance, into understanding!
People are basically just choosing to be ignorant and pollute the commons at this point, when a smarter-than-Olympiad-gold math god sits ready to hold your hand to whatever depth you need, for FREE, 24/7.
Huh, you think it would be better to have no signal than to have a signal with noise? I think you’re making some fundamental mistakes about how significantly you can improve the ratio just by clamping down on a field.
If they're part of the ~20% of people that actually link the data these days, a weak maybe, because then other people / LLM's can scrape the data and use it more cleverly. But otherwise, no, they're just polluting the commons, and that's a big enough problem already, and is already getting worse post-LLM.
I personally think there should already be fully automated-to-rigorous-standards science benchmarks everyone should have to submit their papers to (with LLM's in the loop) prior to submitting to any journals, where the LLM's take your data, try to recreate it in epistemically rigorous ways, decide what they could rigorously conclude from it, and if it doesn't replicate, or if it's p-hacked, or if your abstract or verbal take is significantly different than what they think you can conclude, autoreject it.
Like this is something we can build in a few months, some scientists should already have built this somewhere.
You seem to have a conception of journals as a place where you go to read the final, unalterable truth, such that anything false in there is "pollution". I have the idea of journals as a place where you go to read interesting ideas that have some amount of support, such that keeping things out slows our access.
The problem, as you pointed out, is signal and noise. If the noise keeps increasing, the signal dwindles to nothing, and all the tailwinds are in the "more noise" direction (ie degrees expanding so ever less competent people are publishing, LLM's helping people get through bad and poorly thought out papers sooner and publishing, etc).
Which way do the incentives point? To publish as many spicy, interesting, surprising conclusion papers as you can, to max getting talked about and citations, to win better postdocs or a tenure path or tenure. So the incentives all point directly at "pollute the commons as much as possible," and now LLM's are enabling that at record scale.
If we don't do some sort of mitigation measures, we're going to lose the already-dwindling signal entirely, in an infocalypse of glurge.
Why did you co-author Redefine Statistical Significance if this is your view? That paper seems in tension with what you're saying her, since it seems to be pushing a proposal aimed at reducing the proportion of false positive claims amongst the positive claims being reported in journals?
Unfortunately, probably 20% of people do not link their data. This meta meta analysis of loads of biomedical science papers estimates a 95% (frequentist) CI of 1 to 3%. Based on this paper, there seems to be a lot more people falsely saying that the data is currently available in a repository then there are people who actually make the data available. The authors are too catious to draw this inference themselves, and the statistical estimates they present are not easily interpretable, but this seems to be the likely implication.
This is the kinda thing that could be easily aided with Claude, by the preprint servers? i.e. arxiv.org and bioarxiv could have an agent that scans the incoming paper, looks for the "reproduction package" link, and actually tries to reproduce it. If it can't be reproduced, then a red flag on the abstract page is set (vs a green thumbs-up if the data *can* be reproduced)
personally I would have thought that "Neuroscience" counted as "S" in STEM, but I learned that day how mistaken I was.
Honestly even if AI never progresses at all beyond Fable, it will probably accelerate scientific learning just because each of them *will* have someone available to check their math (and gently tell them that they're p-hacking). And also automated pipelines to confirm all the analysis at scale before we send to review.
It's not like mathematicians are miraculously immune to the perverse incentives that are the root cause of the replication crisis. And I'm telling you this as a mathematician.
There are many papers out there that have holes in their proofs, even when the results are likely true. And there are even papers where the results are downright false but have yet to be corrected. Or even whole subsubfields that are built on highly dubious foundations. I've met all of those issues during my PhD.
People who have a PhD *in* STEM mostly also suck at math. In fact, they even suck at the other areas of their discipline other than the one they work in.
A non-trivial fraction of the time, there are in fact statisticians who collaborate on projects that do anything statistically interesting.
But making mathematicians participate in other work would be a recipe for making mathematicians commit suicide.
Fortunately, there are incentives so that once in a while, someone inside a discipline gets a lot of credit for noticing that a way of calculating things lots of other people are doing is causing problems, and things get tidied up. It might be nicer if we had spare mathematicians to ensure the mess doesn’t get created in the first place. But you don’t want to stop everything until a mathematician takes part in every step. (And vice versa when mathematicians are writing about the math of elections or the math of galaxies and making stupid mistakes about those things.)
I’m sure that if you were reading blog posts by social scientists before the social science replication crisis, they would have told you that it’s a pretty respectable field. But the a-priori trust or distrust of science in the public doesn’t have anything to do with that, or with the actual quality of the results, which obviously the public can’t verify in most cases.
It has to do with the perceived robustness of the system and the perceived honesty of the people in it. The public wasn’t ever given any reason to believe that any other fields of science were immune from the issues that emerged in the replication crisis, so a-priori distrust still makes sense.
A lot of what the neo-Freudians started off was built on and still is used in some form today. For those that don't know:
- Adler's inferiority complex picked out something real even if his framing was flawed.
- Jung developed the introvert/extrovert distinction, which is everywhere now.
- Erikson's psychosocial stages of development are questionable, but children's development is still commonly put into these kinds of stages.
- Horney's breakdown of the different kinds of coping styles for neurotics is analytically correct. What other options are there for interpersonal behaviour beyond moving towards people, against people and away from people?
It's perfectly reasonable to have these four in a chronology of the development of psychological ideas a textbook.
I think sometimes naming those categories are either trivial, or harmful. Like introvert/extrovert where people just can't agree what either of those means, and I'm not even sure if official literatures even have united definition. Or even more, MBTI, which has been criticized over and over already. Sometimes we even need to rip off the old definitions from the root and make a new one.
It's certainly the case that Jung was using a different definition for the pair than is usually used in, say, current day personality research. And that could cause confusion. Still, there's definitely a real personality trait (or set of related traits) being picked out here. Even if in pop culture conversations a lot of things get confused and conflated.
> Erikson's psychosocial stages of development are questionable, but children's development is still commonly put into these kinds of stages.
But that's not an example of psychology research being "mostly fine". That's an example of psychology research being in such a terrible state that they've been producing fake work for many decades and still haven't realized it.
I meant that his work is probably wrong, but there are later formulations of stages of development that are broadly accurate or useful, so you could say Erikson was a forerunner here. Either way, fine to have him in a history of psychology section.
As a psychologist, I would emphasize that there is still a lot of indefensible research in almost every subfield of psychology. In the case of the flicker-memory effect, I think it is entirely reasonable to be curious and to try to replicate the effect in good faith using solid research methods. However, I think most examples of bad research do not originate from this kind of curiosity, but from misaligned incentives that promise greater status or better job opportunities for producing striking results. The field has done too little to eliminate these incentives.
I would also point out that other fields face the same problems but have barely begun to acknowledge them. Psychology looks reasonably healthy to me, roughly on par with fields such as economics or medicine.
100% agree, and I also think this makes people very complacent about replication issues. Thinking that poor replication is specific feature of psychology as a field will lead you to miss a lot fundamental issues with science (p-hacking, publication bias and misuse of bull hypothesis significance testing). Some fields have better standards than others (and in fact psychology has improved significantly over the past fifteen years) but no one can afford to ignore the problem!
I start a Psychology course(mostly because Philosophy wasn't an option) so I'm glad to be more assured of & intrigued about the subject by this article. Thanks.
I also think that since Freud and most Neo-freudians weren’t really trying to be highly empirical scientists but rather shape therapies, literature and self-inquiry in a complex way that reflects the human experience (and inexperience) it’s odd to cover them as part of a replication crisis.
The fundamental problem in psychology and so many academic subjects is that the incentivizes reward novel findings over debunking or rebuttals. Whether it is psychology or philosophy or women's studies good ideas/discoveries should be rare so we should want more rebuttals of novel claims than novel claims. The academic incentives favor the exact opposite.
This is why I advocate a policy that every academic journal (pure math/cs maybe excepted) should have a rule that every time they publish a primary paper they will also publish the best rebuttal/response to that finding submitted after 1 year and then again after 5 years (and maybe more if it gets highly cited). And for empirical disciplines maybe another slot for the first attempted replication. Critically it must be a publication in the same journal (and ideally the cite wouldn't even reveal it is a response until you track it down).
That will solve the problem that people do studies and everyone kinda knows it probably isn't true but the incentivizes discourage anyone ever actually showing that. Of course we need to pressure the journals to adopt it.
Maths are not immune to the issues that plague other sciences. During my PhD I've encountered a few examples of papers with holes in their proofs, or even results that were completely false. In fact, one of my own papers was published in a peer reviewed journal despite containing a glaring error that was somehow overlooked by myself, my PhD advisor, a couple of senior researcher and the reviewer.
I think the movement to provide machine-checkable proofs is good, though I'm not sure specifically Lean (or other current/older proof assistants) is the right way to go.
Yes ofc you get holes but the costs to benefits are very different. It probably doesn't make sense to make a correction a full blown paper.
What we need in math is a combination of arxiv with an ongoing discussion of the work so it always displays the current status and apparent issues can be flagged. But you don't need someone to go run a new study or write a counterargument -- errors are found by people trying to read the proof and mostly just need to have a place to be recorded.
I’m not sure that this is really such a problem. In most academic fields, the limiting factor is finding good new ideas at all. It’s generally better to have more good ideas, a lot of which are wrong, than to try to limit the number of good ideas out there while ensuring they’re all right.
The latter is more relevant for engineering and other implementation, but not for research.
I mean if good ideas are rare surely the important thing is to identify them and distinguish them from apparently good ideas no?
I mean if you think what is valuable is purely hypothetical claims about psychology -- hey you people might behave like X haven't tested it -- then we are making a massive mistake by asking for evidence or before publication. The valuable thing is the thing you might test not the fact it is true.
OTOH if what actually is valuable is the actually true idea because those are much rarer than the ones which merely appear to be true surely it's really important to distinguish the two kinds of ideas. In areas like philosophy with very limited theory (or actually I would say with limited theory we aren't evolved to find obvious) you are going to need lots of effort to prune out the incorrect ideas that look good when tested by someone whose career depends on a positive answer using small sample sizes.
And that matches what people in the field say -- they go to conferences all the time see announced results and don't update their beliefs at all because they know they are almost certainly false. So we end up with a literature which is pretty useless for making inferences about psychology because of the high rate of false positives.
If there's a field of 1000 possible ideas, 20 of which are true, there's some advantage in a process that identifies 50 candidates, 10 of which are true, and there's some other advantage in a process that identifies 5 candidates, 3 of which are true, and there's also some advantage in a process that identifies 1 candidate, which is true. It makes sense that the clinical side of thing wants to do the last of these, but I don't see any reason why it would be better for the research side of things to be at the second rather than the first.
0) I think ultimately we do research because in some sense we care about the answers and that does ultimately mean we really do get more value from knowing what has probability of being true at 90% than at 5% because we want to use the facts. Either clinically or just in understanding and thinking about the world. Unf humans is we really can't use a bunch of low probability discoveries to help us understand the world because we lack the cognitive capacity to sum up the contributions from all those possibilities.
And if psychology had two buckets for the different kinds of ideas that would be fair enough -- the problem that needs fixing is that it doesn't because the incentive to actually do the double checking to put ideas into that higher confidence bucket isn't there.
My proposal exactly creates that incentive. It would especially do so if you require any response studies be pre-registered -- since then when the authors reply they can just be like "well 90% of the attempts to reply actually ended up confirming". Yes pre-registration is generally nice but the problem is that even if you have a general database it's really hard to know what is testing the same things and there is too little incentive to do it -- organizing it as a response to a specific hypothesis in a pub with a guarantee the best response gets a pub creates both.
(specifically pre-registration is least useful when everyone is trying to claim a novel discovery not a confirmation).
1) It isn't clear that generating canidate ideas in psychology is very hard at all. I mean there is nothing that was hard to generate about the idea of power poses or stereotype threat or etc. These didn't clear any high theoretical bar like might be true in particle physics -- it's almost exclusively the experimental justification that is doing any work distinguishing good and bad ideas.
And that's just what it looks like to be a field that is more empirical and less driven by theory. Given the false positive rate is essentially fixed and that correct ideas should get more rare over time something eventually needs to be done.
2) Finally let's assume that instead of figuring out what ideas are really almost certainly correct we want to discover a large range of ideas that might be correct -- ie pass some minimum threshold even if most are still false. If literally every study was pre-registered maybe that makes sense.
But without that what you find out isn't what claims have a chance of being true but just the useless fact that enough people tried that *sort* of thing until a study gave you a positive result.
That's literally the least useful information you can imagine. I mean if you are thinking -- hey X seems plausible I wonder if it is true and you discover: either X is pretty plausible or it isn't but lots of people think it is and tried to prove it and eventually succeeded.
> Whether it is psychology or philosophy or women's studies good ideas/discoveries should be rare so we should want more rebuttals of novel claims than novel claims.
That is definitely not what we want; what we want is for spurious claims not to be made. Your way just causes an infinitely increasing deadweight loss.
The way to get that is to punish people for bad results, not to reward other people for trivial results.
The root problem here is that our policy is that "doing science is good" independent of any results. This has unsurprisingly resulted in all of the results being fake. In a context where someone must rely on the results, things work fine.
Having recently taken a forensic psychology class (so, y'know, totally unbiased information here): not really. They've developed things like risk assessment tools that can predict whether someone is likely to re-offend (they're not perfect, but they're better than going off vibes and much better than chance), done tons of well-replicated experiments on stuff like eyewitness testimony, and invented interrogation techniques that are more effective and less evil than what existed previously.
I don't know much about Not Guilty by Reason of Insanity, because that's a US thing. In Canada, there's two separate categories: Unfit to Stand Trial and Not Criminally Responsible. (The key difference between NCR and NGRI is that NCR is raised at the sentencing phase, after they've already proven that the accused committed the acts.) The forensic psychologist has to do assessments then. I don't know how you could verify them, though. There's no objective test of "is this guy crazy", so there's nothing to compare the professional judgment to.
This is a weird post to me because as far as I can tell the replication crisis is an issue in many other fields,. including "rigorous" sciences such as physics and mathematics.
One of the main causes of the replication crisis is the misaligned incentives that researchers face. Those incentives are basically the same in every scientific field.
Can you share an example from physics and an example from mathematics?
In physics, I can can only think of examples of the form 'we agree this number is ~1.6, but disagree whether it's ~1.61 or ~1.62' and 'we all agree this shadow is delayed. You think it's because light doesn't travel instantaneously, just really fast. I admit that would explain the shadow but refuse to believe light has a finite speed for other reasons'. The best I can come with in maths is some people have pointed out unjustified assumptions in some proofs, which have subsequently required an appendix to be added.
I decided to spot check this by asking Claude about a randomly-selected item from Scott's list of mundane, unaffected sub-fields: neuropsychology, because my d20 landed on 14.
It's a funny one, because traditionally it's not a field that does replication (in the normal sense) at all. They're not even trying to replicate! It would be unethical if they did, since it would mean, like, slicing up somebody's brain the exact same way as another lab did. You're not going to replicate Fineas Gage, that'd be evil.
But "it would be evil to replicate studies in this sub-field" is not the same as "this sub-field is an example of psychology succeeding at doing replication"
And Claude let me know that the modern fMRI version of neuropsychology has suffered a bad replication crisis of its own, due to bugs in the statistics software. I don't trust AI a whole lot, but the papers Claude cited to me do at least exist.
So I guess neuropsychology is a good example of "it has flaws"? But not a good sub-field to cite if you want me to credit psychologists generally with being trustworthy on the basis of successful replication of studies.
I think the overwhelming majority of structure-function research is very well replicated - for example, the role of Broca's area in speech, of the hippocampus in memory, etc, all the way down to the Nobel winning studies on the role of prediction error in reinforcement learning in the nucleus accumbens. Many findings are replicated on rats (RIP) and other primates. And in practice a lot of neuropsychology is testing - for ADHD, for concussion symptoms, etc - and that is "replicated" in the sense of the test reliability, specificity, sensitivity, etc.
I think this is the same error that this article tries to call out - engaging with fields entirely through one article in a popular science magazine about a replication failure, and then assuming that must be a significant portion of the field.
Separately, I think that some of the criticisms of fMRI are kind of unfair - see https://academic.oup.com/brain/article/140/8/e53/4032512 . The famous "dead salmon study" only worked because the people doing the demonstration deliberately avoided doing a multiple comparisons adjustment to make their criticism more attention-grabbing.
I think my engagement is more along the lines of epistemology and warning alarms for slippery rhetoric. For example, you say that, among others, "the role of Broca's area in speech" is "very well replicated". Cool! How? "Many findings are replicated on rats (RIP) and other primates."
I'm skeptical that the findings about the link between Broca's area and speech were replicated using studies of rats and other primates. The reason is that I don't think that those animals can talk. But I'm not an expert. Perhaps you're using specialized terminology that I don't understand. Either way, there's something slippery about those two sentences.
I do think that this particular sub-field, by ethical necessity, must have a special relationship with replication. Something like, "Ethics forbids us from holding human structure-function research to the highest scientific standards of replicability, and yet the discipline produces many reliable results and should be considered trustworthy," I think I would be happy to agree with.
But alarm bells go off when I see a sentence with a list of replicated findings, and then a sentence describing replication methodology, and there's at least one item in the list of findings that seems (to me, a non-expert!) like it couldn't possibly make use of that methodology, because rats and monkeys don't talk.
I'm happy for Claude to be wrong about fMRI studies. Cool.
The deeper issue is that, the more closely psychology approaches to subjects of study that people care about, the less ethical it gets to apply the highest scientific standards. This is tautologically true: "people care about this" equals "don't coldly manipulate it as part of a scientific study". It's great that they've replicated results about language learning and nerves and reaction time; but what we really want to know about, us normies, is why do people form lynch mobs? What was going on with the USSR? Stuff like that. And you can't replicate a lynch mob or a brutal dictatorship in a laboratory -- and even if you could, you shouldn't.
I think it's better to admit that psychology is on the horns of a dilemma, and say, "we can't design replicable studies for some of the stuff we want to study, and we shouldn't, so take our results as being as trustworthy as the methodology we were able to apply warrants". You just have to have lower standards for some subjects, and trust the results proportionally less, and that's okay. As long as you don't try to claim more trustworthiness than the actual methodology warrants.
There are human lesion studies, where they collect dozens or hundreds of patients who were brain-damaged (via seizures, tumors, accidents, whatever), measure exactly where the lesions are, and then quantify what the corresponding symptoms are. I find that lesion studies are often very informative. I would happily trade away dozens of fMRI studies in exchange for one lesion study.
There’s other kinds of (non-fMRI) evidence too: EEG, MEG, direct stimulation studies, TMS, …
Anyway, in my opinion, available evidence to date (especially lesion studies) prove beyond any reasonable doubt that Broca’s area has an important role in speech production.
A great recent technical reference on the neuroscience of language is "Wired For Words" by Hickok. I find that he’s unusually scrupulous in documenting what we can and can’t learn from different types of evidence, etc.
Because of what Steve mentions, I'd place high probability on the ability to trace a mental function to a brain area's involvement in *some* capacity.
But that's a very coarse finding. What's practical from that? If you see someone with Broca area damage, maybe refer them to the speech therapist?
The more useful knowledge is going to be less solid foundationally. Or at least requires more careful study to be equally reliable.
If someone has damage to x% of that brain region, how much recovery would they expect after Y months of Z therapy?
By analogy, I recall a few articles of Scott's about drugs to the effect of "This neurotransmitter has something to do with sadness, so we found a similar chemical to give sad patients and it kind of worked for awhile, then we discovered we were wrong about the mechanism" (Not an exact example) How does that kind of fine grained knowledge hold up in various domains?
This isn't a gotcha, I don't know how psych broadly compares to other research fields, but that's the kind of overview that would be persuasive.
I can’t comment on Broca’s area specifically, but my grandfather wrote a book on the general principle.
Sadly, across a large country’s population, hospitals encounter large numbers of people with head injuries that involve destruction of parts of the brain. You can gather all of that information and correlate with observed symptoms, and given enough data assign some functions to some regions (this is awkward because the brain has some plasticity in the sense of being able to reuse surviving areas for some functions, and also some functions vary between different people or are not localised, and also there are dependencies, but with enough data you can work all that stuff out).
Meanwhile, new patients, sadly, do keep coming; so you look at where the damage is, write down your predictions, and when they wake up see how often reality matches what your map predicts. Other hospitals also do this. And thus you learn whether what you observed replicates.
TLDR: you don’t have to cause the damage in order to study the effects of damage. Plenty enough damage happens anyway. Replicate like an astronomer, not like an engineer.
> For example, you say that, among others, "the role of Broca's area in speech" is "very well replicated". Cool! How?
In that Broca's aphasics tend to have brain lesions there.
The fact that something is going on is well documented. I wouldn't consider this much of a win, though; "the role of Broca's area in speech" is the entirety of the finding. Normally, if someone said something like "we know about the role of X in phenomenon Y", you'd follow up with "what is it?", but you can't do that here.
>They're not even trying to replicate! It would be unethical if they did, since it would mean, like, slicing up somebody's brain the exact same way as another lab did. You're not going to replicate Fineas Gage, that'd be evil.
"People act like the replication crisis has discredited psychology, that the field is now widely agreed to be a sham, that it can be uncontroversially held up as an example of science gone wrong."
Who does that? It's not as if the replication crisis were limited to psychology, but i don't see anyone claiming that biology or pharmacology (for example) are sciences gone wrong.
I assume most people naturally understand that things like classical vs. operant conditioning, long-term vs. short-term memory, the tendency of people to form ingroups and outgroups, the role of hormones in emotions, and the REM vs. non-REM sleep distinction are settled beyond much probability of being overturned by one new data analysis.
For other things, like the efficacy of antidepressants, or the genetic contribution to IQ, you can read the rest of ACX.
I assume you mean "most long term readers of ACX" (which to be fair does include me). I find, at least for myself, it's often surprising just how recent some discovery is, and I suspect many would actually have difficulty placing some category in the safe versus suspect buckets.
Anyway, science and psychology in particular probably aim for more specificity than "tendency to form groups" or "X hormone has a role in fear/joy/etc." and there are presumably follow up studies quantifying these effects. How do they fare? Etc. Maybe Non-social psych 101 holds up, but that doesn't necessarily give anything reassuring about the current state of the practice even if some of the ancient knowledge base is firm.
Great post. Psychology isn’t bullshit. Social psychology isn’t bullshit either. The replication crisis is real, of course, but it’s been badly overgeneralized. Many areas of psychology are just fine — cognitive psychology, personality psychology, psychopharmacology, etc.
Two small corrections: (1) in social psychology, it wasn’t just priming studies that failed to replicate, and (2) developmental psychology also suffered from replication problems. Scott’s basic point still stands.
Nevertheless, the replication crisis was huge for philosophers and other academics who drew heavily on social and developmental psychology. It made heavy reliance on those fields untenable. My own approach shifted dramatically. Like, in my book A Better Ape, I drew on evidence from primatology, behavioral economics, archaeology, etc.
Initially, I thought this sucks. But it was actually a good thing. Philosophy that relies narrowly on social and developmental psych — indeed on any narrow set of empirical fields — was never any good in the first place, even if the studies themselves weren’t bullshit.
Philosophers should integrate multiple sources of evidence across fields. Like for the topics I’m interested in, social and developmental psych are relevant, but so are economics, political science, large-scale social surveys, computational social science.
The more convergent evidence, the better. Plus, some of these fields responded to their own problems by developing methods that better expose causal relationships (the credibility revolution).
So, from my interdisciplinary perspective, the replication crisis was initially a bummer, but the ultimate effect was positive.
There's a idea I like from Karen Horney-- that abusive, neglectful, or intrusive parenting can lead to irrational perfectionism. If being a normal kid isn't good enough for the parents, the child may invent irrational standards for themself. Has this been disproven?
At some point, I got a list of legitimate results from psychology. Unfortunately, the only one I remember is loss aversion. What else is good?
My impression is that the false results from social psychology that catch on are the ones which promise ways to control people on the cheap. Fair?
The Milgram experiment has been rather extensively replicated, most recently in Poland in 2017. Which doesn't exactly invalidate your impression -- it probably became famous because it promises a way to control people on the cheap -- but in this case the result seems to be true.
Wikipedia seems to talk about some criticism of the original study, both in terms of accuracy and validity, but I couldn't find anything about failed replications (except replications that deliberately changed aspects of the experiment, i.e. saying "the learner's health is irrelevant" was so obviously evil that compliance rates dropped).
You're looking at three different modalities (abuse, neglect, and intrusion), all of which have different failure modes (neglect may not be bad for a self-motivated kid), "intrusive parenting" may get a child who is failing to walk to do so "because mommy and daddy are worried". Aka: "these may be pro-social behaviors in the right circumstances" (abuse, not so much, but that's basically corporal punishment Done Wrong).
Parents who hold their kids to standards they cannot reach (whether or not they're rational) tend to get narcissists as children (where the kids "play pretend" that they're good enough).
There’s a ton of psych papers (and more every year) that unquestioningly assume a paradigm in which parents (and schools etc.) directly sculpt what kind of adult a child will grow into, like Bernini on clay, and approximately every single one of those papers is garbage, as proven by behavioral genetics (see §2 of my post https://www.lesswrong.com/posts/xXtDCeYLBR88QWebJ/heritability-five-battles if you’re not already familiar with that).
The methodological problems are usually: (1) neglecting the impact of genes, (2) neglecting the “reverse causation” in which (for example) there’s a child—bless his heart and love him to death—but he has an innate disposition that makes him a defiant prick, and therefore his parents will naturally yell at him more, punish him more, etc. (3) boring old p-hacking etc.
>”the forensic psychologists who assess whether criminals are not guilty by reason of insanity”
It’s actually a gigantic point against the field that there is an entire subdivision dedicated to fraudulently allowing violent criminals to escape punishment (at worst) and gumming-up the legal system with frivolous claims and appeals (at best).
Criminals don't "escape punishment" through NGRI. They're diverted to forensic psychiatric hospitals, on average for an amount of time longer than the original prison sentence.
And it's not "fraudulent" - in 70-90% of cases where criminals are found not guilty by reason of insanity, both sides (ie even the prosecution) agree that the criminal was insane. In about 85%, there was already a preexisting diagnosis of a major mental disorder like schizophrenia.
And it's not "gumming up" the legal system - it's a factor in fewer than 1% of all felony cases.
If NGRI isn’t any better than a guilty verdict, then why does anyone ever use the defense? Maybe unsophisticated defendants are being taken advantage of by lawyers who waste their client’s and the public’s time and money to no end, but I suspect that the defendants get some utility out of an NGRI verdict.
And I don’t mean “fraudulent” in the sense that there is some agreed formula that forensic psychologists are deviating from and lying about (though I have my suspicions about that as well), I mean that the whole concept of someone not being culpable for their own actions due to a psychological defect is incoherent.
Imagine for a moment that, “the gods made me do it” was a recognized defense in our legal system, and that astrology was admissible evidence. There would almost certainly be a small industry of very real astronomers who use perfectly legitimate science to testify in court as to the exact locations of various planets and comets at the date and time of the defendants’ birth. The fraud would not be that these testified locations of heavenly objects are wrong, the fraud would be that *none of this matters at all* and is a massive waste of time and resources.
Maybe you would disagree that this constitutes “fraud” on behalf of those astronomers. I admit I used the term a bit loosely, but Astronomy as a discipline scrupulously guards against any implication of paranormal or moral significance in their routine findings. It’s hard to imagine a science blogger defending the honor of astronomy by appealing to horoscopes.
Regarding the “1% of all felony cases” statistic, I’d like to see it weighted by lawyer billable hours. That would be a much better estimate of societal cost than number of cases.
Sounds to me like your picture of insanity defense cases was infuriating, but also really inaccurate, and that having heard the facts from Scott you are now trying to shore up the case for being furious. Why not consider the possibility that your picture of these cases was just wrong, and there's not much to be mad about? Hey, don't worry, there are still plenty of other things in life to be mad about.
> "If NGRI isn’t any better than a guilty verdict, then why does anyone ever use the defense?"
Many voters believe that, when there's a person with severe mental defects that are a factor in them being criminally violent, they should receive mandatory medical treatment and care.
In fact, such a view is an aspect of the majority of people's philosophy of justice. Our elected officials respond accordingly.
I understand that this can be upsetting and frustrating to people with very unusual philosophies of justice.
You seem to have misread the comment you're responding to as "why does anyone ever prosecute for insanity?", despite having quoted it and presumably having seen that it says the opposite of that.
Your comment is about what you want to do to criminal lunatics. How is it supposed to be responsive to the question "why do criminal lunatics ask for this treatment?"?
>If NGRI isn’t any better than a guilty verdict, then why does anyone ever use the defense?
I wonder about that, too. I've got three guesses:
1. The control isn't perfect. The statistic is that normalized by criminal charge, people found NGRI spend more time institutionalized than people found guilty spend in prison. We don't know if the NGRI people would have spent typical amounts of time in prison or not, and it may be that NGRI people would have spent much longer in prison than typical because of some combination of getting in more trouble while incarcerated, being less sympathetic during the sentencing phase, and being less likely to be offered light sentences as part of a plea bargain.
2. Related to #1, people who aren't found NGRI can still be committed to mental institutions if they're adjudicated to be dangers to themselves or others. If you're found NGRI, you're released when you're no longer considered dangerous, which averages longer than the typical sentence but might be shorter if you respond well to treatment. If you're found guilty and then institutionalized while serving your sentence, then you're stuck serving your sentence (unless you're pardoned or paroled) even if you're released from the institution before your sentence is up, and if you're still a danger after your sentence is up you will stay committed until you're not. So you're likely to wind up in mental institutions just as long either way, but with NGRI you have a chance to be released early.
3. Prison and mental institutions are different experiences despite both representing confinement and loss of freedom. It's plausible to me that some people may reasonably prefer going directly to the mental hospital rather than to prison even if they'd spend similar amounts of time in both.
> If NGRI isn’t any better than a guilty verdict, then why does anyone ever use the defense?
My understanding is that in certain jurisdictions, sex offenders, etc., can be found guilty, imprisoned, then sent to mandatory psychiatric treatment after serving their sentences instead of being released, so it's in their interest to skip the first part.
What do you mean “isn’t any better than a guilty verdict”? Are you replying to “ diverted to forensic psychiatric hospitals, on average for an amount of time longer than the original prison sentence”? It seems like someone might legitimately prefer years in treatment to years in prison together with a criminal record. And not just the person subject to the treatment, but others, if the treatment is of relevance to the persons condition.
There’s a ton of evolutionary psychology papers (and more every year) that unquestioningly assume the paradigm of “evolved modularity”, where e.g. there’s a specifically-evolved language algorithm, a specifically-evolved “intuitive biology” algorithm, a specifically-evolved “intuitive physics” algorithm, an “intuitive human social relations” algorithm, all somewhat intermingled for sure, but all innate, and in the cortex. I think that paradigm is pretty wrong for reasons here: https://www.lesswrong.com/posts/wBHSYwqssBGCnwvHg/intro-to-brain-like-agi-safety-2-learning-from-scratch-in (and see also a related argument in book form https://www.cambridge.org/core/books/beyond-evolutionary-psychology/4731C143A9557767DC37A59526F1F525 ). That doesn’t mean each individual result is 100% wrong, of course; often there’s some nugget of value wrapped that can be extracted from its confused interpretative framework.
Are you thinking of Piaget and Kohlberg? Curious what makes you skeptical of them - I feel like my children behave in about the way that the theory predicts.
I'm thinking of the stuff that relies on tracking the "gaze" of infants. I haven't looked into it thoroughly recently, but a quick search does show that there is some effort being put into doing larger / better studies.
I took a child psychology class in university. In it, the professor told us about a study showing that at a certain age, children who play with a large toy (a rideable toy car, a slide) and are then shown a miniature version of the toy, will then try to play with the miniature toy in the same way. My professor told us this was due to a lack of connection between parts of the brain involved in object recognition vs. planning physical movement.
A very impressive and surprising result, which has hundreds of citations (top 1% of research) and even has its own Wikipedia article: https://en.wikipedia.org/wiki/Scale_error
At timestamp 00:59, the researcher can be heard saying, "why don't you pretend like it's the big one and you can get in and take it for a ride." The narrator continues describing how the foolish child ignores the size of the car and tries to get into it. "They did not modify their behaviour with the toys, even though they could see that they were smaller."
Yeah, because the researchers told them not to!
There's a possibility it was just the film crew telling them to pretend so they could get some footage. But why would they have to do that if the effect was real?
Neither the professor of my class, nor the other students (as far as I know) caught the apparent fraud, nor apparently has the subfield of developmental psychology.
At the time I was already aware of the replication crisis, and this further affected my trust in science. Whether or not this specific researcher lied, they had an incentive to lie and easily could have, and I'm just supposed to trust them?
It's 2026 now. Video is cheap. I wonder if we should have researchers film all interactions with subjects (or even all data collection of any type) and upload them so they can be scrutinized. Faces could be blurred as necessary etc. Wouldn't work in some cases, but I think for most studies, most participants wouldn't even care that their face was going to end up in a database somewhere.
Less relevant to this discussion, but I just remembered I have also been a participant in a bullshit study. I think it was something about showing you a word, and then a quick flash of an image, and you had to click a button if the image was X, or a different button if it was Y (I forget what X or Y were). But as a participant, every image was very obviously a blend between two different images, one X and one Y. I guess the idea was something about determining whether the word would prime you to see a certain image, but it was always obviously both. I was never asked if I noticed more than one image at a time, and I never told them.
All of Piaget's theories are his personal fantasies supported by nothing. That would be one of the first examples I'd reach for to support the case "psychology is a sham".
Parts of psychology aren't shams, but if you see Piaget mentioned, you're hearing about a fraud.
There’s a famous result, widely cited by textbooks etc., that very young infants imitate the facial expressions of adults (e.g. sticking out the tongue). I think it’s probably false, see http://www.replicatedtypo.com/sticking-the-tongue-out-early-imitation-in-infants/6082.html . That makes me a bit paranoid about believing other claims about infant psychology, but I dunno.
…That said, my assumption has been that the gaze-tracking methodology is basically sound. I’m open to correction, I haven’t really scrutinized it.
Of course, individual studies can still be bad and/or misinterpreted, even if the basic experimental paradigm is sound. For example, lots of studies draw flawed big-picture conclusions even if their data is fine, because they don’t appreciate that infant behavior is much more driven by the brainstem rather than cortex, compared to adults, see https://scholar.google.com/scholar?cluster=11144225329320125688&oi=gsb&hl=en&as_sdt=0,22
I enjoyed this article and was happy you brought up Freudian-Mother-Loving, which I believe is part of motivation underpinning genomic research in California.
A useful heuristic I found, comes from the Datacolada folks. Basically, if it's "duh we already knew that" then you can get away with tens of subjects per box. If it's subtle self-delusional effects like "smokers think smoking is less dangerous than nonsmokers", then you need *hundreds*.
I pulled out a 2002 psych textbook (Gray 4th ed), and used an RNG to pick page numbers, and then examine the topic discussed on the first sentence of that page.
(417) Talking about how fear-of-heights develops at the same time as self-locomotion in early human childhood, and why this makes evolutionary sense. Seems fine. ✓
(163) Discussion of Wernicke’s area. It’s not great, but it’s sufficiently vague and high-level that it’s basically fine. ✓
(327) “Sensory memory” including the phonological loop. Ditto, I think some of the details are a bit misleading, but it’s mostly pretty vague and high-level, so that’s fine. ✓
(111) Biofeedback training to lower blood pressure — I don’t know anything about this area. Skip that one.
(191) What does the word “lesion” mean and how can you create a lesion in an experimental animal? All good! ✓
(83) The more polygynous a species, the more sexual dimorphism, with males getting much bigger and stronger than females. And why that makes sense evolutionarily. All good! ✓
So for the 5/6 random samples that I feel capable of judging, the 2002 book is basically fine in every case. Didn’t expect that!
Strong skepticism of a field is a pro social heuristic for most people to have if the most influential work in that field is complete horseshit used to support destructive policies and elevate the most anti social segments of society
It’s better for most people to ignore a field, and to mock anyone who wants to use a single recent study from any field to make an argument, but not to actually be skeptical of the field itself.
Better for whom? Academics and other psychology related professionals who'd rather avoid well-deserved scorn for not speaking against or worse cheering on fake news because it flattered their ideological beliefs?
A virtue of heuristics is that it enables unsophisticated people to make judgments without much time invested or expertise required. If woke-flattering psychological research is what they're going to see, what benefit is there in familiarizing themselves with the literature of numerous psychological subfields to make an informed judgment as opposed to just dismissing the hot new study they might see referenced in the morning paper telling them that The Science(tm) proves they're racist?
I didn’t say people should familiarize themselves with multiple subfields! I did say they should mock anyone who wants to use a single new study to argue something.
There was a steady stream of anti-racism supporting studies, not only one. Misinformation peddled by the credentialed only ever goes one way - left. So why shouldn't the everyman simply dismiss those studies? Sure, maybe each makes only one specific argument, but many are generally connected, eg around the idea that injustice is why certain groups are incarcerated more than others
Was there some specific effect supported by multiple studies? If you’re just flattening each study into “does it support racism or anti-racism?” then you’re probably not actually looking at the studies and shouldn’t pay attention.
We're back to my my first comment, which argued that most people should be strongly skeptical of psychology studies. Most people don't actually look at the studies injected into the mainstream and thus shouldn't think much of them. I said be strongly skeptical of them, but I could agree to your lesser recommendation of not paying them attention
Not just the stupid mind you, but also the midwits (see Redditors). So most people do in fact flatten studies into broad categories. My guess is that the tastemaking classes, basically all of whom are leftist, know this and this is why academic studies with anti-racism and environmental-determinism flavour were promoted
>Strong skepticism of a field is a pro social heuristic for most people to have...
And also later:
>A virtue of heuristics is that it enables unsophisticated people to make judgments without much time invested or expertise required. If woke-flattering psychological research is what they're going to see, what benefit is there in familiarizing themselves with the literature of numerous psychological subfields to make an informed judgment as opposed to just dismissing the hot new study they might see referenced in the morning paper telling them that The Science(tm) proves they're racist?
I think Scott is going a bit easy on Psychology here--I WISH the field were full of eccentric cranks! As it is the field is much too agreeable--but he's far closer to the mark than the hardcore haters.
And anyway, everyone "knows" Psychology is bullshit mainly because the phycologists told them so. So doesn't that put you in a liar's paradox?
Okay, established psychology is still mostly okay, lesson learned. But when the media holds up "psychology study finds weird thing!" I should still probably be skeptical, right?
You should be extremely skeptical when media publishes any article about any scientific or technical field whatsoever, be it psychology, physics, biology, structural engineering, or anything else. I would go so far as to say that the only item of value these media posts provide is the link to the original scientific publication (on arXiv or wherever); everything else is exaggeration and fluff.
The 'goodness' of science isn't just "how many studies have been shown to be incorrect", but also "how good are the processes for fixing mistakes when they arise." (Since that tells you something about how many studies might be wrong and we just don't know). I think some of the high-profile failures in psych are infamous less for the fact that they overturned some understanding, but rather because, once the fraud was discovered, the process for setting the record straight was long and torturous, with a lot of institutional inertia ("processes bad -> probably more is wrong that we just don't know about").
Whether the error correction mechanisms are any worse in psych VS other fields, and whether the cited tweets really reflect this, and, even if they do, if that's still an over-correction or not are separate questions.
In 2009 I took an intro psychology class in college (UC Berkeley). The course material was boring and easily replicable, but one of the class requirements was to participate in two psych experiments. I signed up for two, unfortunately I don't remember the first of the two, but the second one was meant to find something like the primal color groups people settle on if your society hasn't taught you what the different colors are.
To do this, they would read random strings of syllables and expect you to select which color most fits that string. None of the colors had anything to do with the syllables they were reading and I was basically just picking random colors. Apparently after each participant did the experiment, they would somehow use those responses to adjust which colors get shown to the next participant. Over time they expected some sort of convergence to the primal color groups.
I think there's a difference between what a field has established and what practitioners waste their time doing
In 2009 I took an intro psychology class in college (UC Berkeley). The course material was boring and easily replicable stuff like in this book, but one of the class requirements was to participate in two psych experiments. I signed up for two, unfortunately I don't remember the first of the two, but the second one was meant to find something like the primal color groups people settle on if your society hasn't taught you what the different colors are.
To do this, they would read random strings of syllables and expect you to select which color most fits that string. None of the colors had anything to do with the syllables they were reading and I was basically just picking random colors. Apparently after each participant did the experiment, they would somehow use those responses to adjust which colors get shown to the next participant. Over time they expected some sort of convergence to the primal color groups.
True, but what most people of think of when they think of psychology is Social Psychology. At least that's what I see most forwarded on Twitter and cited by journalists. But I do have some sympathy for social psychologists. I think the main reason they had such an issue is that it's really easy to replicate a social psychology study. An RCT is way more expensive and a natural experiment is basically impossible: all you can do is check their specifications (if you're lucky enough to get the raw data from them).
I think your point is fundamentally right, but "I asked Claude and it made up some numbers to justify it" is not going to convince anybody. How long would it take to do an actual analysis of those numbers?
(Claude's analysis could be right, but it's unwise to trust it with no verification.)
I doubt that all of these subfields of psychology you list here are reliable (i.e. functioning in a way where most of the central positive conclusions they report are true positives). Many other areas of the social and biomedical sciences have major issues besides social psychology. See, for example, the below linked article, from Megan Stevenson who's mentioned in the tweets you discuss, arguing that:
``most reforms and interventions in the criminal legal space are shown to have little lasting effect
In general, I think only a small percent of scientific research is highly rigorous. See for example this study arguing that `Most randomised trials are bad' and characterising only 8% of 1640 clinical trials reviewed as having `low risk of bias'.
Also, as you know, bias towards preferred narratives (political otherwise) is rampant all over science. I wonder what you think of this short popular article arguing that psychologists keep insisting on talking about `psycopaths' despite no evidence that there actually are people who have what are supposedly the definitive characteristics (and probably evidence against this if you did the analysis).
As one final case, see the below article targeting `psychology' broadly (I'm not sure exactly what type of articles were covered, but not just social psychology). They argue that:
`most claims in the literature are so critically underspecified that attempts to empirically evaluate them are doomed to failure — they are not even wrong'
This is based on reading psychology papers that were published in the registered reports publishing format and scrutinising the connection between the statistical analyses and substantive theoretical claims. Registered reports means the new publishing format, that arose in response to the replication crisis, where a planned study gets peer reviewed before data collection and then the journal pledges to publish the study regardless of the results if the authors follow through properly on the methodology agreed upon at peer review (or justify changes transparently). The study reports that:
`For most articles [...] we were unable to establish exactly which test results informed which hypotheses and how they affected the authors’ conclusions. Worse, in many cases not even the hypotheses themselves were unequivocally stated.'
Note that the articles examined in this study are all likely to be way way above the average quality of research within their own subfield, since all the studies were registered reports. It's a small subset of highly methods-conscious researchers in psychology who are into this sort of thing.
The idea that the lessons learned from the replication crisis only apply to social psychology seems like almost as much of an overcorrection as the overly sceptical narrative you're pushing back against.
>See for example this study arguing that `Most randomised trials are bad' and characterising only 8% of 1640 clinical trials reviewed as having `low risk of bias'.
Nope. You are being very misleading. The study you are quoting did say that only 8% of the studies they looked at clearly had a low risk of bias, but it also said that for another 30% the researchers were unable to tell how high the risk of bias was. So for 38% of studies there was either definitely no bias (for 8%) , or no clear evidence of bias(for another 30%). Of course 62% with evidence of high risk of bias is bad news, but nowhere near as bad as the impression your posts leaves with readers here, which is that 92% of studies have a high risk of bias. In your post you are doing the equivalent of p hacking: reporting only the number that supports what you want to be true.
Thanks, I can see how, in context, my comment might have misleadingly given the impression that the other 92%, rather 62%, had been judged by the authors to be clearly *not* `low risk of bias' rather than an `unclear'.
However, regarding the `unclear' studies this is probably mostly studies that didn't pre-register their outcome measures and statistical analysis plan properly. See below cochrane risk of bias tool that was used in nearly all the meta-studes synthesised in this article:
Under the `selective outcome reporting' category it says that a study could get an `unclear' becuase of (amongst other things):
`insufficient information to permit judgement of ‘Yes’ or ‘No’. It is likely that the majority of studies will fall into this category' (looks like they were optimistic)
Personally, I think not pre-registering a clinical trial properly is pretty bad. It points to researchers who are unwilling to follow, or disengaged from, fairly simple and easy to implement best practice methodological standards that obtain with the clinical trials research community.
That said, not pre-registering properly does not mean the study's result could just be dismissed or anything like that. Even a study classified as `high risk of bias' could potentially be excellent quality depending on how this classification scheme is implimented (e.g. if a study gets unfarily slammed for outcome switching even though the switch is obviously well justified).
Is it possible that the reason the replication crisis started in psychology is not because the field is unusually bad, but because it's unusually good? Perhaps other fields have worse problems, but they're not honest, curious, or self-critical enough to publicly admit it.
Extremely <chef’s kiss> for so many of the “waow lmao psychology is all woke HR ladies who hate data” to then just not go and look at what ‘the data’ says.
Like I think a lot of the people who are *extremely* eager to delegitimize the discipline of psychology as being 'not rigorous' and 'dominated by the woke therapyspeak of power-posing Gen-Z-boss-in-a-minis who abhor rationality and rigorous data-driven decisionmaking' (etc) don't actually ever go and look at what studies do and don't replicate and how the domains that trend toward not replicating relate to the discipline as a whole.
Many psych101 students (esp students who took the class some years ago) are likely to have their minds disproportionately captivated by the "bad science" parts, very similarly to the way those "cool result but doesn't replicate" studies got incentivised in the first place. It feels possible that the class spends 1/16 of the time on social psychology, but then 5 years later the students' remaining memories of the class are 80% social psychology.
In other words, I'd argue that pointing out X% of psychology research is on not-social-psychology is helpful for debunking "psychology research is dumb and broken" but not helpful for debunking "what people learned in psych 101 is dumb and broken".
The second tweet frames a dichotomy/conflict between "humanities" and "social sciences". Call me a physicist, but I've always treated these two as basically the same thing so I found this funny. Is this an outgroup/fargroup thing or am I just unforgivably ignorant? 😂
Ironically, "applied theology" may end up becoming a real discipline in the near future, at least in the sense Vinge used the term in _A Fire Upon the Deep_.
There is a pretty big difference between social scientists (like linguists, economists, psychologists, sociologists) and humanists (literary theory, any department with the name of a language in it or the name of a part of the world). There’s a few fields where social scientists and humanists coexist - anthropology, history, political science/political theory. But it’s not exactly a sharp line, any more than the question of whether mathematicians count as scientists, or whether biochemistry belongs in the school of medicine or the school of physical sciences.
Thanks, this makes sense. I think when I first learned the term "humanities", it was taken as separate from "arts". The only humanities at my school were history and geography. The curious among us were told that economics was another one, but we didn't do that at our school. So the only humanities I knew about were the borderline ones.
Doing some googling now, I realise this isn't how the term is usually used. Of course, if humanities includes the arts, then it's very different from social science.
Arts is usually separate! For some reason literature is usually in the humanities rather than the arts, while music, visual art, art history, film, dance, theater are usually in arts (when arts and humanities are separate). Lots of universities do some lumping, often even putting all of arts, humanities, social sciences, and natural sciences into a single school of “arts and sciences” or “humanities and sciences” or “liberal arts” (though the latter term sometimes doesn’t include sciences). This is common at universities with big professional schools like law, engineering, medicine, agriculture, and so on.
If psychology as a whole doesn't want to be tarred with the brush of the social psychologists then it has the responsibility to police its own. Placing reputational pressure on the general brand is a completely rational and socially productive way for wider society to reject obvious malfeasance. This is no different from people being able to sue a police dept for the behavior of individual officers or corporations can be sued if their employee harms a customer in the line of duty.
This is a good and healthy process and I think your attempt to rehab the reputation of psychology as a whole is premature and bad. Psychology as a discipline allowed this nonsense to fester under its watch and it was happy to accept the funding grants and increased attention when politically-correct nonsense was dominating the headlines. I'm sorry but the field happily endorsed propaganda and I don't think that pointing out that "hey it's not all bad" is an appropriate defense. The field needs to be held accountable. It needs to reckon with what it allowed to happen. Until it does then I will continue to think of it as more akin to charlatanry than science.
I'd like to see you address the generalizability crisis by Tal Yarkoni seems a lot more ruthless to the foundations of psychology research than the replication crisis.
Indeed. But it keeps replicating when people try to use it to produce results.
Something I've been interested in for a long time is the legal landscape of psychology. There are many laws against, for example, using psychology on other people. The actions are legal - people routinely do them, every day - but not if you call them "psychology", and maybe not if you call yourself a "psychologist". This is one of the innumerable cases where the law hinges on the use of magic words.
It seems clear that what happened was that people saw that "science" produced great, deterministic results, and that when somebody pioneered the concept of calling himself a "scientist" of basic human interactions, a bunch of laws were passed to prohibit "science" from achieving the same thing in conversations that it had in ballistics... or rather, to assume that this would happen, and prohibit anyone from applying the knowledge that would inevitably be produced.
Leaving aside the fact that there is no way to enforce such a prohibition, the mental model of what "science" means is badly broken.
It seems disingenuous to ignore the impact of people like Gino and Ariely on the wider world beyond the academic psychology literature. Both are/were professors in business schools in the US. Their impact on psychology literature proper is probably negligible and I wouldn't be at all surprised if experts in the field were onto them long ago. But they, and others like them, have introduced nonsense into the world that the unaware take as gospel. Overturning it then undermines general credibility in science.
Handwaving it away as 'not all psychologists' seems a little loose to me. And claiming that '[most psychology research] consists of good ideas and trustworthy research' seems to me bland optimism. Good ideas are much too rare in all fields of research - of the 100k papers per year published in psychology it's hard to believe that more than 1,000 are important enough that they'll be remembered in 20 years time. In the fields I'm familiar with, the pressure to publish is relentless and standards dropped through the floor years ago - not much of the material is outright wrong but a lot is misguided noise with no signal underneath.
The biggest problem is people *wanting* something to “take as gospel”, and wanting a field where more than 1% of published papers are memorable. Research has to be seen as much more risk-taking than venture capital. Being too strict up front means you miss out on most of your successes, even if the success *rate* ends up higher. It’s really not a problem that most published claims end up false, as long as randos off the street aren’t reading journals and believing that a single study proves it.
I would have thought there was near-unanimous consensus that researchers need to do more to make their research accessible to the man on the street, to have an impact on society, to progress some relevant problem. Open access and so on are surely intended to incentivise non-professionals to engage.
I have a PhD in maths, and worked fifteen years in combinatorics and algebra -- my own research was to satisfy my own curiosity, and while it connected to the real world generally, it was not directly applicable. It's a different discipline, of course, but my whole area was drowning in incremental papers and pointless generalisations (what if we vary this assumption and brute force the 3 smallest cases) and re-discoveries. It's exactly what you'd expect when people are incentivised to produce papers no-one reads.
Researchers have to be allowed to take risks. The tenure track discourages this greatly - to get an NSF grant, you have to be doing what the people on the panel think is important. (In my field when I was there, what was popular in MIT was the only thing funded.) Researchers also need to be allowed to report failings - they don't currently. And I'd argue that tenured professors who don't research should be assigned other duties...
I’m not convinced that the outreach component is a valuable thing. Math research contributes to the world not by random people on the street reading it, but rather by some amount of it turning out to add up to a larger pattern that later helps someone understand some generalization that then helps a physicist or sociologist or evolutionary biologist or someone else understand something in their field that then enables more discoveries that then eventually have some more direct impact. Incremental papers and apparently pointless generalizations contribute to that, as long as there is someone beyond the author who thinks it’s worth reading (this is what peer review is for).
Open access is sometimes said to be about letting non-professionals engage, but it should be more about letting outsiders generally (who are likely to be specialists in something else) read a bunch of papers that they eventually figure out add up to something that no individual paper was.
I'm broadly in agreement, but I think the number of papers in which single-digit numbers of people are interested is relatively high, and they don't contribute to much of anything. One of the research activities I enjoy is surveying areas of my field, going from initial developments in C19th to the current day. To a surprising extent (or maybe I'm just good at scoping survey topics), there are 2-3 ideas underlying 50-100 papers. I think that clarifying that is a worthwhile exercise, but I end up a little jaded refereeing new papers which rediscover things I've read about from 70 years before (this happens frequently when electrical engineers write about sequences, codes and designs). On one hand, it's good that they're engaging with the material; it's building expertise and maintaining interest in the area; but it's not contributing anything fundamentally new to the field. (Often I'll point out a result is basically known and the paper will be published anyway - standards have dropped in the past 10 years from 'novel' to 'not wrong'. Maybe that's a little harsh, but directionally true.)
I'd be surprised if other disciplines weren't broadly similar.
Yeah I think it’s definitely true that there’s growing pressure on referees to not gatekeep on the basis of novelty or interest if things seem correct, and I think that’s bad.
I think most research in most fields is not going to be accessible to most people, even if it has huge practical impact in the future. To write a paper for other experts in my field, I can't give the kind of background knowledge that would be necessary for someone from far outside my field to follow it.
Explaining your research to outsiders is worthwhile, but whether it's workable or not (and to what depth) depends a lot on what your field is. "I study how people and animals fight off viruses" is easy; "I study how the early signaling between antigen-presenting cells and naive T cells affects the production of different cytokines that determine whether the resulting immune response is mostly cellular or humoral, and this is all done in mice but we think it tells us some stuff about how it works in humans" is less accessible.
Or "I work on ways to keep the internet secure even when quantum computers become available" is kind-of accessible; "I work on bounding the decapsulation failure probability of bit-flipping codes to get a better security proof on some code-based KEMs" is less so.
I think it's a bit more complicated than that. A difference I've noticed between the US and Europe is that more people in the US strongly prefer black-and-white answers. Something is either 'good' or 'bad'. The don't like nuance and they don't like uncertainty. I don't mean this as only a political observation, but I do think that Trump communicates to at least some of these people in a remarkably effective way.
An honest scientist who says 'the data doesn't support any definite conclusion', which is logically equivalent to 'we fail to reject the null hypothesis' doesn't appeal to those people.
> 'the data doesn't support any definite conclusion', which is logically equivalent to 'we fail to reject the null hypothesis'
I don't think this is right. You can run many experiments, over and over again.
If you fail to reject the null hypothesis each time, "we fail to reject the null hypothesis" gets stronger over time, while "the data doesn't support any definite conclusion" gets weaker. This must mean that they aren't equivalent.
I'd rephrase Padraig's point this way (and hopefully I got it right):
People who are careful to speak only on what they know or have strong evidence for don't do as well in US media / political culture as people who make clear statements with apparent certainty, whether they really can back them up or not. (I expect this is true other places but I don't really know other cultures well enough to be sure.)
Scientists would ideally be careful to be very clear about what they knew/didn't know, what was likely based on the evidence and what was unlikely, etc. But that kind of statement doesn't make a good quote in a news story and doesn't get clicks and doesn't sound good in a political speech. What sounds good in those places is "We know X, and it's only the denialists and wreckers who deny X." And this sounds just as good when X = "life as we know it arose via evolution" (which is extremely solid) as when X = "there is no biological meaning of sex" (which is silly and crazy but ideologically useful to say). Just sounding confident while spreading your nonsense is pretty effective, as our president demonstrates every single day.
This selects for scientists, journalists, historians, detectives, economists, et al in the public eye who make very broad and confident-sounding claims when they should make much more narrow and contingent claims. The ones who are cautious and careful to be right don't get quoted or interviewed or upranked by the algorithm.
The null hypothesis is a theory of how something works. It *isn't* the absence of a theory. When you fail to reject the null, that is evidence supporting a definite conclusion, to wit, the null hypothesis. If you do it a lot, then that's a lot of evidence supporting a definite conclusion.
Thank you! That's a large part of what I meant. I had a slightly broader idea in mind: for the sake of example consider a study which suggests drug X treats disease Y in mice. If a businessperson is asked to interpret this, they will say that drug X is the next big thing to treat disease Y in humans, with the goal of pumping their stock price. A scientist will say that mouse models differ from humans, clinical trials will be necessary, but *the data doesn't support any definite conclusion*.
Supporting evidence is not a definitive conclusion. The scientist who explains carefully and comprehensively what the underlying hypotheses are, how these are violated in practice and hence how the outcome is hard to predict is not who the public wants to hear. Patients with disease Y want to be told the cure is imminent, not there's a 20% chance their lifespan will be extended by 3 months or more relative to a control group.
> of the 100k papers per year published in psychology it's hard to believe that more than 1,000 are important enough that they'll be remembered in 20 years time.
This is hard to interpret. Certainly more than 1,000 will be remembered in 20 years time, by the people who wrote them. But that is either being remembered for a reason other than importance, or being remembered for a kind of importance that I assume you don't want to count.
If we move to some kind of "objective importance", less than 1,000 papers are important enough to be remembered the day after they're published.
I didn't write that well - your second interpretation is what I intended. And the 20 years is arbitrary, but approximately the longest time scale in which you'd hope to see citations to a paper as a measure of its impact. Citations are a bad way of measuring impact but still the closest thing to 'objective importance' that is widely measured.
This is true in every field. Most papers will be forgotten, most that are important will be built on and appear at best in the references of later papers.
I think the "replication crisis" is two things masquerading as one:
One, science - not priming, not social psych, not psych, all of science - has a problem with various things like publication bias and overuse of bad statistics, which was traditionally solved by replication studies. Those don't get enough funding (that's changing but they still don't), so there's a glut of new findings for every paper checking if they hold up. Weirdly, psychology is one of the fields that from my amateur perspective seems to most enthusiastically publish replication studies, and therefore the failure of many results to replicate being found there first is no surprise.
I would compare the way that all the criticisms of economics as a legitimate field are basically things economists were saying first, and have been correcting ever since they noticed them.
The replication crisis has affected almost all fields of experimental science; social psychology is the only one that's had to squarely deal with it in the public eye. Classic experiments in biology and cancer research have failed to replicate; RCT success rates cratered after clinicaltrials.gov was mandated, phenomena that inspired entire sub-fields in areas like motor learning turned out to be just experimental artifacts, fMRI studies were plagued with false positives and P-hacking until upgrading their methods, you have written before about issues in genetics, etc. The replication "crisis" is a pan-scientific challenge spanning a related set of realizations around statistics, data processing, experimental design, power analysis, publication incentives, and open science (or a lack thereof).
I broadly agree with this post's message (psychology research is unfairly maligned) but the correct conclusion is that there are *many* fields that have been, or still are, affected by the replication crisis. The bigger and more rigorous ones have generally dealt with it more effectively, while smaller and more backwater fields have not even realized they have a problem yet. I think psychology on the whole does much better than average now: top psychology journals often publish studies that are either pre-registered, or involve a second replication phase within the same paper; this is still practically unheard of in many other fields.
It's worth keeping in mind that basically nobody over the age of ~45 came of-age in science with any notion that it might be a bad thing to keep collecting data until your p-values are below 0.05, or try a dozen different ways of binning your data, then publish the one that gives you a significant result. If you weren't reading Gelman's blog or Retraction Watch in the late '00s / early 2010s you didn't even realize there was a problem -- or if you did, surely it was "those other sloppy scientists," not *your* lab.
If the effect is smaller, but statistically significant, the follow-up study would be considered a replication. It's authors, though, would likely comment that if the actual effect size is as small as the one they found, the phenomenon is a lot less interesting and important than was thought, based on the original study.
A lot of the issues I have with social psych aren't in the portion that doesn't replicate.
Suppose someone writes a study:
"We surveyed prison populations. We discovered that they were much higher % men than the broader population! Clearly there is massive bias against men in criminal justice!"
When someone comes to replicate this study, it will replicate successfully. The result being measured is real.
The interpretation of the result being measured is much more questionable, and is not being replicated.
>"We surveyed prison populations. We discovered that they were much higher % men than the broader population! Clearly there is massive bias against men in criminal justice!"
This is a straw man. Studies are not this dumb. Can you find real examples of lots of dumb people replicating something in psychology that's obvious?
I think that was just meant as a toy example to show how things can be wrong. There's stuff like this all the time, though. I recently fell for statistics showing homeschoolers scoring significantly higher than other types of students. However, this doesn't account for the fact that bot all homeschoolers take these sorts of standardized tests, and if you are smart, you're more likely to have a good reason to take one of those tests, thus the statistics don't show the full story.
Not a psychologist, but dated a clinical psychologist. From what I gathered, there's a fundamental disconnect between how expensive testing new methodologies is and what resources can be marshalled for the purpose. As a consequence, studies will be woefully underpowered, which encourages p-hacking, traveling the garden of forking paths and other misbehavior. In my ex's case, she had what was by any reasonable standard an exceptional study in the field for her career stage. She preregistered the study, which at the time was new and trendy. Unsurprisingly, it failed to meet a clear bar of significance. I have no doubt without that it would have been able to launch her onto a much more prestigious path.
My takeaway here is that basically everything that isn't in the dead center of mainstream treatment hasn't been studied seriously enough to be confident in what the literature says. Even standard recommendations in major fields, such as the Gottmans for relationships, are built on a foundation so shaky that you can't lend it real credence. I think it's clear the human insights that practitioners have developed have value, but it's very difficult to say they've met a scientific standard.
This may be taking the point further than intended, but I feel there is a broader point here, along the lines of "If there is a large academic discipline with a massive number of researchers across a vast number of institutions and a deep history, it probably isn't stupid/crazy for some trivial reason."
The same reaction I have when people will try to dismiss all economics for "not realizing people aren't always rational all the time" or "they don't think inequality is real" or something. And economics is actually a discipline with some more or less universal assumptions. Anyone who says "All history departments are X" is even further off.
I hear people talk about the opinions you can never express on a college campus, and I'm reminded of the many courses (undergrad and grad) I took on the Middle East, Arabic, Islam, etc.-- which should be as full of taboos as it gets-- looking very much like a settler, and not only did I never feel constrained in the views I was expressing, I recall being actively encouraged by one professor (a Syrian refugee) to be freer with my criticisms of the piece we were reading (about an Arab traveler's impressions of the United States, which I noted were both racist and elitist).
This is not to say there aren't problematic trends in all sorts of disciplines and I'm not saying all disciplines are equal in how problematic these trends are, but the kind of absolutist remarks about giving up on psychology (or sociology or whatever) are pretty much always wrong.
I had to take Russian History to get any discussion of taboo subjects (in this case, the number of Jews in Soviet Communism*, around the turn of the revolution).
*This is considerably less "third rail" than discussing the movement in Weimar Germany in such terms, obviously.
You haven't expressed taboo subjects in school until you've been outright failed for expressing them (I know someone who got failed for expressing a viewpoint on Romania, the teacher's country). It's pretty much the definitional "what is taboo." (and any responses that include current frameworks like racism and elitism are almost by definition "not taboo" -- they're well within post world war II western civilization ---> if you were expressing blatantly racist opinions yourself, that might be taboo).
At an early stage in my psychology studies I heard a professor say “the less interesting a subject is in psychology, the more sure we can be that we are right about it.” I have found this to be largely true (and is perhaps born out by this article)
1) I did fMRI research around 2010 (as part of psychology studies) and the state of the field was truly awful, maybe worse than social psychology despite the 'sciency' veneer. P-hacking was the norm and practitioners had limited understanding of how the physical signals were converted into data.
2) I'm no longer active in the field of psychology, but from the outside the general quality seems to have genuinely improved since 2014 or so. Not universally, but there is a lot more awareness of the replication crisis and what kind of things need more rigour.
I remember a lovely article called "The Control Group Is Out Of Control". I seem to remember it casting a pessimistic shadow over existing science, on the basis of methods and publication standards being so flawed as to permit complete nonsense.
Now you're telling us everything is mostly OK after all?
Psychology outside social priming has not necessarily fared much better. For example, a recent study found that roughly 80% of psychology articles discussing a nonsignificant result interpreted it as evidence of no effect. Clinical psychology has its own well-documented problems with weak control conditions, small samples, researcher allegiance, selective reporting, short follow-up periods, and limited generalizability. And industrial/organizational psychology is hardly an independent comparison field: large parts of it are essentially applied social psychology.
You mention that psychology is widely agreed to be a sham, give tons of examples of why it is a sham, without even grappling with the main criticisms - much of the field is based on mumbo jumbo and the effects of therapy are consistent with the null hypothesis - and conclude that "Psychology Research is mostly Fine"?
There was a pretty useful research paper showing that outdoor exercise outperformed all forms of talk therapy. Just because you have "pretty well understood theory-facts" that are supported by the research, doesn't actually mean "the Research Is Bad."
What you're criticizing is the direction and money spent on researching talk therapy (and, trust me, that's not anywhere near the worst aspect of modern research -- there's plenty of third rails).
Well, yes, it's always nice to have a paper to point to that confirms my biases :-) but the point is that the strongest prior, the null hypothesis, hasn't been convincingly dethroned. Eg if I had to bet I'd guess that outdoor exercise, done carefully, ALSO has "no effect". And GIVEN the sad state of science in general and psychology in particular, I'm not sure what would even change this opinion.
The rest of the comment section is an indictment of science in general - but even within science, I think that the branch that brought us Rorschach tests and the superego has to be uniquely bad.
Evo psych is the bad psychology. It's essentially "never, ever provable" (because generally we're looking at "only humans" in which case you've got a Single Species and No Control).
Why do people get depressed? (Evo psych will tell you that it's adaptive, citing energy conservation).
Outdoor exercise, at bare minimum, fixes white people's melatonin issues (and gets them vitamin D), assuming they aren't exercising after dark -- proper amounts of melatonin aid sleep.
Otto Rank, once Freud's golden child, who was cancelled by the Vienna clique, was the first to criticize early psychology's scientific dream to exclude the subject's data. Of course, patient centered psychology went on to a useful development in the profession, but it is interesting that Rank still remains outside the circle. His critique extended to accusation that Adler, Freud and Jung were all mostly interested in establishing a new ideology; will to power, biology or collective consciousness. Ernest Becker's introduction to his 1973 book Denial of Death simply recommends to the reader that they read Rank. Terror management theorists Greenberg, Solomon, and Pyszczynski did make some substantive contributions but this principle of life remains mostly removed from contemporary psychology research.
In my corner of the world, there's educational psychologists who do things like controlled-randomized trials and their results pass both replication and the sniff test, and the other kind of education research called Scholarship of Teaching and Learning. Comparing the two gives me an immense respect for psychologists. Watching the second kind being totally wrecked by AI gives me an immense satisfaction.
The most culturally visible failure mode tends to become the mirror for the entire field.
Social priming produced clean headlines, political implications, and spectacular claims so its collapse became more legible to culture than thousands of ordinary, incremental findings elsewhere in psychology.
The popular narrative about the replication crisis misses a deeper insight about what this episode brought to the development of scientific rigor. Calling it a "crisis" suggests fields hit a wall of malfeasance unique to the 2000's or to specific fields like social psychology. Malfeasance has always existed in the literature. The replication crisis was named for what it is: published research that didn't replicate — not "people made up results." Non-replication doesn't require fraud, just a low prior probability of holding up, which can happen through p-hacking, forking paths, publication bias, underpowered studies, or motivated analysis choices. None of that requires anyone to knowingly publish something false.
It wasn't "we caught some bad actors and are working to close up the loopholes they found and uniquely exploited." It was a broad rise in intellectual hygiene — preregistration, larger samples, replication requirements, more scrutiny of researcher degrees of freedom. Bad science still gets published, and fraud hasn't disappeared. But in response to the recognition of the problem brought about by the replication crisis, we have new tools to use to help us get at the truth.
We shouldn't treat some earlier decade as a rigor baseline later work fell from. Ioannidis's point was never "things got worse". The problem with replication was likely always there, we just hadn't measured it. Science is better now not because one field, or one group of researchers was uniquely fraudulent in a unique time. The whole field learned about a systemic problem and learned to better prevent, detect, and correct it. Yes, some fields had more room for improvement than others, but forest plots, pre-registration, better statistical hygiene, etc. have all made every field better.
Well, I mock "the psychometricians who measure the genetic contribution to IQ." But they're still churning out their studies irrespective of what the geneticists are telling them (and still recycling a bunch of sketchy twin studies). We've all been through this argument dozens of times now on ACX, but I'll happily throw links at peeps who want to step up to defend the field in another round (especially since I've got more studies up my sleeve now). :-)
Don't forget the psychophysicists who study hearing and audition! It's not all visuals... Technically the field of sound perception is called "psychoacoustics" but when I testify in court I just call it "sound perception" because opposing lawyers think "psychoacoustics" sounds made up or paranormal.
"the Freudian psychologists who help you understand how all your dreams involve unconscious impulses about sex with your mother.
the Jungian psychologists who help you understand how all your dreams involve the the Divine Feminine engaging in mythopoesis by telling you to have sex with your mother.
the developmental psychologists who tell you that you and your friends are Kegan Level Five, but the people who disagree with you are underdeveloped psychologically-rigid Kegan Level Four stick-in-the-muds."
No need for reminding me, I was already sold on mocking Psychology before. I make an exception for the Buddhist monk FMRIs and similar reductionist research but anything in Psych that is "high level" or tied to "culture war values" enough to be influenced by the the views of the political side currently dominant in academia is very likely to be garbage.
The replication crisis was actually one of the best things that could have happened to psychology. Social priming didn't just have bad stats, it had a bad premise: treating context-sensitive social behaviour like a fixed physical constant, the way you'd treat retinal response time or receptor binding. That's why psychophysics and neuropsych didn't need this reckoning and social psych did. Once you drop the fantasy that social research can just mirror human nature back to us, more precisely as the studies pile up, you're pushed toward something like Gergen's future-forming research: work that owns the fact that studying people changes what they become, instead of pretending to hold up a neutral mirror.
In "everything else you love to hate - is in chapter 12 of 16. The other fifteen chapters are comparatively innocuous." 'fifteen' is probably a relic of an earlier draft and should read 'eleven'.
Would love to hear more on the Neo-Freudians - in common with many commentators I thought Jung had done work on personality types that had stood the test of time well.
My comment may make more sense in context. All of the universities in California are involved in different genomic research.
Berkeley genomic research "Our mission: Unlock the promise of safe, accessible, and powerful germline genomic engineering for humanity. Berkeley Genomics Project, an independent nonprofit"
Germline editing, if successful, hypothetically would permit someone to reproduce through incest without fear of genetic consequences, correct?
It *is* true that philosophy includes many subfields, some of which even have interesting things to say.
But it’s *also* true that most of the world means ethics when they use the word philosophy. This is simply a reality about word use, and you can bemoan it all you like, but it won’t change the fact.
In this same way, it feels like a debate-team move to make Scott’s complaint.
Yes, it’s technically true (and, for example, we’re all happy to use codecs that have been informed by the kind of technical measurements he describes make up “real” psychology).
But this misses the point, that what people *mean* by psychology, especially in this context, is social psychology.
Wriggling out of the issue on technical grounds, rather than conceding the essential point (albeit perhaps poorly expressed) is exactly why the university elites are so hated.
You don’t get to complain that the public no longer unconditionally throw money at you if your response to every charge of misbehavior is not “you’re right, and we have been doing this badly since 1965; here’s how we’re fixing it” but is instead “you’re so stupid you don’t even know the correct way in which to phrase your charge”…
Can someone explain the section “When you mock “psychology” or “psychologists”, you’re mocking” to me? It seems to be a list of obviously good things but then also a list of maybe fake things. What is the take away supposed to be?
"Social psychology fell into this trap especially hard because it naturally touches on hot topics of popular interest and political explosiveness."
I don't think this is necessarily the main reason for the "failure" of social psychology relative to other subfields. The hotness or popularity of a research question shouldn't automatically make it prone to false findings. I think the main culprit is the scientific credentials and epistemic detachment of those investigating the question. And social psychology just happens to attract more of these pseudo- or half-baked scientist types than other subfields. It explains why there was an explosion of interest in "hot topics" to begin with. The latter is the result of the former (low quality research minds).
💡 Du dogme à la science : pourquoi la psychologie moderne sauve des vies
L'article d'Astral Codex Ten (« Psychology Research Is Mostly Fine ») rappelle une vérité essentielle : si la psychologie sociale a souffert de la crise de la réplication, l'immense majorité de la science psychologique et de la neuropsychologie repose sur des bases solides, loin des théories spéculatives du passé.
Mon histoire personnelle en est une illustration concrète :
20 ans de déni dogmatique : Suite à un trauma causé par une intoxication involontaire (soumission chimique), j'ai été suivi par un psychiatre psychanalyste de l'École de la Cause Freudienne. Pendant deux décennies, la cause réelle de mon trauma a été niée, ramenée à du fantasme, et mes symptômes ont simplement été étouffés sous médication.
Le virage par la science : Il y a 5 ans, tout a changé grâce à une prise en charge en neuropsychiatrie. En traitant le trauma pour ce qu'il est réellement — une réaction neurobiologique et un dysfonctionnement du traitement de l'information par le cerveau — ce suivi m'a enfin libéré.
En conclusion : La psychanalyse orthodoxe a trop souvent médicalisé le déni au détriment des faits. Il est crucial de privilégier les approches modernes et fondées sur la preuve (neurosciences, TCC, EMDR), qui ne cherchent pas à masquer la douleur, mais à réparer ce que le trauma a brisé.
I'm a bit late to the party, but as an ACX grantee funded to look at reproducibility, I feel obligated to comment. Replications are very popular in psychology now, so we have a lot of data on reproducibility across different fields of psychology, and there is considerable variance. See https://metascienceobservatory.org/replications-database/by-discipline
Of course, none of this negates the main points of this post, which I think are valid! Too many people over-index on the replication crisis stuff.
> The replication crisis primarily affected the sub-field of social psychology, and most centrally the sub-sub-field of social priming research - the study of how seemingly trivial interventions can have outsized effects on behavior.
I think that this touches on one of the things that makes this field ore vulnerable to bad research and also to being "controversial": There is a dichotomy or scale: The "simplified liberal" position is that people do what society tells them to do. The consequence of this is that human behavior is infinitely malleable. The "simplified conservative" position is that people do what they are incentivized to do, that the social context can cause them to do thing but only because that behavior is somehow coupled to something they *already* want. This suggests that there are distinct limits to how much human behavior can be modified.
This then couples social psychology research to the more ideological of our political divisions, which causes the general public to care about the results more and tempts the researchers to obtain the results that they emotionally prefer.
Psychology is divided into the "nothing to see here" crowd and the rest of us.
We are still going through the replication crisis and have yet to go through the citation crisis which means that, on average, more than half of what a psychologist might be saying is not based on replicable results.
"he found no effect. I take some of the blame for this"
What blame is there to take? The whole point is that no-effect results are super valuable! I suggest that the stigma of reporting no-effect results is a huge driver of whatever replication issues exist.
We need more people paid to replicate bold claims being proud of reporting no results, not fewer!
Ya, I'd argue there's no shame in doing a replication study of something exciting that is likely false.
People usually call it 'debunking'. But it's neat regardless of label.
This is Scott's humor. The part about converting to Catholicism on the spot was also mostly humor.
are you sure about this? I saw some people speculate that it was humour, but did Scott confirm it was a joke? like, it's kinda lousy, unintelligible humour if that's the case... it's played completely straight, in a non-fiction article, which is deliberately documenting real events, to make a real point... so was all that about it being the first big discussion with his wife also false? Like in a Louis CK skit you'd know something can be false, because it's in a context where things are made up for humour, but something coming from a mostly non-fiction author, in a non-fiction article, telling it as part of other events expected to be true, seems like really, really bad communication.
Same thing here... are we supposed to assume it's humour, because someone familiar with the topic would know it makes no sense to "take the blame" for this, and that it'd make no sense for Scott to think it was bad? Same as it makes no sense to convert to catholicism on the spot if seeing a miracle? Like I don't know, the beliefs in the rationalist community are out there enough that it seems likely that he'd just actually have nonsensical notions like converting to catholicism because of a miracle, because of some galaxy brain argument he thought up... and the readers would learn to spot these kinds of attempts at humour based on how consistent he was, but if he plays it completely straight like this, it's extremely likely a tonne of people will just take it at face value, it can even end up coming across as a bad faith attempt to get plausible deniability for his out there beliefs by claiming it was some weird attempt at humour.
And further, if it really is humour, then are we supposed to assume that what he claimed in his "My AI Opinions" article about the simulation argument could also be humour? it's a short section, he plays it straight as usual, but it's also something I'd consider nonsensical and bad epistemology to believe, but again, I know some rationalist beliefs are out there, so I assume it's meant in earnest. Having these weird attempts at completely straight humour in between just gives extremely mixed signals
I don't know, if these are genuinely meant as humour, I think it's bad form
He wrote he never jokes. And i am his true believer.
Also, there are several post with a *strong* Christian vibe. And a good friend got very Catholic. Also his history of being fascinated by ... I'd call it 'magical history BS' as Atlantis. (He learned diving to check out some underwater structures.)
Looking forward to pray a whole rosary with him. Ubi caritas et amor, deus ibi est.
> Looking forward to pray a whole rosary with him.
Is this the catholic equivalent of going for a pint?
Catholic, not Baptist. The Catholic equivalent for going for a pint is going for a pint.
We drink wine, too. While Baptists don't do rosaries. ;)
Just seems Scott is a a teetotaller :(
Holy Mary, gebenedeit bist du under den Frauen.
Scott doesn't joke much.
But he is often witty.
"because someone familiar with the topic would know it makes no sense to "take the blame" for this, and that it'd make no sense for Scott to think it was bad? Same as it makes no sense to convert to catholicism on the spot if seeing a miracle? "
Yes, pretty much. You can assume that if Scott professes something completely stupid, like "blaming" himself for a replication failure, this is dry humor and he's counting on you to get that, UNLESS, the thing in question is a commonly held rationalist idea, like AI doom or something, which you may or may not consider stupid, but it is an exception to the above principle nonetheless. Maybe he was a little serious about Catholicism, but also overstating it for fun.
This post demonstrates why it is in fact important to be ambiguously humorous sometimes, especially if you suspect that people may care about what you think!
okay, why? hopefully not to laugh at the people that don't get it. I really don't see what it provides other than plausible deniability for the more idiotic claims made, and confusing people about what you actually believe. You can be witty without muddying the points you're making.
If whatever points are being made need thinking about, the listener should evaluate them for themselves, and in my opinion it shouldn't matter what the maker thinks.
Sure, if you were writing a chemistry textbook it would be out of line to invent new elements. But we're not talking about a textbook, just somebody describing their thoughts in the moment.
Is it? Mea culpa if so. The rest of the paragraph read like an explanation of why Scott chose to give that grant in spite of all the red flags that in retrospect should have made it obvious the result would be no-effect; hence my response of no, please, hindsight is always 20/20, we need more grants/research like this, not less. But maybe I misinterpreted.
We need more "replication research" sure, but if all we're doing is doing the "sexy" replication research, there's going to be a lot of people with head trauma (or maybe that's just German Sleep Researchers...).
[Yes, this is in jest. A slap to the head is part of the protocol!]
If I remember correctly the original study had some flaws that were obvious in retrospect*, so the no-effect results could have been predictable without spending money on the study.
(*"Obvious in retrospect" is rarely obvious, and I agree that no-effect results are valuable, and admitting that the study was unneeded in retrospect was also holding themselves to a very high standard for science, which I appreciate).
This xkcd feels relevant: https://www.xkcd.com/3275/
(Also, +1 to that having no results when trying to replicate something is an immensely valuable contribution to science.)
Well, it's valuable to the extent that the study would have been taken seriously otherwise. It can be a waste of time if there were obvious flaws to begin with. (Edit: or even non-obvious flaws that are nevertheless disqualifying and are deductible from their data or methods)
…demonstrably, at least one smart person took it seriously enough to pay someone cash money to look into it! That’s not nothing.
(Isn’t spotting whatever flaws can reasonably be spotted, without trying to replicate, the purpose of having a publication review process?)
That's true, but unfortunately bad papers still get through. And the likelihood that a paper that gets published was actually rigorous is some function of both the journal's impact factor and its tendency to weigh quality over flashiness of the results.
To be cynical, the publication review process is for shaping your papers to flatter the reviewers, and possibly for checking to see if your English is parseable by a non-expert (don't laugh, it's quite possible to be able to math perfectly well, but not English good).
Cynic notes that large scale submission of entirely faked AI papers is ongoing, and largely undetected. Peer Review is just as dead as it's ever been (Millikan's oil drop experiment was peer reviewed, wasn't it? Implausible research continues to get published to this day.)
I figured it was taking the blame for spreading the idea the first time around.
It's fair but I know the guy and Sasha himself said basically that the replication could have been avoided because a deeper read of the studies ended up leaving him with the conclusion that the whole thing was shoddy enough it wasn't even really worth bothering replicating. You have to pick what to use resources on and if a study claims something already unexpected and was also conducted in a clearly poor manner the chances of it being bunk skyrocket.
Like, we do we even give creed to studies that have not been replicated yet? Any study that has zero replications so far should just be considered proivisonal and not even worth reporting on. Or maybe we should always use the word "allegedly" like you do in a court until it's been proved (by an independt second or third study).
"New study allegedly shows X, but is yet to be confirmed independently."
If Psychology held itself to this standard I'd be onboard calling it a science.
I would add two things pointing in different directions.
1. fMRI experiments in psych are also pretty bad on average
2. Having spent a lot of time mucking around in machine learning lately, I am not convinced social psychology is uniquely bad. Much of biology likewise. Parts of ML strike me as closer to actively dishonest.
I propose the conjecture that most research is just flawed in general (or at least should be suspected by default) because it's hard to get things right even if you're competent and professional, and there are silent failures that won't yell at you or your reviewers "hey, this invalidates the whole thing" etc, and because the existing filters for good vs. bad science are just not selective enough.
I know this is a crude first approximation and fields are likely different in that respect etc
Saying “most research is flawed” sounds like a damning judgment, but on reflection it’s probably exactly right - most of everything is flawed, even the best things. Not so flawed as to be crap, but it has flaws.
It doesn't matter how you excuse it. The question is: is everything mostly fine except that one subfield of psychology, or is everything "flawed" enough to be riddled with false/unreplicable results.
Are the results true or not? That's what matters.
My bet is on the latter: most fields, perhaps all fields, are have plenty of results with subtle flaws that invalidate some important points/takeaways of the paper. And it's mostly not due to lack of competence and diligence on behalf of the authors: it's just that the process is really difficult and the filters are inadequate, so it's the business of the entire community to double- and triple-check each other's results, but attention to replication is a scarce resource and some results don't get it.
How dare you mock the fMRI experiment with a dead salmon! Now we have the TRUE neural correlates of piscine after-death experience!
That one was actually mocking the others!
What parts of ML strike you as closer to dishonest? Are there specific claims or sorts of claims that you think we should disbelieve?
At least as of 2022 or so I found the literature on deep active learning to be nearly entirely useless
Lots of ML people don't know their stats, so they publish results that are just noise (eg a 1.1 pp improvement on a benchmark with ~50-100 questions) or they aren't rigorous about giving their "baseline" comparator a fair shot -- e.g. they tune hyperparameters for Our New Genius Method but don't tune them for Boring Baseline Method.
ML is also not really peer-reviewed so it does suffer more from oversights - my sense is that more of the knowledge about "oh the stuff from this paper or this group is mostly bunk" just lives in the unwritten knowledge within the field to a greater extent than other fields of science.
There's also just a huge amount of ML research, so per Sturgeon's law, there's a lot of bad ML research out there.
This is sort of a side effect of computer scientists presenting stuff at conferences and moving on, leaving the conference proceedings as the closest thing to a written record of what they did, without ever bothering to make something good enough to get into a journal.
'ML is also not really peer-reviewed so it does suffer more from oversights'
Why do you say this? All the major publication venues I'm familiar with involve peer review.
Because you can just post a paper to arXiv whether or not it's accepted at a conference. Preprint servers carry a lot more weight in ML than in other disciplines.
My impressions are somewhat different -- that preprint servers play a significant role in many disciplines now -- but I don't see an easy way to resolve it.
That's part of Scott's point, though--Psychology as a whole is not any worse than many other disciplines.
Exactly how bad that is, is not entirely clear. Not as good as we would like, but clearly better than "burn it all to the ground".
"Not any worse than many other disciplines" can be very bad. "Burn it all to the ground" bad, depending on one's outlook.
If one’s outlook is, better to have no progress than to have flawed progress.
When you write papers that are "known" lies, and do it over and over again, including paying people to write MORE lies (by faking research)... this is NOT progress.
... now, am I talking about psychology?
<mildSnark>
Well, I guess one measure would be how often researchers in a field display a reputation for honesty that earns them the dreaded "You should run for public office." epithet. :-)
</mildSnark>
I'm not really sure, is there actual numbers for this? Especially focusing only on public-facing part, since this is what makes public trust nosedive.
The public-facing part is a real difficulty. Psych has a problem that "harder" sciences do not, which is that you can fairly easily do shoddy work that generates widespread public interest.
Economics has a similar problem, but they have a much more combative culture and will gladly fight each other in the public eye. Psych has a very agreeable, very fragmented "let a thousand flowers bloom" type culture, so dubious public-facing communication goes largely unchallenged.
Well, sure, if you count the disciplines where the research gets cancelled by political pressure. (This isn't physics/chemistry and mostly not biology either). Psychology is about what you'd expect, given that post-World War II Western Society is built on a bunch of lies. Many lies are promulgated in "Psychology 101" -- some of them quite famous experiments...
There are a few fields where "burn it all to the ground" might be a good idea.
Marketing psychology's the obvious "the government should not only not fund distributing this, but should actively sin-tax it" where the field is of negative societal value.
There are other fields where the field is conceivably of positive value but could arguably benefit from tabula rasa - the history looked at by "grievance studies" is potentially interesting, but the people currently running and manning those programs are mostly useless for that and outright worse than useless if they're allowed to select their replacements (and many of them will not co-operate with training replacements they do *not* approve of), so it's plausibly better to just declare all credentials in the area void and try to rebuild that wing of academia from scratch in 20 years or so.
I know that ML academics can grind out low-quality/questionable papers, and sometimes they will massage the conclusions to make their results look deceptively "SOTA" (some people rightly push back against this "everything's SOTA" standard of publication), but ML research has an incredibly powerful test in the industrial applications. It's much harder to fake that your algorithm is working IRL and doing what the business needs. (Not impossible, of course, but if my models aren't providing anything useful, my bosses will notice.)
If your models are telling them what they want to hear and help them justify doing what's in the interest of their own careers (whether or not it's in the interest of the shareholders), it might not matter whether or not the models are accurate.
So...test passed?
When I was doing this at big corps and for financial companies, we very much measured the financial impact of the models our teams built, and this was communicated regularly and at high levels, and audited a couple of ways.
I feel like if it grounds out in actual revenue and / or profit, you can't be faking that much.
I think you can’t blame people for not making the distinction. Maybe you can blame the media for sharing the flawed attention-grabbing studies.
This “social priming“/social psychology/sociology subfield spread misinformation like no other field of academia and probably influenced politics to a significant degree. There are still midwits in 2026 citing the Stanford prison experiment as if this is a reliable result.
Those other “fine“ subfields of psychology just seem so minor in comparison.
Also many people are distrusting academia as a whole, not just psychology.
This is a good point. Social Psychology may have been 0.5% of all psychology papers but it had a massively outsized effect on the politics of the Second Social Justice Era (2012-2022) being the veneer of scientific respectability for many things.
The rest of psychology should be enormously angry at the few researchers who delighted in the attention and enthusiastically rode the tiger of feeling important in politics.
No different from how some small fraction of climate scientists abused their technical credibility to say that science says we need to do rapid decarbonization via X, Y pathways. It’s heady stuff to be on the national morning shows. But it undermines the entire field.
The rest of the field should jettison such types.
> There are still midwits in 2026 citing the Stanford prison experiment as if this is a reliable result.
The problem with the Stanford Prison Experiment is not that it fails to replicate, it's just that it was never clear what it was supposed to show. The main flaws aren't things that we only found out later, the flaws were on the page from day one.
What's the supposed conclusion of the Stanford Prison Experiment? That if you get a bunch of undergraduates and manipulate them into acting like dicks then they'll act like dicks? We knew that already! We have lots of evidence of people acting like dicks.
There's nothing in the experimental design that can tell us anything about why people act like dicks, except that they can at least sometimes be manipulated into acting like dicks under one particular set of circumstances.
The supposed conclusion of the Stanford prison experiment, as it was sold to the public, was that if you give ordinary people a bit of authority and freedom, they quickly turn into evil tyrants.
And obviously it was fraudulent since it was closer to some roleplay session organized by the experimenters since they were much more “hands-on“ than they initially let the public know.
Well, people can be manipulated to act like dicks is an important result for me, as the other hypothesis, "some fraction of people are dicks, no matter the influence" seems to be the default assumption.
Gino and Ariely were both professors in business schools, and I don't know what their standing was in psychology proper. I would have liked to see the interdisciplinary aspect teased out a bit more - I think there's a lot of bad interdisciplinary research, though you hear less emphasis of it now than 10 years ago.
When I taught a 'research fundamentals' course to business students, one of their projects was to analyse the original 'power pose' study. Most of them could see through it right away without any particular training in physiology or statistics. I primed them, of course. But that could not have gone viral without the complicity of the media.
I half agree. I think the emphasis on the heterogeneity of the discipline and the likely risks of extrapolating from one subfield. My sense is that the problem was, although maybe particularly bad in social priming, likely due to much broader norms around experimental practice, which likely apply more widely (indeed, not just in psychology).
Another point you didn’t make that I think is important here - social psychologists (not all of them obviously, but as a field) deserve a lot of credit for facing up to these issues and working to change some of these norms and practices. I would be more confident in the field now than I would be in some adjacent disciplines where no such reckoning occurred.
Science Banana isn't just objecting to the replication crisis, but to more general psychological methods; see e.g. https://carcinisation.com/2020/12/11/survey-chicken/ or https://carcinisation.com/2020/01/27/ignorance-a-skilled-practice/ .
Or since you mention personality psychology and the Big Five, it's worth pointing out that Science Banana is very intentionally addressing this as part of the problems with psychology: https://carcinisation.com/2020/07/04/the-ongoing-accomplishment-of-the-big-five/
That jumped out at me, too. I have long been annoyed at "MTBI bad, Big 5 good" discourse on the grounds that two of the Big 5 dimensions are fairly well correlated with MTBI dimensions, and two more correlate enough to suggest they're at least sort of measuring the same thing: introversion/extraversion is about the same in both (R^2 = 0.72), Openness is sensing/intuition, (0.77). Agreeableness correlates somewhat with thinking/feeling (-0.44) and consciousness somewhat with judging/perceiving (-0.49). Those correlations seem strong enough that if one of them is worthless nonsense then the other one probably is, too. The anti-MTBI discourse probably has a point that reporting the scores as a binary type on each axis provides a lossy signal compared to reporting the raw scores, but the scales themselves are either both nonsense or both have at least something to then even if they're different enough that one might be more precise or more applicable than the other. Or they might both be measuring different real.things that are proxies for one another, like how height and weight are moderately well correlated.
One thing that strikes me that I haven't heard talked about much is that Myers Briggs frames the dimensions in a relatively value-neutral way while Big 5 dimensions aref framed more to be good or bad. Especially neuroticism and consciousness.
Scott Alexander likes the MBTI: https://slatestarcodex.com/2014/05/27/on-types-of-typologies/
Thanks for the reminder. Yeah, he's not specifically the people I am annoyed by on this point.
The means matter as well as the ends in research: MBTI was developed by a mother-and-daughter team based on Jung, if I remember right, while big 5 used a regression on data collected (probably from first year psych undergrads). They're certainly measuring something, as is IQ. It's not clear that it's well-defined with cardinal directions. It's probably more like a surface in a high-dimensional space which can be projected down and measured in lots of ways and none is really much better or worse than the others.
Personally, I'm of the opinion that tools like this (when I'm provocative I list them as Big 5, astrology, Myers-Briggs and Harry Potter's sorting hat) can be useful for self reflection and analysis but they're not 'scientific', whatever that means.
Well, related to the point above, I read once that MBTI Ts are ~75% male and Fs are ~75% female.
And this always bothered me when I heard the standard invective against the MBTI; there is no way for a spurious dimension to divide like that.
Women seem more likely to sympathize with thieves and other "bad people" than men do (at least as seen in the fMRI studies) -- men see thieves and bad people getting punished as "karma" and women just get upset when anyone gets punished, regardless of whether or not they deserve it.
I think this is a good correlative study to "thinking" versus "feeling" (which are only going to be fun if you want to say "women don't think, they feel arguments, and make decisions based on said feelings." I've been talking to advertisers again, haven't i?)
I remember reading this a while back, and being aghast how bad this article was. I have not read this blog since
Anything in particular that you find bad about it?
Big Five is an empirical finding, not a hypothesis or theory, and so the very idea that it would be falsifiable is kind of weird and missing the point. You can't falsify a finding: you can fail to replicate it, fail to find it useful, or find exceptions and deviations which lead you to a better model. But to even think that falsification is the goal means you have misunderstood the nature of what researchers are doing.
There are many different theories people have proposed to explain why these factors continue to pop up, and none have gained consensus. One such hypothesis is that the traits are somehow "real." Another hypothesis is that they are just one particular way of representing the data. Currently most researchers lean towards the latter view, and that is also where I would put my money. But even for those who claim they are real, none are claiming the Big Five are exhaustive for what personality is. They are merely claiming it is a pragmatic tool.
Having misunderstood this, Banana goes on to cherry pick studies to critique the model. For example, I've never once heard "inspiration" discussed in the context of the Big Five, and yet he is treating it as if it were representative. But more representative is things like educational achievement, income, number of friends, how often you smile, mental illness, sensitivity to disgust. These are not nearly as vague as "inspiration." Meanwhile, other behaviors are cultural or political (mask wearing), and yet he ignores that Big Five would actually predict a negative correlation between mask wearing and conscientiousness for rightwing individuals. He also has a weirdly unempirical critique of surveys where he thinks they have no correlation to behavior, which is not a critique that even critics of the Big Five take seriously.
He also claims ALL the correlations are trivial, which is simply not true. Compare Big Five to any lay personality test, and you will find large differences; artistic appreciation is not correlated with introversion but with intelligence, there is no "fiery" personality type and the whole idea of types has little scientific validity, happiness is not the opposite of neuroticism but instead better correlated with extroversion, unstable emotions and sadness correlate very highly, motivation and disgust correlate highly, highly agreeable people lie a lot. These are genuine findings, and not intuitive. Even the correlations which are intuitive to people are often downstream of psychology popularizing the correlation.
The best part of his essay is his criticism of how general the factors are (though his point about non-weird populations seems overstated). But it is important to note that his points are well known limitations of the Big Five mode made by Big Five researchers themselves, not just their critics. Just because a model has limitations does not mean it is not useful. Here is a good summary of similar points: https://blog.arturnilsson.se/2022/09/01/five-myths-about-the-big-five-personality-model/
And it's not like I am a big defender of Big Five! I vastly prefer Mischel and Shoda's CAPS Theory to Big Five because of issues around the limitations or surveys, and the lack of theoretical foundation for the five traits. Typically, I am the core audience for critiques like this. But Banana's critiques are just too bad to take seriously. He claims Big Five is simultaneously trivial while also not correlating to anything in the real world, which is contradictory on its face. But more importantly, he seems to genuinely not understand the very thing he is trying to critique, the whole essay stenches of a hit job, not a genuine review of the strongest evidence.
(Sorry for the length)
Please tell me which of the following subfields of psychology (which is what this article is talking about as good psychology!) use surveys:
psychophysicists, psychopharmacologists, psychometricians, personality psychologists, experimental psychologists, cognitive psychologists, psychophysiologists, forensic psychologists, abnormal psychologists, clinical psychologists, industrial psychologists, child psychologists, behaviorist psychologists, neuropsychologists.
Hint: there's not many.
> When you mock “psychology” or “psychologists”, you’re mocking:
> the cognitive psychologists who <...> secretly laugh at you when you mess up.
Come on, those guys mocked us first!
Well played. He also mentions we're mocking child psychologists. How old do they have to be before it's ok to mock them?
... you haven't realized they're all "abnormal"? The blackhats are even weirder.
As if Scott should accuse someone else of being abnormal!
The psychology replication crisis did much more to me. Before, “science” was on this pedestal, similar to how HPMOR did it. A way to find truth. Something to be awed by. A path toward moving humanity forward.
Unfortunately, while I knew that was an idealized version, I didn’t realize how strongly the industry of scientific research tarnished that vision. Now, post replication crisis, even when “science suggests something” (covid opinions, flu vaccine suggestions, responses to spiking AQI in the Midwest, etc) I am immediately suspicious.
While engineering in the world of capitalism, or pharmaceutical companies may use science to develop something new, I’m no longer optimistic about any technology or knowledge to be obtained from universities or government funded research.
I’d love a few bullet points of government funded/university funded research that has been released in the last 5 years that isn’t just a headline. As far as I can tell, that large expenditure is vaguely worthless as the low hanging fruit of research has mostly been completed and if it were valuable for humanity, humans would pay (capitalist companies) to do it.
To be clear, the replication crisis on only social priming psychology shined light on the POTENTIAL fallibility of all other research. One bad apple. So while you might suggest it’s only 20% of psychology effected, I would argue that all research went from 90% accurate in my head-model to 60% accurate.
In theory, science should get credibility because of its ability to discard former mistakes.
In practice, every discarded mistake gets taken as evidence that Science Is Debunked (and the only thing you can trust is conspiracy theories).
>In theory, science should get credibility because of its ability to discard former mistakes.
In the abstract, yes.
In the particular, it's difficult to tell whether any individual theory is correct or one of these mistakes which will be discarded later on, which is unfortunate because most "science-based" policy is based on individual theories, not on "science" as a whole.
Stolen credibility, in the case of psychology. An overall low-quality field with untrustworthy results that are not gatekept. And, unless I missed it, Scott didn't even mention its possibly best proven subfield, IQ. Amusingly, this is instead violently rejected by many psychologists. So much for truth seeking.
That is not to say only psychology is guilty of plentiful trash results. The big replication crisis was in pharma and related, as I recall. Beyond that, many such cases!
You missed it.
> the psychometricians who measure the genetic contribution to IQ
Do research psychologists, by and large, disapprove of IQ research? I had the impression that psychologists themselves were mostly favourable to it, and that most of the fundamental criticisms come from outside the field. I don't have good data on this, though.
IIUC, they accept IQ in the abstract, while acknowledging that every IQ test is culture-bound. So they're probably good with a properly standardized group of test takers.
Actual research psychologists note wryly that we really don't measure IQ except in children, so we miss the decay of IQ as people grow older and their brains become less cognitively flexible.
In practice a lot of the experiments are done with such a complex and expensive setup that replication is impossible.
I think the sexy promise of science is to have something theories that's generally true but is wrong on the margins, and as we improve observations we can shrink those margins (even if it actually required massive foundational change in the background). This is how it works for physics. But this is certainly not true in something like nutritional science which keeps flipflopping between whether chocolate, wine, fat being good or not.
In other words, science theories should be converging. But some of them seem to be stagnant or even diverging.
Cocoa has tons of antioxidants. A friend of mine just got out of a single-blind single-participant study on cocoa's effects on bone regrowth.
Cocoa is very good for you. Too much fat is bad for you. These are easy things to prove. Wine studies are run by wine sellers, and do not show anywhere near the amount of antioxidants that cocoa has (cocoa is a serious outlier, compared to most other fruits and vegetables).
Andrew Gelman's "Why is the replication crisis centered on social psychology?" discusses how replications are cheaper in that field than others, and thus failed replications more likely to turn up https://statmodeling.stat.columbia.edu/2018/05/07/replication-crisis-centered-social-psychology/
So, given this, psychology should really be among the top fields in replicating and verifying results?
Current Science as a social construct does not reward such activities much, as is well-known. This is of course a problem, for example when the field has a lot of trash results. If we assume the 20% bad value, every journal issue should include 1-2 retractions of previous contributions and there should in general be a decent wait before proclaiming a result accepted.
Ideally, failed replications would be published in the same journals as the original studies, and they would then subsequently be cited more than the original study. Unfortunately, reality is not like that.
Indeed.
There are thousands of research papers that psychologists do every single year, and none of them are getting written up because the psychologists don't have the time to write them (or analyze them, honestly).
"As far as I can tell, that large expenditure is vaguely worthless as the low hanging fruit of research has mostly been completed and if it were valuable for humanity, humans would pay (capitalist companies) to do it."
There's still a lot of low hanging fruit in rare-diseases.
Pharma companies fundamentally don't care about any disease that only affects a few hundreds families around the world unless one of the families is the royals of europe or similar.
But there's a lot of nasty diseases like that. it's a long long tail.
I've seen people go all the way from discovery to miracle-cure.
A few folks from the Clinic for Special Children, world's experts on genetic conditions common in Amish and Mennonite populations, have launched an effort in exactly that space. Got the gene therapy for MSUD working in cows and mice; humans next, dear FDA? But market is too small for pharma companies to be interested in carrying that ball across the finish line. Worldwide, ~700 babies a year are born with the condition. Therefore:
https://www.plowsharetherapies.com/
Capitalism is bad at doing long time horizon research. If we are just a little bit more research away from something valuable, and we might have a product in 5 years, capitalist companies can do that. (But often the insight comes from universities anyway)
The longer term and more abstract something is, the harder it is for any one company to get the benefit.
Pharma companies have longer horizons than that, haven't they? Something like 10-15 years from patent to medicine, I believe, with many failures.
And if we take our favorite field of AI, private venture capital has funded it munificently for at least 15 years now, so much that the public is getting scared and shouting about bubbles on a daily basis.
Uncomfortable but true: Pharma companies are mostly right about this.
Developing a new medical cure costs around $1B, and that money is better spent on curing millions than thousands.
Not all cures have equal costs.
"Studies show", lol.
> I’d love a few bullet points of government funded/university funded research that has been released in the last 5 years that isn’t just a headline.
The GLP 1 drugs (Ozempic and similar) were developed based on government-funded research into the underlying biology. While that foundational research was generally done more than 5 years ago, the practical applications (medicines) from the last 5 years wouldn't have happened without it. A lot of government funded research is foundational with like that that takes a long time to reach practical applications, so it's too early to tell which recent government funded research will pay off.
> As far as I can tell, that large expenditure is vaguely worthless as the low hanging fruit of research has mostly been completed and if it were valuable for humanity, humans would pay (capitalist companies) to do it.
This only works for research with a clear path to monetization. Clinical trials are expensive, so companies will only fund them if they have the patent on the drug. For supplements and off-patent drugs, that incentive isn't there, so the research on which supplements are effective for what and new uses for old drugs mostly doesn't happen.
You should be less optimistic about markets.
I've seen hundreds of million of dollars go past me to things that will never, ever, ever, benefit a single human. Only shareholders. Of the things I've seen that will benefit humans, they were mostly sidelined in favor of things that were easier to sell.
Remember: the thing the market loves the most is whatever is the most like opium. Cheap to make, easy to sell, Feels good the first time, and you live long enough to have all of your value extracted before dying. If a company does something other than try to become more opium like, they are either not traded or having a gun held on them by the state or being irresponsible in the fiduciary sense.
>"I've seen hundreds of million of dollars go past me to things that will never, ever, ever, benefit a single human. Only shareholders."
Shareholders are humans.
Debatable.
Something released in the last five years can’t be known to be much more than a headline. It takes more years than that for there to be enough progress in the noise of research to become confident that something is a really productive new direction.
One of the few things I can think of in the past few decades that was obviously a big deal in less than five years was CRISPR. Another was the breakthroughs in LIDAR-aided archeology in South America. There are probably a few others in some other fields.
But in general, the present cutting edge shouldn’t be what anyone outside the field is paying attention to - instead you should be paying attention to the building consensus based on the work of a decade or two ago.
Ancient DNA has moved fast, to add to your list of examples. Weren't diffusion models also immediately important SOTA, unlike transformers?
It sounds a bit like you're "overswinging" the other way. It makes sense to be suspicious, but the conclusion "I’m no longer optimistic about any technology or knowledge to be obtained from universities or government funded research" seems like you're over-updating based on the data.
Science has always been imperfect. Philosophy of science has been mapping all these ways for decades, but also e.g. something like e.g. Ioannidis, J. P. (2005). Why most published research findings are false. PLoS medicine, 2(8), e124. (very influential, cited 15k+ times) shows how imperfect the whole thing can be. But imperfect =/= useless.
When it comes to research becoming valuable the term you're looking for is "academic spin-offs" - companies started out of academic research. See here for a report on them: https://content.dealroom.co/uploaded/2025/11/European-Spinouts-Report-2025.pdf?x94554 it's a market of 100s of billions which wouldn't exist without academic research.
Lastly, your statement "if it were valuable for humanity, humans would pay (capitalist companies) to do it." --> this isn't true. Companies will not take the risks that universities can. The famous story here is that e.g. a smartphone is filled to the brim with technology that would be impossible without fundamental research funded by governments (GPS, touch screen, the internet, etc.). Business is better at building applications than fundamental innovations/research.
The replication crisis is alive and well far outside of psychology. I work at an immunology CRO (animal models). Less than 20% of the results we get asked to replicate by clients (results from a single paper) actually replicate. Even after multiple attempts. (The number may be more like 5% - I haven't looked in a while, I'm trying to report a conservative percentage.)
We warn clients about that - sometimes they want to try anyway.
We are very good at this, and this is consistent with what other labs see. The problem is p-hacking and related bad scientific practice.
OTOH, if a result has been published as replicated, even once, usually it replicates when we try it. Results from some labs have MUCH better chance of replication than average (obviously good labs with careful practices; they are rare).
Just because something is wrong doesn’t mean it’s bad science! The point of science is to be wrong over and over again in ways that correct. 20% replication rate is still better than just choosing an idea that seems reasonable and testing it, even if it’s not nearly as good as you would like. But insisting on enough rounds of replication before even the first publication would slow down the sharing of knowledge.
Having worked and collaborated in multiple very different areas of science, I'm convinced the majority of science is essentially fraudulent. Maybe 70-90% of it.
Of that bad work, maybe 97% of the problem is insufficient rigor which is, in fact totally discernable beforehand, and 3% is outright dishonesty. I think this also varies enormously by field, psychology being one of the worst offenders that still qualifies as science. The variability by lab, institution, subfield, etc. is, I think what would most likely invalidate these estimates of mine, because one can only ever see a small corner of the world.
I'm sort of surprised by your responses in today's comments. You're a very smart guy, and philosophy of science is something you are professionally expert in, if I'm not mistaken. Do you mind if I ask how much experience scrutinizing actual scientists' work you have, from a correctness of empirical work perspective? I sort of expect the discrepancies between our opinions of science is based on in-the-weeds level, seeing-the-sausage-being-made impressions, so I'm curious.
In case it wasn't clear, as a professional scientist, I consider science to be in generally terrible shape, although of course I love the ideal.
I agree - the problem is mostly insufficient rigor (>90%), not outright dishonesty.
My experience is that the people who publish irreproducible results really believe that if they poke and prod at their data enough, they can turn noise into signal. They don't know they're p-hacking. They're just not smart enough to understand.
Which leads to my other hobbyhorse - we have too many scientists, esp. in biology. Being a good scientist takes a lot of intelligence. And intelligent people have lots of other, often more lucrative, fields to go into - finance, engineering, etc. There aren't enough smart-enough people to go around. But NIH and NSF funding ensures that lots of people who shouldn't really make the cut become scientists. I think this is why the *average* quality of science was higher before WW2 (before the funding). The total output is today much greater, and probably more *good* science is done now than then. But there's an awful lot more chaff to dig thru to find the wheat.
I believe what you're saying to be a problem. What would be a potential solution then? What about an increased knowledge of statistics and accountability for the "noise" in a lot of the research? But then it begs the question, who would pay for the accountability anyway? Lots to think about there. I'm not even sure if there is a legitimate, clear cut solution. I've heard a myriad of potential options put out in chemistry and biomedical science trade journals.
How does the NSF ensure that lots of people who don't make the cut become scientists? I know that universities and professors themselves are engaging in some sort of grade inflation and passing people on to the next course independently on the NSF, to my knowledge. Maybe the higher GPA and professors mentoring students to go on to the next level of education is the role the NSF plays.
Re solution, the best I've been able to come up with (been thinking about this for 15 years or so) are ways to make the reproducibility of published results affect authors' careers. One idea is a website where scientists can exchange info about published results and results of attempted replications, with careful tracking of credit/demerit. (I have a 25 page writeup from 2014).
Re how NSF (and even more NIH) funding does harm - by pumping money into the system without regard to how much money can usefully be absorbed. There are only so many good people (good in the sense of ethical and honest as well as intelligent) who want to do science/biology. If you pump more money into the system than needed to keep those people busy, you have to lower the bar for who gets in, to use up the money. This happens. The result is not just wasted money (that too tho) but the generation of *noise* which buries the signal of good scientific results. At some point more funding makes progress go slower, not faster.
For your first paragraph, isn't there already bioRxiv and the like? Granted arXiv is much more widely used in the physical sciences that don't overlap with biology (i.e. not so much for biophysics, biochem). Unless you mean registering clinical trials and the results, which is not such a bad idea.
W.r.t. to the second paragraph, how would we determine the ethical, honest, and intelligent people before they enter graduate programs? I guess programs already try to filter out based on GPA and letters of recommendation, but I'm not sure about moral character.
Maybe you're missing my point. The problem is not that most published results need 5 replication attempts (1 in 5 = 20%) before they replicate. It's that 80% of published results don't replicate at all no matter how many times you try, because the claimed effect is not real. If the authors had tried even a little to verify their own results they would have found that out, and not published.
But they didn't. This seems (here I'm offering the standard speculation) to be often because they know very well that most apparent new results aren't real, and that if they tried to replicate it, most of the time they wouldn't have a publishable result anymore. Grad students are routinely taught (in bad labs; i.e. most of them) to publish rather than replicate, lest they have nothing to publish.
There's every incentive to publish without verification, and every disincentive to scrutinize "interesting" results carefully for flaws before publishing. The result is that other labs waste astounding amounts of time and money trying to replicate spurious results. Interestingly, the probability of published results from commercial labs (pharma, etc.) replicating seems to be far above the average for academic labs - probably because they have less "publish or perish" pressure.
A related problem is p-hacking; you can Google on that.
Good labs (the few) don't publish until they're pretty sure they have something real.
I got the point that 80% of published claims are false. But you’re wrong to then say “if the authors had even tried a little to verify their own results they would have found that out”. Sometimes the experiment is so expensive and difficult that they don’t have funding to do another instance of it. And even if they did, sometimes the issue is some unrepresentative feature of the population they’re studying, or the equipment they have. It just isn’t valuable to have the first group do its own replication - you want it published so that someone else can do the replication, if the result matters to anyone (which it might not for a while, so it’s good to have it in the record for that future person).
What I said in a different thread is that there’s 1000 possible results that might be true, but only 50 of them work out once well enough to be published, and out of those only 10 are real enough to replicate. Publishing those first 50 is what you want, because insisting that nothing be published until it has that level of certainty means that everyone else has that much less to work on (because these 10 results aren’t available until years later, and the other 40 that don’t replicate often worked the first time because they’re adjacent to something real.)
Commercial labs have higher hit rates because they’re not working at the frontier of knowledge - they’re turning moderately well-understood things into commercializable products.
Most labs don’t publish things until they’re pretty sure it’s real, but that’s because you have to convince yourself to ignore the broader statistics to not get discouraged.
I think you're misreading the costs and benefits involved. My experience, as stated, is in rodent immunology work. Our lab runs hundreds of studies every year. We have an excellent idea of when a result is solid and when it's probably noise. So, I assume, does any experienced scientist in a competent academic lab (we have lots of people who came from such labs working for us). When an academic lab nonetheless publishes results they should know very well is unlikely to be correct, they waste vast time and effort in other labs attempting to replicate it. If you have a result you think is promising, I think it's your duty to check it out enough to at least think it's *probably* correct before publishing. But in fact that's rare, and it makes science incredibly inefficient. Yes, it takes some time and money, but it avoids wasting even more elsewhere.
What a carefully crafted article to evade political hotpoints, like chapter 10.3. Well done.
I think you're defending psychology from the wrong direction. The replication crisis paper (https://www.science.org/doi/10.1126/science.aac4716) really only tested social psychology. But like you say, similar (also modern) replication attempts, even in other fields, e.g. biology, seem to have equally dire results, or worse. So psychology is not necessarily in worse shape than the rest of science, it just happened to have an unusually well done, broad replication attempt, and it was in social psychology.
I don't think the correct conclusion though is that we should trust most psychology experiments, we should just be skeptical of all non replicated work, until this problem is more broadly solved in science.
I'm actually not aware of a broad replication attempt that didn't find massive failure to replicate, though last I looked deeply into this was two years back.
So I don't think your implication that an introductory psychology textbook is broadly correct, is true. We can't say that just because of an absence of evidence on replication, when the best available base rate is really bad
But it is a credit to psychology that it is the field where such an impressively methodological and broad replication attempt was done
> I'm actually not aware of a broad replication attempt that *didn't* find massive failure to replicate, though last I looked deeply into this was two years back.
There was this: https://www.bps.org.uk/research-digest/these-nine-cognitive-psychology-findings-all-passed-stringent-test-their (it described a preprint but the final paper has been published)
Quoting from that article:
---
A new paper currently available as a preprint at PsyArXiv has tested [...] nine key findings from cognitive psychology, related to perception, memory and learning. [...] all the effects in question were replicated on all occasions [...] The nine cognitive effects that showed this robustness were:
* The Simon Effect: Participants had to press computer keys to indicate as fast as possible the colour of a on-screen circle. They were faster to respond if the appropriate keyboard key was on the same side of space as the circle they were responding to. This facilitatory effect of "stimulus-response" consistency is important for the design of human-machine interfaces.
* The Flanker Task: Participants had to press the correct keyboard key as fast as possible to indicate whether a target stimulus was a vowel or consonant. Participants were faster to respond if the target was surrounded by distracting letters associated with the same response (e.g. a target vowel surrounded by irrelevant, distracting vowels), as opposed to being surrounded by distractors associated with a different response (e.g. a vowel surrounded by consonants). The task shows how we can't help but process irrelevant information to a certain degree.
* Motor Priming: Participants had to press the appropriate keyboard key as fast as possible in response to left- or right-facing arrows flashed on-screen. Preceding arrows (known as a prime) gave advance warning of which way the target arrows would point: sometimes these primes were accurate, which led to faster performance, as you'd expect; if the primes pointed the wrong way, they slowed performance. Crucially, some of the primes were "masked" to make them subliminal (i.e. not consciously visible), in which case the effects were reversed, with primes pointing the wrong way leading to faster responses. The finding shows how information that's not consciously perceived can affect our behaviour, and that it can have an opposite effect when subliminal than when consciously perceived.
* Spacing Effect: Participants were presented with dozens of words in sequence, with some of the words shown more than once in quick succession (i.e. the repetitions appeared close together in the sequence, known as "massing"); the repetition of other words was spread out in the sequence with other words appearing in-between (known as "spacing"). Tested on their memory for the words later, participants showed superior memory for the spaced words than the massed words, a basic phenomenon of memory that has important implications for learning and study.
* False Memories: Participants were shown sequences of words of related meaning. Tested on their memory of the words later, participants were more likely to mistakenly say that a new word of similar meaning had been present in the earlier sequence than a new word with a meaning unrelated to the earlier list. This is a basic demonstration of the fallibility of memory and how easy it is to feel like we've experienced something before when we haven't.
* Serial Position Effect: Participants were challenged to remember lists of twenty words. They later showed better memory for words that appeared near the beginning and end of the lists.
* Associative Priming: Participants had to indicate as fast as possible whether target words flashed on-screen were real words or non-words. They were faster to identify target real words if they'd been preceded by a word (a "prime") of related meaning, consistent with the idea that activation spreads through networks of related information in the mind.
* Repetition Priming: Presented with lists of words, some more common than others, participants had to indicate as fast as possible whether each word was a real word or a non-word. Some lists were repeated. Participants were faster to make their judgments when re-tested on a word shown earlier, and this priming effect was more pronounced for less common words – a finding that shows an interesting interaction between short-term memory (based on a recent encounter of a word) and long-term memory (based on how common the word is).
* Shape simulation: Participants were shown successive pairs of sentences and pictures. For each picture they had to say whether the object it presented had featured in the preceding sentence. Participants were quicker to respond correctly to picture-sentence matches when the orientation of the object was similar in both cases – for example, if the sentence "The ranger saw the eagle in the sky" was followed by an image of a flying eagle as opposed to a perched eagle. The finding shows how we automatically represent the visual images implied by sentences we've read.
That's interesting, I never saw this, thanks!
This exchange is kind of like the platonic ideal of an ACX comments discussion
Any field of science where (a) one result is more exciting than the other and (b) data points are hard enough to come by that you need to worry about p-values, is probably problematic.
This is not all of science. If you find yourself measuring the genitals of moray eels then your statistics are probably robust. If you study the mechanical properties of jamming in grain silos then you're probably okay. The vast majority of science is fine because it doesn't deal with these sorts of issues.
But if you study whether x is related to y, and it would be much more interesting if it is than if it isn't, and if gathering each data point is expensive so you don't have a lot of them, and the trend you're looking for isn't incredibly strong, then you are in serious danger of doing lousy science. At best you have a file-drawer effect, and at worst you're tempted to massage your data in whatever way it takes to get an interesting rather than boring paper out of your very expensive experiment.
That's a very good summary of the sorts of research that are more and less reliable.
Though I have to say, if my job was to measure the genital sizes of moray eels, faking data might seem tempting.
> That's a very good summary of the sorts of research that are more and less reliable.
I feel like Melvin's forte is precisely this - seeing to the crux or heart or dominant failure mode of something, and making a pithy comment highlighting it or summarizing it. He's high on my "ACX commenters to search for" list for precisely this reason.
If it's hung like an eel
And it has sex appeal,
It's a moray.
Bravo!
If a thing on the reef
Snaps at you with big teeth,
That's a moray!
Stick your hand in a crack
And you won't get it back
That's a moray!
This made my morning! Thank you.
The sad thing is that the more relevant research is to people's lives, the more likely it is to be inaccurate.
The shame is also that the media and/or the rest of the academy don't properly communicate how unlikely it is.
In contrast, when there was the possibility that the EM drive worked (or the superluminal neutrinos) there was a lot of excitement that we might be on the verge of discovering a new understanding of the universe while at the same time there was a lot expectation tempering about how this is probably just some strange measurement error.
That we don't have the same happening is why we should trust stuff like this less.
I always joked to myself that it was way easier to get a good statistical analysis on the movement, trajectory, and potential collision course of some asteroid in a distant celestial body several hundreds of thousands of light years away then it is to get good data on the correlation and causation of whether a monkey consuming potentially carcinogenic food would get cancer and other diseases over a long period of time.
“I'm actually not aware of a broad replication attempt that didn't find massive failure to replicate”
Not sure if you meant psych specifically, but if not, this study estimates 70% replication success rate in experimental philosophy:
https://www.researchgate.net/profile/Joshua-Knobe/publication/324496210_Estimating_the_reproducibility_of_experimental_philosophy/links/5c069ed6a6fdcc315f9c0f05/Estimating-the-reproducibility-of-experimental-philosophy.pdf?origin=scientificContributions
Even granting that many studies don't replicate, I think this is thinking about things on the wrong level.
Here are what I consider to be some of the most interesting few psych discoveries of the past few decades:
- Human behavior is sculpted by reinforcement learning controlled by a prediction-error-based algorithm in the nucleus accumbens.
- The spine, brain stem, and midbrain gate pain signals coming from the body based on attempts to predict their veracity
- Psychedelics work by overloading 5HT2A receptors, suggesting they have a role in some sort of construct like plasticity or rigidity
- Schizophrenia appears to be a neurodevelopmental disorder with a highly genetic component, possibly related to the immune system pruning synapses wrong and mediated by glutamate
- Depression seems to involve too little synaptic branching, especially in the hippocampus, probably downstream of BDNF in some way
- Most traits are about 50% genetic and 50% random, with seemingly little role of family environment
- There sure are lots of optical illusions, and that's because our brain makes complicated design decisions that balance speed vs. accuracy.
I think most of these will turn out to be true. Even the ones that turn out false won't fail just because one guy forgot to adjust for multiple comparisons and a replication attempt proves it wrong.
You could say that I'm cherry-picking especially robust or good results, but I think that most of what psychology teaches us, or what the average person should know about psychology, consists of things like these, and not of the latest n = 10, p = 0.0499 study.
"I think that most of what psychology teaches us"
Is this a claim about what people in fact learn from psychology?
I do not think it is commonly known that e.g. "Psychedelics work by overloading 5HT2A receptors."
Instead, it's the flimsy social psych findings that are most popularized and influential in policy discussions.
I don’t think massive failure to replicate is a sign of a problem, depending on *how* massive. If there are thousands of promising ideas, and only 50 of them have a successful trial that can be published, and then only 15 of those can be replicated, then I’d say the field is doing pretty well, even though 70% of published studies fail to replicate. Insisting on waiting for the replication before you get any publication just means that other labs aren’t aware of the promising lead until much later, and you have a lot more undiscovered true effects, and you have less of a sense of who is doing productive work.
On the topic of intro psych textbooks, here is a recent study analyzing IP textbooks to see how they treat studies that have been replicated versus not replicated. In a nutshell, IP books talk about replication in the method section of course, and then largely ignore replication status of the studies that are cited and examined in the rest of the book.
https://journalofscientificexploration.org/index.php/jse/article/view/3485
Thank you for writing this, Scott.
And we should remind ourselves: who collected the data needed to critique the field? Who applied for funding, submitted the IRBs, and organized and ran the tedious studies to assess reproducibility and hectored everyone to improve their research practices?
Psychologists.
Are the sub-portions of psychology captured by the two concluding chapters of your textbook - disorders, therapy and treatment- also "healthy" or is it fair for the skeptics to be concerned about those as well?
Depends on the level at which you're thinking about them.
I think we'll find that a lot of our categories aren't natural, or that a lot of medications work less well (or have more side effects) than expected. I think the best science has already priced a lot of this in (for example, the psychotherapy consensus is that no particular psychotherapy works any better than any other, and it all seems to be a sort of active placebo caused by good relations with the therapist, although I don't think they'd frame it in those exact terms)
I also think we've made a lot of progress since the time when our response to people behaving strangely was drilling holes in people's skulls to let the demons out. I wouldn't be a psychiatrist if I didn't think we could do good work.
For a concrete example, the construct "ADHD" means much less than most people think (see https://slatestarcodex.com/2017/12/28/adderall-risks-much-more-than-you-wanted-to-know/), but people paying attention know the amount it means, it's much more than zero, and Adderall definitely works!
I've written slightly more about this at https://lorienpsych.com/2020/11/11/ontology-of-psychiatric-conditions-dynamic-systems/ and https://lorienpsych.com/2020/10/30/ontology-of-psychiatric-conditions-taxometrics/
I think I concur - "it all seems to be a sort of active placebo caused by good relations with the therapist"
But that's a pretty stark way to summarize a field. Oncology and hematology and pick-your-own-ology may be capable of understanding/treating a lot of associated ailments, but in psychology, our treatments largely fall to simply being a non-traditional-friend of sorts with the patient.
To the extent that's true, that seems a profound condemnation. In some ways, it's worse than mere replication. Replication is somewhat fixable in the near term if we want. And replication also rears its head in other fields. But this impasse on bold effectiveness is pretty unique.
Not that psychology isn't hard and thus force us into this situation, just that it lacks the solid-foundations that a lot of the rest of science does.
I feel like a barber will be fairly resentful if you're regularly taking up their chair for an hour.
Why? That just makes you a regular customer. Vendors of all kinds love those.
Regular customers must have a lot more hair than I do.
> In psychology, our treatments largely fall to simply being a non-traditional-friend of sorts with the patient.
>
> To the extent that's true, that seems a profound condemnation.
I don't think that should be considered a condemnation. If the right kind of social support is a necessary component of healthy functioning, and some people don't have that, then a sort of artificial substitute for that kind of social support might be exactly what a lot of people need. Conceptually, it seems to me like the same idea as an insulin pump being a "non-traditional-pancreas" for someone with Type 1 diabetes.
That's fine as far as it goes, but the field blithely continues with the charade that various therapies actually have non-trivial consequential content apart from the so-called "therapeutic alliance". Surely it's time to put up or shut up at some point?
I agree I am not very intellectually excited about psychotherapy. I used the "research psychology" term in this title to clarify that I meant psychology as the science of understanding the brain/mind, not psychotherapy the treatment modality.
I don't think failure to make intellectual progress in psychotherapy necessarily condemns our psychological knowledge. Wanting to make intellectual progress in psychotherapy is a crazy request! You can't cure cancer by talking about it, so our prior should be that you can't cure depression by talking about it either. The fact that you *sort of* can is quite impressive and I don't think we should demand further miracles from it.
I think psychology the science of understanding the brain is on a better intellectual foundation, which is what I try to describe here.
Psychologist here: I think it's true that the active ingredient in most general psychotherapy, i.e. psychotherapy that isn't targeting a circumscribed problem, is the relationship with the therapist. I wouldn't call that a placebo effect, though. Having a supportive person who likes you and is interested in you is genuinely helpful. Here's a simple example from my life: I am currently getting twice weekly physical therapy to remedy muscle weakness and muscle shortening that are left over from a spine problem I had surgically corrected. I often do my PT exercises at home on the off days, and I am sure I am more likely to do them because I think of my kind and friendly PT as a supporter. His presence in my mind helps keep me in a frame of mind where I am more likely to help myself. I often see similar things happening in the lives of patients of mine who feel understood and cared about. They take heart and start doing things that improve their situation.
And it is important to bear in mind that many people who come for psychotherapy are not at all easy to develop a warm, supportive relationship with because they are addicted to putting up a certain false front, or withdrawn, or extremely mistrustful, or reflexively nasty and sarcastic. And it doesn't help them much to fake-like them. You have to find a way past their bullshit in order to develop real rapport.
Also, there are some forms of psychotherapy where the active ingredient is a technique that targets a particular problem. Really thoroughgoing, thoughtful cognitive therapy is one example. Exposure treatment of phobias and OCD is another. Both of those treatments are hard to do right. They are genuine skills. And exposure treatment is also daunting because it is actually harmful if done wrong.
> Psychology is no different. It has flaws, but most of it consists of good ideas and trustworthy research
I'm broadly with you on the “Psychology is no different” part. But a commenter on the replication post, Jess Kerlin, pointed me to John Ioannidis's “Why Most Published Research Findings Are False” article: https://pmc.ncbi.nlm.nih.gov/articles/PMC1182327/
That article's title is rather bleak. Maybe psychology is no different, because other fields are also broken? What if we are more likely to notice brokenness in psychology because all of us are humans dealing with other humans all of the time? Very few of us pay enough attention to sedimentary rocks or sequencing bacterial genomes to complain about studies involving these thing.
That said, psychology does have specific incentives that make it less likely for studies to replicate. E.g. you doing hot Brain™ Science™, you can always say that a replication didn't draw from the same population as you did, and no one's going to sue you for a random badly run study the same way they might sue you for a badly run drug trial.
People do interact with other humans, but also, to replicate any psych study, you have to run an experiment about as large and involved as the one that found the finding. This isn't true in other disciplines (at least, the ones that don't mostly rely on experiments). Most econ papers can be replicated by checking the code and data, and if you want to make sure they hold up in another setting it's still easier usually to do something that other papers have already done. A bunch of linguistics research is much easier to verify (you can just ask a native speaker if something is correct) than to deduce the initial patterns, or document the vocabulary. I'm less familiar with the hard sciences but my impression is that the same is true there, like if you describe a rock as having some type of structure you can falsify it (by showing it doesn't do X thing characteristic of the structure) more easily than you can prove it (because there are other structures that don't do X thing, idk I don't know geology).
That being said, there are some other heavily experimental disciplines and subdisciplines like language acquisition, although I think compared to psychology it's actually much easier to notice if language acquisition predictions are true in the real world. I think they're probably most at danger for having a replication crisis, especially since they get less attention than big social psych studies.
Hi, I need someone to test a joy-inducing task (affective science here), interested?
Am I keeping my pants on?
Thank you - I was reading the comments specifically to see if anybody would bring up Ionnidis' work, and was surprised Scott didn't mention it in his article, in the sense that psychology's failure to replicate isn't terribly remarkable, because that's some huge percentage of studies across all science disciplines.
If you read the contents of "Why Most Published Research Findings Are False," it extends more broadly than psychology (although he's mostly talking about medicine,) but it also doesn't suggest that most of the *accepted knowledge* in most fields is false. We've gotten to a pretty dysfunctional place in terms of researchers' career incentives re: publishing, but "most published research findings are false" does not actually equate to most of the accepted knowledge in a field being incorrect.
I did read it and I do mostly agree re: accepted knowledge.
My main point is more like: what if other studies have their own equivalent of priming that we never hear about because very few of us have to deal with other fields the same way we have to deal with other humans — and acquire a model of their psychology one way or another through practice.
See https://slatestarcodex.com/2013/02/17/90-of-all-claims-about-the-problems-with-medical-studies-are-wrong/
Very interesting, thanks!
FWIW people often misinterpret the Ioannidis thesis (not saying you are); his point in that paper is that "false" includes a mix of (a) false positives caused by p-hacking, HARKing, etc, and also (b) false negatives because the study was too small to detect the effect, no appropriate power analysis beforehand, etc. Reading Ioannidis can get pretty doomy but on the whole I think medicine is at least marginally better than the average scientific field.
I think you’re misreading Ioannides if you think he’s saying most fields are broken. Most published findings being false is totally compatible with published findings still being our best bet for promising ideas. It just means you shouldn’t believe something on the basis of one study.
Your Ioannidis push is worth extending one more turn. The conditional "maybe psychology's no different because everything is broken" — taken seriously — doesn't just reverse Scott's defense direction, it shifts the shape of the discussion itself. The question "trust psychology?" may not be well-formed against a background of "everything is broken," because "trust" depends on an evaluator seated in some assumption base, and "everything is broken" removes the default seat. You semi-explicit this shift — worth articulating fully: not only "Scott's defense reverses" but "the question's well-formedness depends on tacit acceptance of a default assumption base." That tacit acceptance holds throughout this comment section — not because commenters don't realize, but because raising the point sits out-of-bounds in the discussion's game. Your "everything is broken" conditional has more force than its rhetorical function here typically permits.
"Social psychology fell into this trap especially hard because it naturally touches on hot topics of popular interest and political explosiveness"
On top of that, the cases where it fell hardest would have been far far more likely to be the kinds of cases where people **love** to quote the result at their political opponents. "CheckMATE republicans!" style.
So even if it makes up 0.5% of psychology it's likely to make up a vastly higher percentage of [the psychology that people throw in the faces of their opponents.]
So when their opponents watch a huge portion of the things they've had thrown in their faces for years crumble into dust it's not so unreasonable to also doubt the very foundation on which those were built and the departments that produced them.
Is psychology relevant with neurobiology (neuropsychology) so prevalent?
The content of brains is not what was assumed in the 20th C.
https://www.eneuro.org/content/7/4/ENEURO.0069-20.2020
> The content of brains is not what was assumed in the 20th C.
Aren't neurobiology and related neuro fields the reason why we know that?
I have already answered your question.
As a psychologist reading this ... sees the title "oh good! :) " ... read the rest "oh he doesn't mean me (in social psych) :( " ... I am between social and cognitive so I will just keep doing what I am doing straddling the cognitive side as much as I can!
I mean, it doesn't actually say that doing social psych is bad, just that many/most people doing social psych are doing it badly.
Not arguing the broader point of "some of this is probably fine", but I'll say that the rot extends far beyond psychology. A number of years ago, a friend asked on FB for help with "nonlinear curve fitting". At the time I was deep in the weeds of levenburg-marquardt analysis (follow-ups to this: https://danparshall.com/papers/parshall_2014_PRB89.pdf ), so I offered to help... and it was "I tried a linear fit, but it wasn't significant, so now I'm trying other things"... no mention, of course, of multiple-hypothesis corrections. This was the same person who also asked me to remind them how to take an average... as in literally, the arithmetic mean. They had a PhD in neuroscience :/
I've had to sit down with people and gently tell them they're p-hacking.
From the inside it often doesn't feel like it. When someone isn't confident with code and stats it feels more like they've fucked up doing the analysis somehow and need to find the correct test. A few hours or days later they've p-hacked their way to a test that yields a result without realising they've been multiple-testing the whole time.
That's understandable for a grad student. But if you're a postdoc, with multiple papers, and you're still doing that kind of thing, then your contribution towards actual Knowledge is very close to zero.
It's honestly amazing that we learn anything at all. But even p-hacked papers are probably right more often than not, and eventually the signal adds up.
with p-hacked papers at least we can do a funnel plot with all the barely-significant results to figure out the likely true value.
I suspect that generally, people who have PhD outside STEM mostly suck at mathematics.
Following the ancient tradition of division of labor, perhaps each of them should have assigned a mathematician to check their equations? Seems like a more reasonable approach than expecting that people who on purpose chose "not math" as their field of study will magically get good at math. Many of them are quite aware that they suck at math, they just can't do much about it.
> Many of them are quite aware that they suck at math, they just can't do much about it.
Many would suggest they should not be doing science at all in these cases, and instead should be doing honest work like tilling the fields (or serving our coffees, I guess, in this age) instead of "publishing stuff I don't understand just because" and literally polluting the epistemic commons, *reducing* the sum total of knowledge in the world instead of adding to it.
But I've thought for a long time that 30% of people getting undergrad degrees is probably 2-3x too many, because it dilutes the signal to meaninglessness.
Not everyone is a statistician.
Someone can be an amazing surgeon, even a really top notch biochemist while having near zero ability to run stats tests on tables of data.
They can however often offer an understanding of the subject being studied that typical statisticians lack.
"30% of people getting undergrad degrees is probably 2-3x too many"
But I'm sure yours is the good kind.
Sure, but I doubt many "top notch X's" are publishing absolute crap that they don't understand, and just running it through as many R packages as they can before finding significance.
Part of being "top notch" is understanding your epistemic limitations and the boundaries of your knowledge, and taking on collaborators on your papers to fill those, it's a time honored practice.
Manifestly, the people Viliam are talking about do not understand any epistemics, or the boundaries of their own knowledge, or take on co-authors, and are just polluting the informational commons because it's better than tilling the fields / serving the coffees for them personally (but not for the world, which was my point). It's the selfish / defect thing and polluting the commons I'm complaining about, not the fact that they didn't get a degree in stats.
A perhaps amusing example
"Erroneous conversion of gene names into other dates and other data types has been a frustration for computational biologists for years. ...
What is not funny though is having genomics spreadsheets suffer from auto-conversion of gene names like SEPT8, DEC1 and MARCH3 into dates, a problem first characterised in 2004. A 2016 article on this topic led the Human Gene Name Consortium to change many of these gene names to be less susceptible to autocorrect. Despite this, our work here shows that gene name autocorrect errors continue to accumulate in supplementary genomics spreadsheet files at a rapid pace."
https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1008984
yeah, spreadsheets should absolutely NOT be used for science. The famous Reinhart & Rogoff result about debt-to-GDP ratio was actually from a grad student who didn't drag the Excel formula down the full column (to their credit, they didn't throw the student under the bus).
But part of the reason this was found in the first place was because it's common practice in Econ for grad students to reproduce existing results. Would that other fields held the same standard!
People shouldn't be using gene names in big collections of data, full stop. Never mind the stupid Excel problem, the names themselves change too often to be useful. Gene IDs are the way to go.
/rant
Every PhD first requires a statistics PhD.
More seriously, I'd say the field has a long-standing problem with clearly explaining how to do things properly (as well as not doing things improperly) to the less involved practitioners. Though in this brave new world, perhaps ChatGPT can do the proper experimental design etc for us.
Stats professors have a weird propensity for being amazingly capable people while being the absolute rock bottom worst teachers...
I remember during an intro seminar for incoming Phd students they had a stats professor come to give a talk that was *supposed* to revolve around laying out how to contact people in the stats department for consultation and collaboration.
What he *actually* did was to try to use his hour long slot to give a rushed stats lecture where he threw a huge amount up on a projector without explaining what he was showing.
This may depend on country and university, so not sure how much my experience generalizes, but... people seem to underestimate how horrible are math teachers outside the STEM bubble. And statistics is the most difficult part of math that still somehow concerns non-mathematicians.
I took lessons at three universities. When I studied computer science, the math lessons were great, in general. Only statistics was like learning a set of ad-hoc rules, without really explaining *why*. I was generally good at math, but when we got to the part about confidence intervals, it just completely stopped making sense to me. I was like: "wait, isn't this actually the other way round?" But no one in my class seemed to understand it; it was just like: memorize this, pass the exam, you will probably never see it again. -- Decades later, when I read on Less Wrong about the Bayes Theorem, it finally clicked: it indeed was the other way, that's why it didn't make sense. (It's like confusing "X implies Y" with "Y implies X", except here it is "X implies Y with probability P" and "Y implies X with probability P". People understand that this is about X, Y, P; but they misinterpret the connection.)
When I studied economy, the math teachers there didn't have any deep mathematical understanding, only a few memorized formulas they followed blindly; and that's what they trained their students to do. (For example, they taught you how to calculate the determinant of the matrix, but they had no idea that if two lines in the matrix are the same, the determinant must be zero. They insisted on "calculating it properly".)
When I studied psychology, the stat teachers just told us to put everything in SPSS.
Seriously, people outside of STEM have no chance to get proper math education. There just are not enough people who know statistics well and can explain it to someone who is not a mathematician at heart.
I'd say there's probably no solution to this, but there's definitely a solution to this now!
Claude knows statistics like the back of it's hand, and can spend hours of clock time talking anybody, from any inferential distance, into understanding!
People are basically just choosing to be ignorant and pollute the commons at this point, when a smarter-than-Olympiad-gold math god sits ready to hold your hand to whatever depth you need, for FREE, 24/7.
Huh, you think it would be better to have no signal than to have a signal with noise? I think you’re making some fundamental mistakes about how significantly you can improve the ratio just by clamping down on a field.
If they're part of the ~20% of people that actually link the data these days, a weak maybe, because then other people / LLM's can scrape the data and use it more cleverly. But otherwise, no, they're just polluting the commons, and that's a big enough problem already, and is already getting worse post-LLM.
I personally think there should already be fully automated-to-rigorous-standards science benchmarks everyone should have to submit their papers to (with LLM's in the loop) prior to submitting to any journals, where the LLM's take your data, try to recreate it in epistemically rigorous ways, decide what they could rigorously conclude from it, and if it doesn't replicate, or if it's p-hacked, or if your abstract or verbal take is significantly different than what they think you can conclude, autoreject it.
Like this is something we can build in a few months, some scientists should already have built this somewhere.
You seem to have a conception of journals as a place where you go to read the final, unalterable truth, such that anything false in there is "pollution". I have the idea of journals as a place where you go to read interesting ideas that have some amount of support, such that keeping things out slows our access.
The problem, as you pointed out, is signal and noise. If the noise keeps increasing, the signal dwindles to nothing, and all the tailwinds are in the "more noise" direction (ie degrees expanding so ever less competent people are publishing, LLM's helping people get through bad and poorly thought out papers sooner and publishing, etc).
Which way do the incentives point? To publish as many spicy, interesting, surprising conclusion papers as you can, to max getting talked about and citations, to win better postdocs or a tenure path or tenure. So the incentives all point directly at "pollute the commons as much as possible," and now LLM's are enabling that at record scale.
If we don't do some sort of mitigation measures, we're going to lose the already-dwindling signal entirely, in an infocalypse of glurge.
Why did you co-author Redefine Statistical Significance if this is your view? That paper seems in tension with what you're saying her, since it seems to be pushing a proposal aimed at reducing the proportion of false positive claims amongst the positive claims being reported in journals?
Unfortunately, probably 20% of people do not link their data. This meta meta analysis of loads of biomedical science papers estimates a 95% (frequentist) CI of 1 to 3%. Based on this paper, there seems to be a lot more people falsely saying that the data is currently available in a repository then there are people who actually make the data available. The authors are too catious to draw this inference themselves, and the statistical estimates they present are not easily interpretable, but this seems to be the likely implication.
https://pubmed.ncbi.nlm.nih.gov/37433624/
This is the kinda thing that could be easily aided with Claude, by the preprint servers? i.e. arxiv.org and bioarxiv could have an agent that scans the incoming paper, looks for the "reproduction package" link, and actually tries to reproduce it. If it can't be reproduced, then a red flag on the abstract page is set (vs a green thumbs-up if the data *can* be reproduced)
personally I would have thought that "Neuroscience" counted as "S" in STEM, but I learned that day how mistaken I was.
Honestly even if AI never progresses at all beyond Fable, it will probably accelerate scientific learning just because each of them *will* have someone available to check their math (and gently tell them that they're p-hacking). And also automated pipelines to confirm all the analysis at scale before we send to review.
Can confirm.
The standard has jumped upwards the last few years just because the bots can reliably knock out basic data analysis scripts.
It's not like mathematicians are miraculously immune to the perverse incentives that are the root cause of the replication crisis. And I'm telling you this as a mathematician.
Gwern has some analysis of when proofs turn out to be false; it's not as quick as you might imagine.
There are many papers out there that have holes in their proofs, even when the results are likely true. And there are even papers where the results are downright false but have yet to be corrected. Or even whole subsubfields that are built on highly dubious foundations. I've met all of those issues during my PhD.
Gwern's discussion:
https://gwern.net/math-error
Includes a great quip: "The problem with wrong proofs to correct statements is that it is hard to give a counterexample."
A citation therein:
https://mathoverflow.net/questions/35468/widely-accepted-mathematical-results-that-were-later-shown-to-be-wrong
Did you ever see this before? It keeps getting worse as you read on.
https://diabetesjournals.org/care/article-abstract/17/2/152/17985/A-Mathematical-Model-for-the-Determination-of?redirectedFrom=fulltext
To be fair, this was indeed a valuable technique! The author just lacks priority by a few hundred years.
People who have a PhD *in* STEM mostly also suck at math. In fact, they even suck at the other areas of their discipline other than the one they work in.
A non-trivial fraction of the time, there are in fact statisticians who collaborate on projects that do anything statistically interesting.
But making mathematicians participate in other work would be a recipe for making mathematicians commit suicide.
Fortunately, there are incentives so that once in a while, someone inside a discipline gets a lot of credit for noticing that a way of calculating things lots of other people are doing is causing problems, and things get tidied up. It might be nicer if we had spare mathematicians to ensure the mess doesn’t get created in the first place. But you don’t want to stop everything until a mathematician takes part in every step. (And vice versa when mathematicians are writing about the math of elections or the math of galaxies and making stupid mistakes about those things.)
I’m sure that if you were reading blog posts by social scientists before the social science replication crisis, they would have told you that it’s a pretty respectable field. But the a-priori trust or distrust of science in the public doesn’t have anything to do with that, or with the actual quality of the results, which obviously the public can’t verify in most cases.
It has to do with the perceived robustness of the system and the perceived honesty of the people in it. The public wasn’t ever given any reason to believe that any other fields of science were immune from the issues that emerged in the replication crisis, so a-priori distrust still makes sense.
A lot of what the neo-Freudians started off was built on and still is used in some form today. For those that don't know:
- Adler's inferiority complex picked out something real even if his framing was flawed.
- Jung developed the introvert/extrovert distinction, which is everywhere now.
- Erikson's psychosocial stages of development are questionable, but children's development is still commonly put into these kinds of stages.
- Horney's breakdown of the different kinds of coping styles for neurotics is analytically correct. What other options are there for interpersonal behaviour beyond moving towards people, against people and away from people?
It's perfectly reasonable to have these four in a chronology of the development of psychological ideas a textbook.
I think sometimes naming those categories are either trivial, or harmful. Like introvert/extrovert where people just can't agree what either of those means, and I'm not even sure if official literatures even have united definition. Or even more, MBTI, which has been criticized over and over already. Sometimes we even need to rip off the old definitions from the root and make a new one.
It's certainly the case that Jung was using a different definition for the pair than is usually used in, say, current day personality research. And that could cause confusion. Still, there's definitely a real personality trait (or set of related traits) being picked out here. Even if in pop culture conversations a lot of things get confused and conflated.
> Erikson's psychosocial stages of development are questionable, but children's development is still commonly put into these kinds of stages.
But that's not an example of psychology research being "mostly fine". That's an example of psychology research being in such a terrible state that they've been producing fake work for many decades and still haven't realized it.
I meant that his work is probably wrong, but there are later formulations of stages of development that are broadly accurate or useful, so you could say Erikson was a forerunner here. Either way, fine to have him in a history of psychology section.
As a psychologist, I would emphasize that there is still a lot of indefensible research in almost every subfield of psychology. In the case of the flicker-memory effect, I think it is entirely reasonable to be curious and to try to replicate the effect in good faith using solid research methods. However, I think most examples of bad research do not originate from this kind of curiosity, but from misaligned incentives that promise greater status or better job opportunities for producing striking results. The field has done too little to eliminate these incentives.
I would also point out that other fields face the same problems but have barely begun to acknowledge them. Psychology looks reasonably healthy to me, roughly on par with fields such as economics or medicine.
100% agree, and I also think this makes people very complacent about replication issues. Thinking that poor replication is specific feature of psychology as a field will lead you to miss a lot fundamental issues with science (p-hacking, publication bias and misuse of bull hypothesis significance testing). Some fields have better standards than others (and in fact psychology has improved significantly over the past fifteen years) but no one can afford to ignore the problem!
Seeing someone saying "post-replication crisis" always makes me squirm. Like the crisis has ever ended.
I start a Psychology course(mostly because Philosophy wasn't an option) so I'm glad to be more assured of & intrigued about the subject by this article. Thanks.
I also think that since Freud and most Neo-freudians weren’t really trying to be highly empirical scientists but rather shape therapies, literature and self-inquiry in a complex way that reflects the human experience (and inexperience) it’s odd to cover them as part of a replication crisis.
The fundamental problem in psychology and so many academic subjects is that the incentivizes reward novel findings over debunking or rebuttals. Whether it is psychology or philosophy or women's studies good ideas/discoveries should be rare so we should want more rebuttals of novel claims than novel claims. The academic incentives favor the exact opposite.
This is why I advocate a policy that every academic journal (pure math/cs maybe excepted) should have a rule that every time they publish a primary paper they will also publish the best rebuttal/response to that finding submitted after 1 year and then again after 5 years (and maybe more if it gets highly cited). And for empirical disciplines maybe another slot for the first attempted replication. Critically it must be a publication in the same journal (and ideally the cite wouldn't even reveal it is a response until you track it down).
That will solve the problem that people do studies and everyone kinda knows it probably isn't true but the incentivizes discourage anyone ever actually showing that. Of course we need to pressure the journals to adopt it.
Maths are not immune to the issues that plague other sciences. During my PhD I've encountered a few examples of papers with holes in their proofs, or even results that were completely false. In fact, one of my own papers was published in a peer reviewed journal despite containing a glaring error that was somehow overlooked by myself, my PhD advisor, a couple of senior researcher and the reviewer.
I think the movement to provide machine-checkable proofs is good, though I'm not sure specifically Lean (or other current/older proof assistants) is the right way to go.
Yes ofc you get holes but the costs to benefits are very different. It probably doesn't make sense to make a correction a full blown paper.
What we need in math is a combination of arxiv with an ongoing discussion of the work so it always displays the current status and apparent issues can be flagged. But you don't need someone to go run a new study or write a counterargument -- errors are found by people trying to read the proof and mostly just need to have a place to be recorded.
I’m not sure that this is really such a problem. In most academic fields, the limiting factor is finding good new ideas at all. It’s generally better to have more good ideas, a lot of which are wrong, than to try to limit the number of good ideas out there while ensuring they’re all right.
The latter is more relevant for engineering and other implementation, but not for research.
I mean if good ideas are rare surely the important thing is to identify them and distinguish them from apparently good ideas no?
I mean if you think what is valuable is purely hypothetical claims about psychology -- hey you people might behave like X haven't tested it -- then we are making a massive mistake by asking for evidence or before publication. The valuable thing is the thing you might test not the fact it is true.
OTOH if what actually is valuable is the actually true idea because those are much rarer than the ones which merely appear to be true surely it's really important to distinguish the two kinds of ideas. In areas like philosophy with very limited theory (or actually I would say with limited theory we aren't evolved to find obvious) you are going to need lots of effort to prune out the incorrect ideas that look good when tested by someone whose career depends on a positive answer using small sample sizes.
And that matches what people in the field say -- they go to conferences all the time see announced results and don't update their beliefs at all because they know they are almost certainly false. So we end up with a literature which is pretty useless for making inferences about psychology because of the high rate of false positives.
If there's a field of 1000 possible ideas, 20 of which are true, there's some advantage in a process that identifies 50 candidates, 10 of which are true, and there's some other advantage in a process that identifies 5 candidates, 3 of which are true, and there's also some advantage in a process that identifies 1 candidate, which is true. It makes sense that the clinical side of thing wants to do the last of these, but I don't see any reason why it would be better for the research side of things to be at the second rather than the first.
0) I think ultimately we do research because in some sense we care about the answers and that does ultimately mean we really do get more value from knowing what has probability of being true at 90% than at 5% because we want to use the facts. Either clinically or just in understanding and thinking about the world. Unf humans is we really can't use a bunch of low probability discoveries to help us understand the world because we lack the cognitive capacity to sum up the contributions from all those possibilities.
And if psychology had two buckets for the different kinds of ideas that would be fair enough -- the problem that needs fixing is that it doesn't because the incentive to actually do the double checking to put ideas into that higher confidence bucket isn't there.
My proposal exactly creates that incentive. It would especially do so if you require any response studies be pre-registered -- since then when the authors reply they can just be like "well 90% of the attempts to reply actually ended up confirming". Yes pre-registration is generally nice but the problem is that even if you have a general database it's really hard to know what is testing the same things and there is too little incentive to do it -- organizing it as a response to a specific hypothesis in a pub with a guarantee the best response gets a pub creates both.
(specifically pre-registration is least useful when everyone is trying to claim a novel discovery not a confirmation).
1) It isn't clear that generating canidate ideas in psychology is very hard at all. I mean there is nothing that was hard to generate about the idea of power poses or stereotype threat or etc. These didn't clear any high theoretical bar like might be true in particle physics -- it's almost exclusively the experimental justification that is doing any work distinguishing good and bad ideas.
And that's just what it looks like to be a field that is more empirical and less driven by theory. Given the false positive rate is essentially fixed and that correct ideas should get more rare over time something eventually needs to be done.
2) Finally let's assume that instead of figuring out what ideas are really almost certainly correct we want to discover a large range of ideas that might be correct -- ie pass some minimum threshold even if most are still false. If literally every study was pre-registered maybe that makes sense.
But without that what you find out isn't what claims have a chance of being true but just the useless fact that enough people tried that *sort* of thing until a study gave you a positive result.
That's literally the least useful information you can imagine. I mean if you are thinking -- hey X seems plausible I wonder if it is true and you discover: either X is pretty plausible or it isn't but lots of people think it is and tried to prove it and eventually succeeded.
> Whether it is psychology or philosophy or women's studies good ideas/discoveries should be rare so we should want more rebuttals of novel claims than novel claims.
That is definitely not what we want; what we want is for spurious claims not to be made. Your way just causes an infinitely increasing deadweight loss.
The way to get that is to punish people for bad results, not to reward other people for trivial results.
The root problem here is that our policy is that "doing science is good" independent of any results. This has unsurprisingly resulted in all of the results being fake. In a context where someone must rely on the results, things work fine.
Typo: you have a double "the" in "involve the the Divine Feminine"
Typo, or subtle test of psychology?
Jungian slip
I am laughing at you for mis-naming colors, but thank you for this: You wrote it so I don't have to. Also, they used to pronounce her name Horn-EYE.
> the forensic psychologists who assess whether criminals are not guilty by reason of insanity
They shouldn't be mocked?
Having recently taken a forensic psychology class (so, y'know, totally unbiased information here): not really. They've developed things like risk assessment tools that can predict whether someone is likely to re-offend (they're not perfect, but they're better than going off vibes and much better than chance), done tons of well-replicated experiments on stuff like eyewitness testimony, and invented interrogation techniques that are more effective and less evil than what existed previously.
I don't know much about Not Guilty by Reason of Insanity, because that's a US thing. In Canada, there's two separate categories: Unfit to Stand Trial and Not Criminally Responsible. (The key difference between NCR and NGRI is that NCR is raised at the sentencing phase, after they've already proven that the accused committed the acts.) The forensic psychologist has to do assessments then. I don't know how you could verify them, though. There's no objective test of "is this guy crazy", so there's nothing to compare the professional judgment to.
Possibly of interest: https://slatestarcodex.com/2013/11/10/highlights-from-my-notes-from-a-forensic-psychiatry-conference-last-week/
See also: https://slatestarcodex.com/2013/11/10/highlights-from-my-notes-from-a-forensic-psychiatry-conference-last-week/
Yeah I thought they're like those facial expression reader that shamefully is on TV in my nation sometimes.
This is a weird post to me because as far as I can tell the replication crisis is an issue in many other fields,. including "rigorous" sciences such as physics and mathematics.
One of the main causes of the replication crisis is the misaligned incentives that researchers face. Those incentives are basically the same in every scientific field.
Can you share an example from physics and an example from mathematics?
In physics, I can can only think of examples of the form 'we agree this number is ~1.6, but disagree whether it's ~1.61 or ~1.62' and 'we all agree this shadow is delayed. You think it's because light doesn't travel instantaneously, just really fast. I admit that would explain the shadow but refuse to believe light has a finite speed for other reasons'. The best I can come with in maths is some people have pointed out unjustified assumptions in some proofs, which have subsequently required an appendix to be added.
I decided to spot check this by asking Claude about a randomly-selected item from Scott's list of mundane, unaffected sub-fields: neuropsychology, because my d20 landed on 14.
It's a funny one, because traditionally it's not a field that does replication (in the normal sense) at all. They're not even trying to replicate! It would be unethical if they did, since it would mean, like, slicing up somebody's brain the exact same way as another lab did. You're not going to replicate Fineas Gage, that'd be evil.
But "it would be evil to replicate studies in this sub-field" is not the same as "this sub-field is an example of psychology succeeding at doing replication"
And Claude let me know that the modern fMRI version of neuropsychology has suffered a bad replication crisis of its own, due to bugs in the statistics software. I don't trust AI a whole lot, but the papers Claude cited to me do at least exist.
So I guess neuropsychology is a good example of "it has flaws"? But not a good sub-field to cite if you want me to credit psychologists generally with being trustworthy on the basis of successful replication of studies.
I think the overwhelming majority of structure-function research is very well replicated - for example, the role of Broca's area in speech, of the hippocampus in memory, etc, all the way down to the Nobel winning studies on the role of prediction error in reinforcement learning in the nucleus accumbens. Many findings are replicated on rats (RIP) and other primates. And in practice a lot of neuropsychology is testing - for ADHD, for concussion symptoms, etc - and that is "replicated" in the sense of the test reliability, specificity, sensitivity, etc.
I think this is the same error that this article tries to call out - engaging with fields entirely through one article in a popular science magazine about a replication failure, and then assuming that must be a significant portion of the field.
Separately, I think that some of the criticisms of fMRI are kind of unfair - see https://academic.oup.com/brain/article/140/8/e53/4032512 . The famous "dead salmon study" only worked because the people doing the demonstration deliberately avoided doing a multiple comparisons adjustment to make their criticism more attention-grabbing.
I think my engagement is more along the lines of epistemology and warning alarms for slippery rhetoric. For example, you say that, among others, "the role of Broca's area in speech" is "very well replicated". Cool! How? "Many findings are replicated on rats (RIP) and other primates."
I'm skeptical that the findings about the link between Broca's area and speech were replicated using studies of rats and other primates. The reason is that I don't think that those animals can talk. But I'm not an expert. Perhaps you're using specialized terminology that I don't understand. Either way, there's something slippery about those two sentences.
I do think that this particular sub-field, by ethical necessity, must have a special relationship with replication. Something like, "Ethics forbids us from holding human structure-function research to the highest scientific standards of replicability, and yet the discipline produces many reliable results and should be considered trustworthy," I think I would be happy to agree with.
But alarm bells go off when I see a sentence with a list of replicated findings, and then a sentence describing replication methodology, and there's at least one item in the list of findings that seems (to me, a non-expert!) like it couldn't possibly make use of that methodology, because rats and monkeys don't talk.
I'm happy for Claude to be wrong about fMRI studies. Cool.
The deeper issue is that, the more closely psychology approaches to subjects of study that people care about, the less ethical it gets to apply the highest scientific standards. This is tautologically true: "people care about this" equals "don't coldly manipulate it as part of a scientific study". It's great that they've replicated results about language learning and nerves and reaction time; but what we really want to know about, us normies, is why do people form lynch mobs? What was going on with the USSR? Stuff like that. And you can't replicate a lynch mob or a brutal dictatorship in a laboratory -- and even if you could, you shouldn't.
I think it's better to admit that psychology is on the horns of a dilemma, and say, "we can't design replicable studies for some of the stuff we want to study, and we shouldn't, so take our results as being as trustworthy as the methodology we were able to apply warrants". You just have to have lower standards for some subjects, and trust the results proportionally less, and that's okay. As long as you don't try to claim more trustworthiness than the actual methodology warrants.
There are human lesion studies, where they collect dozens or hundreds of patients who were brain-damaged (via seizures, tumors, accidents, whatever), measure exactly where the lesions are, and then quantify what the corresponding symptoms are. I find that lesion studies are often very informative. I would happily trade away dozens of fMRI studies in exchange for one lesion study.
There’s other kinds of (non-fMRI) evidence too: EEG, MEG, direct stimulation studies, TMS, …
Anyway, in my opinion, available evidence to date (especially lesion studies) prove beyond any reasonable doubt that Broca’s area has an important role in speech production.
A great recent technical reference on the neuroscience of language is "Wired For Words" by Hickok. I find that he’s unusually scrupulous in documenting what we can and can’t learn from different types of evidence, etc.
Because of what Steve mentions, I'd place high probability on the ability to trace a mental function to a brain area's involvement in *some* capacity.
But that's a very coarse finding. What's practical from that? If you see someone with Broca area damage, maybe refer them to the speech therapist?
The more useful knowledge is going to be less solid foundationally. Or at least requires more careful study to be equally reliable.
If someone has damage to x% of that brain region, how much recovery would they expect after Y months of Z therapy?
By analogy, I recall a few articles of Scott's about drugs to the effect of "This neurotransmitter has something to do with sadness, so we found a similar chemical to give sad patients and it kind of worked for awhile, then we discovered we were wrong about the mechanism" (Not an exact example) How does that kind of fine grained knowledge hold up in various domains?
This isn't a gotcha, I don't know how psych broadly compares to other research fields, but that's the kind of overview that would be persuasive.
> "very well replicated". Cool! How?
I can’t comment on Broca’s area specifically, but my grandfather wrote a book on the general principle.
Sadly, across a large country’s population, hospitals encounter large numbers of people with head injuries that involve destruction of parts of the brain. You can gather all of that information and correlate with observed symptoms, and given enough data assign some functions to some regions (this is awkward because the brain has some plasticity in the sense of being able to reuse surviving areas for some functions, and also some functions vary between different people or are not localised, and also there are dependencies, but with enough data you can work all that stuff out).
Meanwhile, new patients, sadly, do keep coming; so you look at where the damage is, write down your predictions, and when they wake up see how often reality matches what your map predicts. Other hospitals also do this. And thus you learn whether what you observed replicates.
TLDR: you don’t have to cause the damage in order to study the effects of damage. Plenty enough damage happens anyway. Replicate like an astronomer, not like an engineer.
> For example, you say that, among others, "the role of Broca's area in speech" is "very well replicated". Cool! How?
In that Broca's aphasics tend to have brain lesions there.
The fact that something is going on is well documented. I wouldn't consider this much of a win, though; "the role of Broca's area in speech" is the entirety of the finding. Normally, if someone said something like "we know about the role of X in phenomenon Y", you'd follow up with "what is it?", but you can't do that here.
<mildSnark>
>They're not even trying to replicate! It would be unethical if they did, since it would mean, like, slicing up somebody's brain the exact same way as another lab did. You're not going to replicate Fineas Gage, that'd be evil.
Re Gage, see https://www.astralcodexten.com/p/your-book-review-great-and-desperate
</mildSnark>
I originally read "the experimental psychologists who run through mazes to learn how memory works", and I like this better
"People act like the replication crisis has discredited psychology, that the field is now widely agreed to be a sham, that it can be uncontroversially held up as an example of science gone wrong."
Who does that? It's not as if the replication crisis were limited to psychology, but i don't see anyone claiming that biology or pharmacology (for example) are sciences gone wrong.
Several people in this comment section for starters.
Nice introduction, Scott, but where's the body of the post showing how the other areas of psychology have held up under scrutiny?
I assume most people naturally understand that things like classical vs. operant conditioning, long-term vs. short-term memory, the tendency of people to form ingroups and outgroups, the role of hormones in emotions, and the REM vs. non-REM sleep distinction are settled beyond much probability of being overturned by one new data analysis.
For other things, like the efficacy of antidepressants, or the genetic contribution to IQ, you can read the rest of ACX.
I assume you mean "most long term readers of ACX" (which to be fair does include me). I find, at least for myself, it's often surprising just how recent some discovery is, and I suspect many would actually have difficulty placing some category in the safe versus suspect buckets.
Anyway, science and psychology in particular probably aim for more specificity than "tendency to form groups" or "X hormone has a role in fear/joy/etc." and there are presumably follow up studies quantifying these effects. How do they fare? Etc. Maybe Non-social psych 101 holds up, but that doesn't necessarily give anything reassuring about the current state of the practice even if some of the ancient knowledge base is firm.
Obligatory XKCD: https://xkcd.com/2501/
Great post. Psychology isn’t bullshit. Social psychology isn’t bullshit either. The replication crisis is real, of course, but it’s been badly overgeneralized. Many areas of psychology are just fine — cognitive psychology, personality psychology, psychopharmacology, etc.
Two small corrections: (1) in social psychology, it wasn’t just priming studies that failed to replicate, and (2) developmental psychology also suffered from replication problems. Scott’s basic point still stands.
Nevertheless, the replication crisis was huge for philosophers and other academics who drew heavily on social and developmental psychology. It made heavy reliance on those fields untenable. My own approach shifted dramatically. Like, in my book A Better Ape, I drew on evidence from primatology, behavioral economics, archaeology, etc.
Initially, I thought this sucks. But it was actually a good thing. Philosophy that relies narrowly on social and developmental psych — indeed on any narrow set of empirical fields — was never any good in the first place, even if the studies themselves weren’t bullshit.
Philosophers should integrate multiple sources of evidence across fields. Like for the topics I’m interested in, social and developmental psych are relevant, but so are economics, political science, large-scale social surveys, computational social science.
The more convergent evidence, the better. Plus, some of these fields responded to their own problems by developing methods that better expose causal relationships (the credibility revolution).
So, from my interdisciplinary perspective, the replication crisis was initially a bummer, but the ultimate effect was positive.
There's a idea I like from Karen Horney-- that abusive, neglectful, or intrusive parenting can lead to irrational perfectionism. If being a normal kid isn't good enough for the parents, the child may invent irrational standards for themself. Has this been disproven?
At some point, I got a list of legitimate results from psychology. Unfortunately, the only one I remember is loss aversion. What else is good?
My impression is that the false results from social psychology that catch on are the ones which promise ways to control people on the cheap. Fair?
The Milgram experiment has been rather extensively replicated, most recently in Poland in 2017. Which doesn't exactly invalidate your impression -- it probably became famous because it promises a way to control people on the cheap -- but in this case the result seems to be true.
Eh I thought it wasn't so I skim the Wikipedia for it and seems like it's mostly not replicated?
Here are two modern replications. They've made some edits for ethical reasons, but they seem to confirm Milgram's results.
https://www.apa.org/pubs/journals/releases/amp-64-1-1.pdf
https://www.researchgate.net/publication/315065066_Would_You_Deliver_an_Electric_Shock_in_2015_Obedience_in_the_Experimental_Paradigm_Developed_by_Stanley_Milgram_in_the_50_Years_Following_the_Original_Studies
Wikipedia seems to talk about some criticism of the original study, both in terms of accuracy and validity, but I couldn't find anything about failed replications (except replications that deliberately changed aspects of the experiment, i.e. saying "the learner's health is irrelevant" was so obviously evil that compliance rates dropped).
You're looking at three different modalities (abuse, neglect, and intrusion), all of which have different failure modes (neglect may not be bad for a self-motivated kid), "intrusive parenting" may get a child who is failing to walk to do so "because mommy and daddy are worried". Aka: "these may be pro-social behaviors in the right circumstances" (abuse, not so much, but that's basically corporal punishment Done Wrong).
Parents who hold their kids to standards they cannot reach (whether or not they're rational) tend to get narcissists as children (where the kids "play pretend" that they're good enough).
There’s a ton of psych papers (and more every year) that unquestioningly assume a paradigm in which parents (and schools etc.) directly sculpt what kind of adult a child will grow into, like Bernini on clay, and approximately every single one of those papers is garbage, as proven by behavioral genetics (see §2 of my post https://www.lesswrong.com/posts/xXtDCeYLBR88QWebJ/heritability-five-battles if you’re not already familiar with that).
The methodological problems are usually: (1) neglecting the impact of genes, (2) neglecting the “reverse causation” in which (for example) there’s a child—bless his heart and love him to death—but he has an innate disposition that makes him a defiant prick, and therefore his parents will naturally yell at him more, punish him more, etc. (3) boring old p-hacking etc.
>”the forensic psychologists who assess whether criminals are not guilty by reason of insanity”
It’s actually a gigantic point against the field that there is an entire subdivision dedicated to fraudulently allowing violent criminals to escape punishment (at worst) and gumming-up the legal system with frivolous claims and appeals (at best).
Criminals don't "escape punishment" through NGRI. They're diverted to forensic psychiatric hospitals, on average for an amount of time longer than the original prison sentence.
And it's not "fraudulent" - in 70-90% of cases where criminals are found not guilty by reason of insanity, both sides (ie even the prosecution) agree that the criminal was insane. In about 85%, there was already a preexisting diagnosis of a major mental disorder like schizophrenia.
And it's not "gumming up" the legal system - it's a factor in fewer than 1% of all felony cases.
See https://digitalcommons.nyls.edu/cgi/viewcontent.cgi?article=2137&context=fac_articles_chapters for more information.
If NGRI isn’t any better than a guilty verdict, then why does anyone ever use the defense? Maybe unsophisticated defendants are being taken advantage of by lawyers who waste their client’s and the public’s time and money to no end, but I suspect that the defendants get some utility out of an NGRI verdict.
And I don’t mean “fraudulent” in the sense that there is some agreed formula that forensic psychologists are deviating from and lying about (though I have my suspicions about that as well), I mean that the whole concept of someone not being culpable for their own actions due to a psychological defect is incoherent.
Imagine for a moment that, “the gods made me do it” was a recognized defense in our legal system, and that astrology was admissible evidence. There would almost certainly be a small industry of very real astronomers who use perfectly legitimate science to testify in court as to the exact locations of various planets and comets at the date and time of the defendants’ birth. The fraud would not be that these testified locations of heavenly objects are wrong, the fraud would be that *none of this matters at all* and is a massive waste of time and resources.
Maybe you would disagree that this constitutes “fraud” on behalf of those astronomers. I admit I used the term a bit loosely, but Astronomy as a discipline scrupulously guards against any implication of paranormal or moral significance in their routine findings. It’s hard to imagine a science blogger defending the honor of astronomy by appealing to horoscopes.
Regarding the “1% of all felony cases” statistic, I’d like to see it weighted by lawyer billable hours. That would be a much better estimate of societal cost than number of cases.
Sounds to me like your picture of insanity defense cases was infuriating, but also really inaccurate, and that having heard the facts from Scott you are now trying to shore up the case for being furious. Why not consider the possibility that your picture of these cases was just wrong, and there's not much to be mad about? Hey, don't worry, there are still plenty of other things in life to be mad about.
> "If NGRI isn’t any better than a guilty verdict, then why does anyone ever use the defense?"
Many voters believe that, when there's a person with severe mental defects that are a factor in them being criminally violent, they should receive mandatory medical treatment and care.
In fact, such a view is an aspect of the majority of people's philosophy of justice. Our elected officials respond accordingly.
I understand that this can be upsetting and frustrating to people with very unusual philosophies of justice.
You seem to have misread the comment you're responding to as "why does anyone ever prosecute for insanity?", despite having quoted it and presumably having seen that it says the opposite of that.
Your comment is about what you want to do to criminal lunatics. How is it supposed to be responsive to the question "why do criminal lunatics ask for this treatment?"?
>If NGRI isn’t any better than a guilty verdict, then why does anyone ever use the defense?
I wonder about that, too. I've got three guesses:
1. The control isn't perfect. The statistic is that normalized by criminal charge, people found NGRI spend more time institutionalized than people found guilty spend in prison. We don't know if the NGRI people would have spent typical amounts of time in prison or not, and it may be that NGRI people would have spent much longer in prison than typical because of some combination of getting in more trouble while incarcerated, being less sympathetic during the sentencing phase, and being less likely to be offered light sentences as part of a plea bargain.
2. Related to #1, people who aren't found NGRI can still be committed to mental institutions if they're adjudicated to be dangers to themselves or others. If you're found NGRI, you're released when you're no longer considered dangerous, which averages longer than the typical sentence but might be shorter if you respond well to treatment. If you're found guilty and then institutionalized while serving your sentence, then you're stuck serving your sentence (unless you're pardoned or paroled) even if you're released from the institution before your sentence is up, and if you're still a danger after your sentence is up you will stay committed until you're not. So you're likely to wind up in mental institutions just as long either way, but with NGRI you have a chance to be released early.
3. Prison and mental institutions are different experiences despite both representing confinement and loss of freedom. It's plausible to me that some people may reasonably prefer going directly to the mental hospital rather than to prison even if they'd spend similar amounts of time in both.
> If NGRI isn’t any better than a guilty verdict, then why does anyone ever use the defense?
My understanding is that in certain jurisdictions, sex offenders, etc., can be found guilty, imprisoned, then sent to mandatory psychiatric treatment after serving their sentences instead of being released, so it's in their interest to skip the first part.
What do you mean “isn’t any better than a guilty verdict”? Are you replying to “ diverted to forensic psychiatric hospitals, on average for an amount of time longer than the original prison sentence”? It seems like someone might legitimately prefer years in treatment to years in prison together with a criminal record. And not just the person subject to the treatment, but others, if the treatment is of relevance to the persons condition.
There’s a ton of evolutionary psychology papers (and more every year) that unquestioningly assume the paradigm of “evolved modularity”, where e.g. there’s a specifically-evolved language algorithm, a specifically-evolved “intuitive biology” algorithm, a specifically-evolved “intuitive physics” algorithm, an “intuitive human social relations” algorithm, all somewhat intermingled for sure, but all innate, and in the cortex. I think that paradigm is pretty wrong for reasons here: https://www.lesswrong.com/posts/wBHSYwqssBGCnwvHg/intro-to-brain-like-agi-safety-2-learning-from-scratch-in (and see also a related argument in book form https://www.cambridge.org/core/books/beyond-evolutionary-psychology/4731C143A9557767DC37A59526F1F525 ). That doesn’t mean each individual result is 100% wrong, of course; often there’s some nugget of value wrapped that can be extracted from its confused interpretative framework.
I'm unencumbered by data on this, but I suspect that:
> the child psychologists who investigate how babies learn language / ethics / abstract concepts / etc.
are not going to fare well if mild scrutiny is applied to many widely-used methods.
Are you thinking of Piaget and Kohlberg? Curious what makes you skeptical of them - I feel like my children behave in about the way that the theory predicts.
I'm thinking of the stuff that relies on tracking the "gaze" of infants. I haven't looked into it thoroughly recently, but a quick search does show that there is some effort being put into doing larger / better studies.
I took a child psychology class in university. In it, the professor told us about a study showing that at a certain age, children who play with a large toy (a rideable toy car, a slide) and are then shown a miniature version of the toy, will then try to play with the miniature toy in the same way. My professor told us this was due to a lack of connection between parts of the brain involved in object recognition vs. planning physical movement.
A very impressive and surprising result, which has hundreds of citations (top 1% of research) and even has its own Wikipedia article: https://en.wikipedia.org/wiki/Scale_error
My professor then showed us a video of the experiment. This video: https://www.youtube.com/watch?v=OtngSHtz-cc
At timestamp 00:59, the researcher can be heard saying, "why don't you pretend like it's the big one and you can get in and take it for a ride." The narrator continues describing how the foolish child ignores the size of the car and tries to get into it. "They did not modify their behaviour with the toys, even though they could see that they were smaller."
Yeah, because the researchers told them not to!
There's a possibility it was just the film crew telling them to pretend so they could get some footage. But why would they have to do that if the effect was real?
Neither the professor of my class, nor the other students (as far as I know) caught the apparent fraud, nor apparently has the subfield of developmental psychology.
At the time I was already aware of the replication crisis, and this further affected my trust in science. Whether or not this specific researcher lied, they had an incentive to lie and easily could have, and I'm just supposed to trust them?
It's 2026 now. Video is cheap. I wonder if we should have researchers film all interactions with subjects (or even all data collection of any type) and upload them so they can be scrutinized. Faces could be blurred as necessary etc. Wouldn't work in some cases, but I think for most studies, most participants wouldn't even care that their face was going to end up in a database somewhere.
Less relevant to this discussion, but I just remembered I have also been a participant in a bullshit study. I think it was something about showing you a word, and then a quick flash of an image, and you had to click a button if the image was X, or a different button if it was Y (I forget what X or Y were). But as a participant, every image was very obviously a blend between two different images, one X and one Y. I guess the idea was something about determining whether the word would prime you to see a certain image, but it was always obviously both. I was never asked if I noticed more than one image at a time, and I never told them.
https://en.wikipedia.org/wiki/Scale_error
I wonder if cats are susceptible to this. I once watched my neighbor's cat attempting to stalk a deer in the parking lot.
At least according to Wikipedia, in the original study, children played with the smaller toys like they were big regardless of prompting.
All of Piaget's theories are his personal fantasies supported by nothing. That would be one of the first examples I'd reach for to support the case "psychology is a sham".
Parts of psychology aren't shams, but if you see Piaget mentioned, you're hearing about a fraud.
There’s a famous result, widely cited by textbooks etc., that very young infants imitate the facial expressions of adults (e.g. sticking out the tongue). I think it’s probably false, see http://www.replicatedtypo.com/sticking-the-tongue-out-early-imitation-in-infants/6082.html . That makes me a bit paranoid about believing other claims about infant psychology, but I dunno.
…That said, my assumption has been that the gaze-tracking methodology is basically sound. I’m open to correction, I haven’t really scrutinized it.
Of course, individual studies can still be bad and/or misinterpreted, even if the basic experimental paradigm is sound. For example, lots of studies draw flawed big-picture conclusions even if their data is fine, because they don’t appreciate that infant behavior is much more driven by the brainstem rather than cortex, compared to adults, see https://scholar.google.com/scholar?cluster=11144225329320125688&oi=gsb&hl=en&as_sdt=0,22
I enjoyed this article and was happy you brought up Freudian-Mother-Loving, which I believe is part of motivation underpinning genomic research in California.
I don't understand this comment.
A useful heuristic I found, comes from the Datacolada folks. Basically, if it's "duh we already knew that" then you can get away with tens of subjects per box. If it's subtle self-delusional effects like "smokers think smoking is less dangerous than nonsmokers", then you need *hundreds*.
https://datacolada.org/18
So now if I read something and it sounds suspicious, I check the sample sizes, and can immediately tell if it's worth reading any further.
PSA: The by far least reliable science is the surprising/titillating results that go viral in the media and online.
Whenever you see/hear "a recent study shows" in the news, it has an 80% chance to be wrong/misleading, according to a study of my gut feel.
I pulled out a 2002 psych textbook (Gray 4th ed), and used an RNG to pick page numbers, and then examine the topic discussed on the first sentence of that page.
(417) Talking about how fear-of-heights develops at the same time as self-locomotion in early human childhood, and why this makes evolutionary sense. Seems fine. ✓
(163) Discussion of Wernicke’s area. It’s not great, but it’s sufficiently vague and high-level that it’s basically fine. ✓
(327) “Sensory memory” including the phonological loop. Ditto, I think some of the details are a bit misleading, but it’s mostly pretty vague and high-level, so that’s fine. ✓
(111) Biofeedback training to lower blood pressure — I don’t know anything about this area. Skip that one.
(191) What does the word “lesion” mean and how can you create a lesion in an experimental animal? All good! ✓
(83) The more polygynous a species, the more sexual dimorphism, with males getting much bigger and stronger than females. And why that makes sense evolutionarily. All good! ✓
So for the 5/6 random samples that I feel capable of judging, the 2002 book is basically fine in every case. Didn’t expect that!
Strong skepticism of a field is a pro social heuristic for most people to have if the most influential work in that field is complete horseshit used to support destructive policies and elevate the most anti social segments of society
It’s better for most people to ignore a field, and to mock anyone who wants to use a single recent study from any field to make an argument, but not to actually be skeptical of the field itself.
Better for whom? Academics and other psychology related professionals who'd rather avoid well-deserved scorn for not speaking against or worse cheering on fake news because it flattered their ideological beliefs?
A virtue of heuristics is that it enables unsophisticated people to make judgments without much time invested or expertise required. If woke-flattering psychological research is what they're going to see, what benefit is there in familiarizing themselves with the literature of numerous psychological subfields to make an informed judgment as opposed to just dismissing the hot new study they might see referenced in the morning paper telling them that The Science(tm) proves they're racist?
I didn’t say people should familiarize themselves with multiple subfields! I did say they should mock anyone who wants to use a single new study to argue something.
There was a steady stream of anti-racism supporting studies, not only one. Misinformation peddled by the credentialed only ever goes one way - left. So why shouldn't the everyman simply dismiss those studies? Sure, maybe each makes only one specific argument, but many are generally connected, eg around the idea that injustice is why certain groups are incarcerated more than others
And the question remains: Better for whom?
Was there some specific effect supported by multiple studies? If you’re just flattening each study into “does it support racism or anti-racism?” then you’re probably not actually looking at the studies and shouldn’t pay attention.
We're back to my my first comment, which argued that most people should be strongly skeptical of psychology studies. Most people don't actually look at the studies injected into the mainstream and thus shouldn't think much of them. I said be strongly skeptical of them, but I could agree to your lesser recommendation of not paying them attention
Not just the stupid mind you, but also the midwits (see Redditors). So most people do in fact flatten studies into broad categories. My guess is that the tastemaking classes, basically all of whom are leftist, know this and this is why academic studies with anti-racism and environmental-determinism flavour were promoted
>Strong skepticism of a field is a pro social heuristic for most people to have...
And also later:
>A virtue of heuristics is that it enables unsophisticated people to make judgments without much time invested or expertise required. If woke-flattering psychological research is what they're going to see, what benefit is there in familiarizing themselves with the literature of numerous psychological subfields to make an informed judgment as opposed to just dismissing the hot new study they might see referenced in the morning paper telling them that The Science(tm) proves they're racist?
I think Scott is going a bit easy on Psychology here--I WISH the field were full of eccentric cranks! As it is the field is much too agreeable--but he's far closer to the mark than the hardcore haters.
And anyway, everyone "knows" Psychology is bullshit mainly because the phycologists told them so. So doesn't that put you in a liar's paradox?
Okay, established psychology is still mostly okay, lesson learned. But when the media holds up "psychology study finds weird thing!" I should still probably be skeptical, right?
Yeah.
You should be extremely skeptical when media publishes any article about any scientific or technical field whatsoever, be it psychology, physics, biology, structural engineering, or anything else. I would go so far as to say that the only item of value these media posts provide is the link to the original scientific publication (on arXiv or wherever); everything else is exaggeration and fluff.
The 'goodness' of science isn't just "how many studies have been shown to be incorrect", but also "how good are the processes for fixing mistakes when they arise." (Since that tells you something about how many studies might be wrong and we just don't know). I think some of the high-profile failures in psych are infamous less for the fact that they overturned some understanding, but rather because, once the fraud was discovered, the process for setting the record straight was long and torturous, with a lot of institutional inertia ("processes bad -> probably more is wrong that we just don't know about").
Whether the error correction mechanisms are any worse in psych VS other fields, and whether the cited tweets really reflect this, and, even if they do, if that's still an over-correction or not are separate questions.
In 2009 I took an intro psychology class in college (UC Berkeley). The course material was boring and easily replicable, but one of the class requirements was to participate in two psych experiments. I signed up for two, unfortunately I don't remember the first of the two, but the second one was meant to find something like the primal color groups people settle on if your society hasn't taught you what the different colors are.
To do this, they would read random strings of syllables and expect you to select which color most fits that string. None of the colors had anything to do with the syllables they were reading and I was basically just picking random colors. Apparently after each participant did the experiment, they would somehow use those responses to adjust which colors get shown to the next participant. Over time they expected some sort of convergence to the primal color groups.
I think there's a difference between what a field has established and what practitioners waste their time doing
In 2009 I took an intro psychology class in college (UC Berkeley). The course material was boring and easily replicable stuff like in this book, but one of the class requirements was to participate in two psych experiments. I signed up for two, unfortunately I don't remember the first of the two, but the second one was meant to find something like the primal color groups people settle on if your society hasn't taught you what the different colors are.
To do this, they would read random strings of syllables and expect you to select which color most fits that string. None of the colors had anything to do with the syllables they were reading and I was basically just picking random colors. Apparently after each participant did the experiment, they would somehow use those responses to adjust which colors get shown to the next participant. Over time they expected some sort of convergence to the primal color groups.
True, but what most people of think of when they think of psychology is Social Psychology. At least that's what I see most forwarded on Twitter and cited by journalists. But I do have some sympathy for social psychologists. I think the main reason they had such an issue is that it's really easy to replicate a social psychology study. An RCT is way more expensive and a natural experiment is basically impossible: all you can do is check their specifications (if you're lucky enough to get the raw data from them).
I think your point is fundamentally right, but "I asked Claude and it made up some numbers to justify it" is not going to convince anybody. How long would it take to do an actual analysis of those numbers?
(Claude's analysis could be right, but it's unwise to trust it with no verification.)
I doubt that all of these subfields of psychology you list here are reliable (i.e. functioning in a way where most of the central positive conclusions they report are true positives). Many other areas of the social and biomedical sciences have major issues besides social psychology. See, for example, the below linked article, from Megan Stevenson who's mentioned in the tweets you discuss, arguing that:
``most reforms and interventions in the criminal legal space are shown to have little lasting effect
when evaluated with gold standard methods.''
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4445710
In general, I think only a small percent of scientific research is highly rigorous. See for example this study arguing that `Most randomised trials are bad' and characterising only 8% of 1640 clinical trials reviewed as having `low risk of bias'.
https://pubmed.ncbi.nlm.nih.gov/35655288/
Also, as you know, bias towards preferred narratives (political otherwise) is rampant all over science. I wonder what you think of this short popular article arguing that psychologists keep insisting on talking about `psycopaths' despite no evidence that there actually are people who have what are supposedly the definitive characteristics (and probably evidence against this if you did the analysis).
https://aeon.co/essays/psychopathy-is-a-zombie-idea-why-does-it-cling-on
As one final case, see the below article targeting `psychology' broadly (I'm not sure exactly what type of articles were covered, but not just social psychology). They argue that:
`most claims in the literature are so critically underspecified that attempts to empirically evaluate them are doomed to failure — they are not even wrong'
https://osf.io/preprints/psyarxiv/8w2sd_v1
This is based on reading psychology papers that were published in the registered reports publishing format and scrutinising the connection between the statistical analyses and substantive theoretical claims. Registered reports means the new publishing format, that arose in response to the replication crisis, where a planned study gets peer reviewed before data collection and then the journal pledges to publish the study regardless of the results if the authors follow through properly on the methodology agreed upon at peer review (or justify changes transparently). The study reports that:
`For most articles [...] we were unable to establish exactly which test results informed which hypotheses and how they affected the authors’ conclusions. Worse, in many cases not even the hypotheses themselves were unequivocally stated.'
Note that the articles examined in this study are all likely to be way way above the average quality of research within their own subfield, since all the studies were registered reports. It's a small subset of highly methods-conscious researchers in psychology who are into this sort of thing.
The idea that the lessons learned from the replication crisis only apply to social psychology seems like almost as much of an overcorrection as the overly sceptical narrative you're pushing back against.
>See for example this study arguing that `Most randomised trials are bad' and characterising only 8% of 1640 clinical trials reviewed as having `low risk of bias'.
Nope. You are being very misleading. The study you are quoting did say that only 8% of the studies they looked at clearly had a low risk of bias, but it also said that for another 30% the researchers were unable to tell how high the risk of bias was. So for 38% of studies there was either definitely no bias (for 8%) , or no clear evidence of bias(for another 30%). Of course 62% with evidence of high risk of bias is bad news, but nowhere near as bad as the impression your posts leaves with readers here, which is that 92% of studies have a high risk of bias. In your post you are doing the equivalent of p hacking: reporting only the number that supports what you want to be true.
Also, you misspelled 'characterizing.'
Thanks, I can see how, in context, my comment might have misleadingly given the impression that the other 92%, rather 62%, had been judged by the authors to be clearly *not* `low risk of bias' rather than an `unclear'.
However, regarding the `unclear' studies this is probably mostly studies that didn't pre-register their outcome measures and statistical analysis plan properly. See below cochrane risk of bias tool that was used in nearly all the meta-studes synthesised in this article:
https://media.springernature.com/original/springer-static/esm/art%3A10.1186%2Fs13063-022-06415-5/MediaObjects/13063_2022_6415_MOESM1_ESM.pdf
Under the `selective outcome reporting' category it says that a study could get an `unclear' becuase of (amongst other things):
`insufficient information to permit judgement of ‘Yes’ or ‘No’. It is likely that the majority of studies will fall into this category' (looks like they were optimistic)
Personally, I think not pre-registering a clinical trial properly is pretty bad. It points to researchers who are unwilling to follow, or disengaged from, fairly simple and easy to implement best practice methodological standards that obtain with the clinical trials research community.
That said, not pre-registering properly does not mean the study's result could just be dismissed or anything like that. Even a study classified as `high risk of bias' could potentially be excellent quality depending on how this classification scheme is implimented (e.g. if a study gets unfarily slammed for outcome switching even though the switch is obviously well justified).
Is it possible that the reason the replication crisis started in psychology is not because the field is unusually bad, but because it's unusually good? Perhaps other fields have worse problems, but they're not honest, curious, or self-critical enough to publicly admit it.
Extremely <chef’s kiss> for so many of the “waow lmao psychology is all woke HR ladies who hate data” to then just not go and look at what ‘the data’ says.
Huh?
Like I think a lot of the people who are *extremely* eager to delegitimize the discipline of psychology as being 'not rigorous' and 'dominated by the woke therapyspeak of power-posing Gen-Z-boss-in-a-minis who abhor rationality and rigorous data-driven decisionmaking' (etc) don't actually ever go and look at what studies do and don't replicate and how the domains that trend toward not replicating relate to the discipline as a whole.
People like having something to sneer at. Social media has made us all sneer addicts.
Yes, definitely. “Now other people can SEE how superior I am!”
Many psych101 students (esp students who took the class some years ago) are likely to have their minds disproportionately captivated by the "bad science" parts, very similarly to the way those "cool result but doesn't replicate" studies got incentivised in the first place. It feels possible that the class spends 1/16 of the time on social psychology, but then 5 years later the students' remaining memories of the class are 80% social psychology.
In other words, I'd argue that pointing out X% of psychology research is on not-social-psychology is helpful for debunking "psychology research is dumb and broken" but not helpful for debunking "what people learned in psych 101 is dumb and broken".
Psych101 is teaching you "approved facts as certified by Western Civilization" -- they don't need to be true to be "valuable inculcation."
The second tweet frames a dichotomy/conflict between "humanities" and "social sciences". Call me a physicist, but I've always treated these two as basically the same thing so I found this funny. Is this an outgroup/fargroup thing or am I just unforgivably ignorant? 😂
Pfft, humanities, social sciences, physics, it's all just applied theology in my opinion.
Ugh
Ironically, "applied theology" may end up becoming a real discipline in the near future, at least in the sense Vinge used the term in _A Fire Upon the Deep_.
Prompt engineering _does_ seem to lean in that direction, for sufficiently powerful SOTA AIs...
There is a pretty big difference between social scientists (like linguists, economists, psychologists, sociologists) and humanists (literary theory, any department with the name of a language in it or the name of a part of the world). There’s a few fields where social scientists and humanists coexist - anthropology, history, political science/political theory. But it’s not exactly a sharp line, any more than the question of whether mathematicians count as scientists, or whether biochemistry belongs in the school of medicine or the school of physical sciences.
Thanks, this makes sense. I think when I first learned the term "humanities", it was taken as separate from "arts". The only humanities at my school were history and geography. The curious among us were told that economics was another one, but we didn't do that at our school. So the only humanities I knew about were the borderline ones.
Doing some googling now, I realise this isn't how the term is usually used. Of course, if humanities includes the arts, then it's very different from social science.
Arts is usually separate! For some reason literature is usually in the humanities rather than the arts, while music, visual art, art history, film, dance, theater are usually in arts (when arts and humanities are separate). Lots of universities do some lumping, often even putting all of arts, humanities, social sciences, and natural sciences into a single school of “arts and sciences” or “humanities and sciences” or “liberal arts” (though the latter term sometimes doesn’t include sciences). This is common at universities with big professional schools like law, engineering, medicine, agriculture, and so on.
If psychology as a whole doesn't want to be tarred with the brush of the social psychologists then it has the responsibility to police its own. Placing reputational pressure on the general brand is a completely rational and socially productive way for wider society to reject obvious malfeasance. This is no different from people being able to sue a police dept for the behavior of individual officers or corporations can be sued if their employee harms a customer in the line of duty.
This is a good and healthy process and I think your attempt to rehab the reputation of psychology as a whole is premature and bad. Psychology as a discipline allowed this nonsense to fester under its watch and it was happy to accept the funding grants and increased attention when politically-correct nonsense was dominating the headlines. I'm sorry but the field happily endorsed propaganda and I don't think that pointing out that "hey it's not all bad" is an appropriate defense. The field needs to be held accountable. It needs to reckon with what it allowed to happen. Until it does then I will continue to think of it as more akin to charlatanry than science.
Hello Scott,
I'd like to see you address the generalizability crisis by Tal Yarkoni seems a lot more ruthless to the foundations of psychology research than the replication crisis.
Elementary physics fails to replicate in undergraduate lab courses all the time...
Indeed. But it keeps replicating when people try to use it to produce results.
Something I've been interested in for a long time is the legal landscape of psychology. There are many laws against, for example, using psychology on other people. The actions are legal - people routinely do them, every day - but not if you call them "psychology", and maybe not if you call yourself a "psychologist". This is one of the innumerable cases where the law hinges on the use of magic words.
It seems clear that what happened was that people saw that "science" produced great, deterministic results, and that when somebody pioneered the concept of calling himself a "scientist" of basic human interactions, a bunch of laws were passed to prohibit "science" from achieving the same thing in conversations that it had in ballistics... or rather, to assume that this would happen, and prohibit anyone from applying the knowledge that would inevitably be produced.
Leaving aside the fact that there is no way to enforce such a prohibition, the mental model of what "science" means is badly broken.
It seems disingenuous to ignore the impact of people like Gino and Ariely on the wider world beyond the academic psychology literature. Both are/were professors in business schools in the US. Their impact on psychology literature proper is probably negligible and I wouldn't be at all surprised if experts in the field were onto them long ago. But they, and others like them, have introduced nonsense into the world that the unaware take as gospel. Overturning it then undermines general credibility in science.
Handwaving it away as 'not all psychologists' seems a little loose to me. And claiming that '[most psychology research] consists of good ideas and trustworthy research' seems to me bland optimism. Good ideas are much too rare in all fields of research - of the 100k papers per year published in psychology it's hard to believe that more than 1,000 are important enough that they'll be remembered in 20 years time. In the fields I'm familiar with, the pressure to publish is relentless and standards dropped through the floor years ago - not much of the material is outright wrong but a lot is misguided noise with no signal underneath.
The biggest problem is people *wanting* something to “take as gospel”, and wanting a field where more than 1% of published papers are memorable. Research has to be seen as much more risk-taking than venture capital. Being too strict up front means you miss out on most of your successes, even if the success *rate* ends up higher. It’s really not a problem that most published claims end up false, as long as randos off the street aren’t reading journals and believing that a single study proves it.
I would have thought there was near-unanimous consensus that researchers need to do more to make their research accessible to the man on the street, to have an impact on society, to progress some relevant problem. Open access and so on are surely intended to incentivise non-professionals to engage.
I have a PhD in maths, and worked fifteen years in combinatorics and algebra -- my own research was to satisfy my own curiosity, and while it connected to the real world generally, it was not directly applicable. It's a different discipline, of course, but my whole area was drowning in incremental papers and pointless generalisations (what if we vary this assumption and brute force the 3 smallest cases) and re-discoveries. It's exactly what you'd expect when people are incentivised to produce papers no-one reads.
Researchers have to be allowed to take risks. The tenure track discourages this greatly - to get an NSF grant, you have to be doing what the people on the panel think is important. (In my field when I was there, what was popular in MIT was the only thing funded.) Researchers also need to be allowed to report failings - they don't currently. And I'd argue that tenured professors who don't research should be assigned other duties...
I’m not convinced that the outreach component is a valuable thing. Math research contributes to the world not by random people on the street reading it, but rather by some amount of it turning out to add up to a larger pattern that later helps someone understand some generalization that then helps a physicist or sociologist or evolutionary biologist or someone else understand something in their field that then enables more discoveries that then eventually have some more direct impact. Incremental papers and apparently pointless generalizations contribute to that, as long as there is someone beyond the author who thinks it’s worth reading (this is what peer review is for).
Open access is sometimes said to be about letting non-professionals engage, but it should be more about letting outsiders generally (who are likely to be specialists in something else) read a bunch of papers that they eventually figure out add up to something that no individual paper was.
I'm broadly in agreement, but I think the number of papers in which single-digit numbers of people are interested is relatively high, and they don't contribute to much of anything. One of the research activities I enjoy is surveying areas of my field, going from initial developments in C19th to the current day. To a surprising extent (or maybe I'm just good at scoping survey topics), there are 2-3 ideas underlying 50-100 papers. I think that clarifying that is a worthwhile exercise, but I end up a little jaded refereeing new papers which rediscover things I've read about from 70 years before (this happens frequently when electrical engineers write about sequences, codes and designs). On one hand, it's good that they're engaging with the material; it's building expertise and maintaining interest in the area; but it's not contributing anything fundamentally new to the field. (Often I'll point out a result is basically known and the paper will be published anyway - standards have dropped in the past 10 years from 'novel' to 'not wrong'. Maybe that's a little harsh, but directionally true.)
I'd be surprised if other disciplines weren't broadly similar.
Yeah I think it’s definitely true that there’s growing pressure on referees to not gatekeep on the basis of novelty or interest if things seem correct, and I think that’s bad.
I think most research in most fields is not going to be accessible to most people, even if it has huge practical impact in the future. To write a paper for other experts in my field, I can't give the kind of background knowledge that would be necessary for someone from far outside my field to follow it.
Explaining your research to outsiders is worthwhile, but whether it's workable or not (and to what depth) depends a lot on what your field is. "I study how people and animals fight off viruses" is easy; "I study how the early signaling between antigen-presenting cells and naive T cells affects the production of different cytokines that determine whether the resulting immune response is mostly cellular or humoral, and this is all done in mice but we think it tells us some stuff about how it works in humans" is less accessible.
Or "I work on ways to keep the internet secure even when quantum computers become available" is kind-of accessible; "I work on bounding the decapsulation failure probability of bit-flipping codes to get a better security proof on some code-based KEMs" is less so.
And so on.
A huge amount of the skepticism of science in US culture comes down to either:
a. Some folks wanting science to be some white-robed high priests providing the known truth that all must believe.
b. Some folks cosplaying as those white-robed high priests to win local political/social battles.
I think it's a bit more complicated than that. A difference I've noticed between the US and Europe is that more people in the US strongly prefer black-and-white answers. Something is either 'good' or 'bad'. The don't like nuance and they don't like uncertainty. I don't mean this as only a political observation, but I do think that Trump communicates to at least some of these people in a remarkably effective way.
An honest scientist who says 'the data doesn't support any definite conclusion', which is logically equivalent to 'we fail to reject the null hypothesis' doesn't appeal to those people.
> 'the data doesn't support any definite conclusion', which is logically equivalent to 'we fail to reject the null hypothesis'
I don't think this is right. You can run many experiments, over and over again.
If you fail to reject the null hypothesis each time, "we fail to reject the null hypothesis" gets stronger over time, while "the data doesn't support any definite conclusion" gets weaker. This must mean that they aren't equivalent.
I'd rephrase Padraig's point this way (and hopefully I got it right):
People who are careful to speak only on what they know or have strong evidence for don't do as well in US media / political culture as people who make clear statements with apparent certainty, whether they really can back them up or not. (I expect this is true other places but I don't really know other cultures well enough to be sure.)
Scientists would ideally be careful to be very clear about what they knew/didn't know, what was likely based on the evidence and what was unlikely, etc. But that kind of statement doesn't make a good quote in a news story and doesn't get clicks and doesn't sound good in a political speech. What sounds good in those places is "We know X, and it's only the denialists and wreckers who deny X." And this sounds just as good when X = "life as we know it arose via evolution" (which is extremely solid) as when X = "there is no biological meaning of sex" (which is silly and crazy but ideologically useful to say). Just sounding confident while spreading your nonsense is pretty effective, as our president demonstrates every single day.
This selects for scientists, journalists, historians, detectives, economists, et al in the public eye who make very broad and confident-sounding claims when they should make much more narrow and contingent claims. The ones who are cautious and careful to be right don't get quoted or interviewed or upranked by the algorithm.
I'll rephrase my point:
The null hypothesis is a theory of how something works. It *isn't* the absence of a theory. When you fail to reject the null, that is evidence supporting a definite conclusion, to wit, the null hypothesis. If you do it a lot, then that's a lot of evidence supporting a definite conclusion.
Thank you! That's a large part of what I meant. I had a slightly broader idea in mind: for the sake of example consider a study which suggests drug X treats disease Y in mice. If a businessperson is asked to interpret this, they will say that drug X is the next big thing to treat disease Y in humans, with the goal of pumping their stock price. A scientist will say that mouse models differ from humans, clinical trials will be necessary, but *the data doesn't support any definite conclusion*.
Supporting evidence is not a definitive conclusion. The scientist who explains carefully and comprehensively what the underlying hypotheses are, how these are violated in practice and hence how the outcome is hard to predict is not who the public wants to hear. Patients with disease Y want to be told the cure is imminent, not there's a 20% chance their lifespan will be extended by 3 months or more relative to a control group.
> of the 100k papers per year published in psychology it's hard to believe that more than 1,000 are important enough that they'll be remembered in 20 years time.
This is hard to interpret. Certainly more than 1,000 will be remembered in 20 years time, by the people who wrote them. But that is either being remembered for a reason other than importance, or being remembered for a kind of importance that I assume you don't want to count.
If we move to some kind of "objective importance", less than 1,000 papers are important enough to be remembered the day after they're published.
I didn't write that well - your second interpretation is what I intended. And the 20 years is arbitrary, but approximately the longest time scale in which you'd hope to see citations to a paper as a measure of its impact. Citations are a bad way of measuring impact but still the closest thing to 'objective importance' that is widely measured.
This is true in every field. Most papers will be forgotten, most that are important will be built on and appear at best in the references of later papers.
I think the "replication crisis" is two things masquerading as one:
One, science - not priming, not social psych, not psych, all of science - has a problem with various things like publication bias and overuse of bad statistics, which was traditionally solved by replication studies. Those don't get enough funding (that's changing but they still don't), so there's a glut of new findings for every paper checking if they hold up. Weirdly, psychology is one of the fields that from my amateur perspective seems to most enthusiastically publish replication studies, and therefore the failure of many results to replicate being found there first is no surprise.
I would compare the way that all the criticisms of economics as a legitimate field are basically things economists were saying first, and have been correcting ever since they noticed them.
Second, yeah, social priming is bullshit.
I feel like now is a good time to share this meme I made: https://x.com/NebuPookins/status/2069519068571582713
The replication crisis has affected almost all fields of experimental science; social psychology is the only one that's had to squarely deal with it in the public eye. Classic experiments in biology and cancer research have failed to replicate; RCT success rates cratered after clinicaltrials.gov was mandated, phenomena that inspired entire sub-fields in areas like motor learning turned out to be just experimental artifacts, fMRI studies were plagued with false positives and P-hacking until upgrading their methods, you have written before about issues in genetics, etc. The replication "crisis" is a pan-scientific challenge spanning a related set of realizations around statistics, data processing, experimental design, power analysis, publication incentives, and open science (or a lack thereof).
I broadly agree with this post's message (psychology research is unfairly maligned) but the correct conclusion is that there are *many* fields that have been, or still are, affected by the replication crisis. The bigger and more rigorous ones have generally dealt with it more effectively, while smaller and more backwater fields have not even realized they have a problem yet. I think psychology on the whole does much better than average now: top psychology journals often publish studies that are either pre-registered, or involve a second replication phase within the same paper; this is still practically unheard of in many other fields.
It's worth keeping in mind that basically nobody over the age of ~45 came of-age in science with any notion that it might be a bad thing to keep collecting data until your p-values are below 0.05, or try a dozen different ways of binning your data, then publish the one that gives you a significant result. If you weren't reading Gelman's blog or Retraction Watch in the late '00s / early 2010s you didn't even realize there was a problem -- or if you did, surely it was "those other sloppy scientists," not *your* lab.
How much of "failed to replicate" is that the effect is smaller than claimed and how much is that there's nothing there at all?
If the effect is smaller, but statistically significant, the follow-up study would be considered a replication. It's authors, though, would likely comment that if the actual effect size is as small as the one they found, the phenomenon is a lot less interesting and important than was thought, based on the original study.
When surgeons list something like “feminist informed neurosurgery” on business cards let me know.
The problem is with ideological and political capture. I am amazed it’s tolerated
A lot of the issues I have with social psych aren't in the portion that doesn't replicate.
Suppose someone writes a study:
"We surveyed prison populations. We discovered that they were much higher % men than the broader population! Clearly there is massive bias against men in criminal justice!"
When someone comes to replicate this study, it will replicate successfully. The result being measured is real.
The interpretation of the result being measured is much more questionable, and is not being replicated.
>"We surveyed prison populations. We discovered that they were much higher % men than the broader population! Clearly there is massive bias against men in criminal justice!"
This is a straw man. Studies are not this dumb. Can you find real examples of lots of dumb people replicating something in psychology that's obvious?
I think that was just meant as a toy example to show how things can be wrong. There's stuff like this all the time, though. I recently fell for statistics showing homeschoolers scoring significantly higher than other types of students. However, this doesn't account for the fact that bot all homeschoolers take these sorts of standardized tests, and if you are smart, you're more likely to have a good reason to take one of those tests, thus the statistics don't show the full story.
Not a psychologist, but dated a clinical psychologist. From what I gathered, there's a fundamental disconnect between how expensive testing new methodologies is and what resources can be marshalled for the purpose. As a consequence, studies will be woefully underpowered, which encourages p-hacking, traveling the garden of forking paths and other misbehavior. In my ex's case, she had what was by any reasonable standard an exceptional study in the field for her career stage. She preregistered the study, which at the time was new and trendy. Unsurprisingly, it failed to meet a clear bar of significance. I have no doubt without that it would have been able to launch her onto a much more prestigious path.
My takeaway here is that basically everything that isn't in the dead center of mainstream treatment hasn't been studied seriously enough to be confident in what the literature says. Even standard recommendations in major fields, such as the Gottmans for relationships, are built on a foundation so shaky that you can't lend it real credence. I think it's clear the human insights that practitioners have developed have value, but it's very difficult to say they've met a scientific standard.
This may be taking the point further than intended, but I feel there is a broader point here, along the lines of "If there is a large academic discipline with a massive number of researchers across a vast number of institutions and a deep history, it probably isn't stupid/crazy for some trivial reason."
The same reaction I have when people will try to dismiss all economics for "not realizing people aren't always rational all the time" or "they don't think inequality is real" or something. And economics is actually a discipline with some more or less universal assumptions. Anyone who says "All history departments are X" is even further off.
I hear people talk about the opinions you can never express on a college campus, and I'm reminded of the many courses (undergrad and grad) I took on the Middle East, Arabic, Islam, etc.-- which should be as full of taboos as it gets-- looking very much like a settler, and not only did I never feel constrained in the views I was expressing, I recall being actively encouraged by one professor (a Syrian refugee) to be freer with my criticisms of the piece we were reading (about an Arab traveler's impressions of the United States, which I noted were both racist and elitist).
This is not to say there aren't problematic trends in all sorts of disciplines and I'm not saying all disciplines are equal in how problematic these trends are, but the kind of absolutist remarks about giving up on psychology (or sociology or whatever) are pretty much always wrong.
I had to take Russian History to get any discussion of taboo subjects (in this case, the number of Jews in Soviet Communism*, around the turn of the revolution).
*This is considerably less "third rail" than discussing the movement in Weimar Germany in such terms, obviously.
You haven't expressed taboo subjects in school until you've been outright failed for expressing them (I know someone who got failed for expressing a viewpoint on Romania, the teacher's country). It's pretty much the definitional "what is taboo." (and any responses that include current frameworks like racism and elitism are almost by definition "not taboo" -- they're well within post world war II western civilization ---> if you were expressing blatantly racist opinions yourself, that might be taboo).
At an early stage in my psychology studies I heard a professor say “the less interesting a subject is in psychology, the more sure we can be that we are right about it.” I have found this to be largely true (and is perhaps born out by this article)
The more money you can extract out of a psychological conclusion, in general, the better we are at getting them "right."
... why the quotes?
There's a lot of money in "Science Article Says" as an advertising trick. Many midwits, much stupidity.
Agreed..... I made this same point here
https://caldwellharris.substack.com/p/mastroianni-concludes-psychology
Then what's your mechanism for how many published research findings are false, but most accepted truths in psychology are true?
Are they not based on papers?
Are they based on papers from long ago, before bad research practices took hold?
Are they based only on papers which replicated? Or on important papers which are known to be good, whereas most (unimportant) research papers are bad?
Two points:
1) I did fMRI research around 2010 (as part of psychology studies) and the state of the field was truly awful, maybe worse than social psychology despite the 'sciency' veneer. P-hacking was the norm and practitioners had limited understanding of how the physical signals were converted into data.
The dead salmon as research subject is the perfect example here: https://law.stanford.edu/2009/09/18/what-a-dead-salmon-reminds-us-about-fmri-analysis/
2) I'm no longer active in the field of psychology, but from the outside the general quality seems to have genuinely improved since 2014 or so. Not universally, but there is a lot more awareness of the replication crisis and what kind of things need more rigour.
I remember a lovely article called "The Control Group Is Out Of Control". I seem to remember it casting a pessimistic shadow over existing science, on the basis of methods and publication standards being so flawed as to permit complete nonsense.
Now you're telling us everything is mostly OK after all?
Psychology outside social priming has not necessarily fared much better. For example, a recent study found that roughly 80% of psychology articles discussing a nonsignificant result interpreted it as evidence of no effect. Clinical psychology has its own well-documented problems with weak control conditions, small samples, researcher allegiance, selective reporting, short follow-up periods, and limited generalizability. And industrial/organizational psychology is hardly an independent comparison field: large parts of it are essentially applied social psychology.
https://pmc.ncbi.nlm.nih.gov/articles/PMC11919487/
I really don't understand this take.
You mention that psychology is widely agreed to be a sham, give tons of examples of why it is a sham, without even grappling with the main criticisms - much of the field is based on mumbo jumbo and the effects of therapy are consistent with the null hypothesis - and conclude that "Psychology Research is mostly Fine"?
There was a pretty useful research paper showing that outdoor exercise outperformed all forms of talk therapy. Just because you have "pretty well understood theory-facts" that are supported by the research, doesn't actually mean "the Research Is Bad."
What you're criticizing is the direction and money spent on researching talk therapy (and, trust me, that's not anywhere near the worst aspect of modern research -- there's plenty of third rails).
Well, yes, it's always nice to have a paper to point to that confirms my biases :-) but the point is that the strongest prior, the null hypothesis, hasn't been convincingly dethroned. Eg if I had to bet I'd guess that outdoor exercise, done carefully, ALSO has "no effect". And GIVEN the sad state of science in general and psychology in particular, I'm not sure what would even change this opinion.
The rest of the comment section is an indictment of science in general - but even within science, I think that the branch that brought us Rorschach tests and the superego has to be uniquely bad.
Evo psych is the bad psychology. It's essentially "never, ever provable" (because generally we're looking at "only humans" in which case you've got a Single Species and No Control).
Why do people get depressed? (Evo psych will tell you that it's adaptive, citing energy conservation).
https://en.wikipedia.org/wiki/Evolutionary_approaches_to_depression
Outdoor exercise, at bare minimum, fixes white people's melatonin issues (and gets them vitamin D), assuming they aren't exercising after dark -- proper amounts of melatonin aid sleep.
Saved this one. The line about Psychology Research Is Mostly Fine ... is the one.
Otto Rank, once Freud's golden child, who was cancelled by the Vienna clique, was the first to criticize early psychology's scientific dream to exclude the subject's data. Of course, patient centered psychology went on to a useful development in the profession, but it is interesting that Rank still remains outside the circle. His critique extended to accusation that Adler, Freud and Jung were all mostly interested in establishing a new ideology; will to power, biology or collective consciousness. Ernest Becker's introduction to his 1973 book Denial of Death simply recommends to the reader that they read Rank. Terror management theorists Greenberg, Solomon, and Pyszczynski did make some substantive contributions but this principle of life remains mostly removed from contemporary psychology research.
In my corner of the world, there's educational psychologists who do things like controlled-randomized trials and their results pass both replication and the sniff test, and the other kind of education research called Scholarship of Teaching and Learning. Comparing the two gives me an immense respect for psychologists. Watching the second kind being totally wrecked by AI gives me an immense satisfaction.
The most culturally visible failure mode tends to become the mirror for the entire field.
Social priming produced clean headlines, political implications, and spectacular claims so its collapse became more legible to culture than thousands of ordinary, incremental findings elsewhere in psychology.
The popular narrative about the replication crisis misses a deeper insight about what this episode brought to the development of scientific rigor. Calling it a "crisis" suggests fields hit a wall of malfeasance unique to the 2000's or to specific fields like social psychology. Malfeasance has always existed in the literature. The replication crisis was named for what it is: published research that didn't replicate — not "people made up results." Non-replication doesn't require fraud, just a low prior probability of holding up, which can happen through p-hacking, forking paths, publication bias, underpowered studies, or motivated analysis choices. None of that requires anyone to knowingly publish something false.
It wasn't "we caught some bad actors and are working to close up the loopholes they found and uniquely exploited." It was a broad rise in intellectual hygiene — preregistration, larger samples, replication requirements, more scrutiny of researcher degrees of freedom. Bad science still gets published, and fraud hasn't disappeared. But in response to the recognition of the problem brought about by the replication crisis, we have new tools to use to help us get at the truth.
We shouldn't treat some earlier decade as a rigor baseline later work fell from. Ioannidis's point was never "things got worse". The problem with replication was likely always there, we just hadn't measured it. Science is better now not because one field, or one group of researchers was uniquely fraudulent in a unique time. The whole field learned about a systemic problem and learned to better prevent, detect, and correct it. Yes, some fields had more room for improvement than others, but forest plots, pre-registration, better statistical hygiene, etc. have all made every field better.
Well, I mock "the psychometricians who measure the genetic contribution to IQ." But they're still churning out their studies irrespective of what the geneticists are telling them (and still recycling a bunch of sketchy twin studies). We've all been through this argument dozens of times now on ACX, but I'll happily throw links at peeps who want to step up to defend the field in another round (especially since I've got more studies up my sleeve now). :-)
Don't forget the psychophysicists who study hearing and audition! It's not all visuals... Technically the field of sound perception is called "psychoacoustics" but when I testify in court I just call it "sound perception" because opposing lawyers think "psychoacoustics" sounds made up or paranormal.
"the Freudian psychologists who help you understand how all your dreams involve unconscious impulses about sex with your mother.
the Jungian psychologists who help you understand how all your dreams involve the the Divine Feminine engaging in mythopoesis by telling you to have sex with your mother.
the developmental psychologists who tell you that you and your friends are Kegan Level Five, but the people who disagree with you are underdeveloped psychologically-rigid Kegan Level Four stick-in-the-muds."
No need for reminding me, I was already sold on mocking Psychology before. I make an exception for the Buddhist monk FMRIs and similar reductionist research but anything in Psych that is "high level" or tied to "culture war values" enough to be influenced by the the views of the political side currently dominant in academia is very likely to be garbage.
The replication crisis was actually one of the best things that could have happened to psychology. Social priming didn't just have bad stats, it had a bad premise: treating context-sensitive social behaviour like a fixed physical constant, the way you'd treat retinal response time or receptor binding. That's why psychophysics and neuropsych didn't need this reckoning and social psych did. Once you drop the fantasy that social research can just mirror human nature back to us, more precisely as the studies pile up, you're pushed toward something like Gergen's future-forming research: work that owns the fact that studying people changes what they become, instead of pretending to hold up a neutral mirror.
In "everything else you love to hate - is in chapter 12 of 16. The other fifteen chapters are comparatively innocuous." 'fifteen' is probably a relic of an earlier draft and should read 'eleven'.
Would love to hear more on the Neo-Freudians - in common with many commentators I thought Jung had done work on personality types that had stood the test of time well.
My comment may make more sense in context. All of the universities in California are involved in different genomic research.
Berkeley genomic research "Our mission: Unlock the promise of safe, accessible, and powerful germline genomic engineering for humanity. Berkeley Genomics Project, an independent nonprofit"
Germline editing, if successful, hypothetically would permit someone to reproduce through incest without fear of genetic consequences, correct?
This feels a lot like Philosophy.
It *is* true that philosophy includes many subfields, some of which even have interesting things to say.
But it’s *also* true that most of the world means ethics when they use the word philosophy. This is simply a reality about word use, and you can bemoan it all you like, but it won’t change the fact.
In this same way, it feels like a debate-team move to make Scott’s complaint.
Yes, it’s technically true (and, for example, we’re all happy to use codecs that have been informed by the kind of technical measurements he describes make up “real” psychology).
But this misses the point, that what people *mean* by psychology, especially in this context, is social psychology.
Wriggling out of the issue on technical grounds, rather than conceding the essential point (albeit perhaps poorly expressed) is exactly why the university elites are so hated.
You don’t get to complain that the public no longer unconditionally throw money at you if your response to every charge of misbehavior is not “you’re right, and we have been doing this badly since 1965; here’s how we’re fixing it” but is instead “you’re so stupid you don’t even know the correct way in which to phrase your charge”…
Can someone explain the section “When you mock “psychology” or “psychologists”, you’re mocking” to me? It seems to be a list of obviously good things but then also a list of maybe fake things. What is the take away supposed to be?
"Social psychology fell into this trap especially hard because it naturally touches on hot topics of popular interest and political explosiveness."
I don't think this is necessarily the main reason for the "failure" of social psychology relative to other subfields. The hotness or popularity of a research question shouldn't automatically make it prone to false findings. I think the main culprit is the scientific credentials and epistemic detachment of those investigating the question. And social psychology just happens to attract more of these pseudo- or half-baked scientist types than other subfields. It explains why there was an explosion of interest in "hot topics" to begin with. The latter is the result of the former (low quality research minds).
💡 Du dogme à la science : pourquoi la psychologie moderne sauve des vies
L'article d'Astral Codex Ten (« Psychology Research Is Mostly Fine ») rappelle une vérité essentielle : si la psychologie sociale a souffert de la crise de la réplication, l'immense majorité de la science psychologique et de la neuropsychologie repose sur des bases solides, loin des théories spéculatives du passé.
Mon histoire personnelle en est une illustration concrète :
20 ans de déni dogmatique : Suite à un trauma causé par une intoxication involontaire (soumission chimique), j'ai été suivi par un psychiatre psychanalyste de l'École de la Cause Freudienne. Pendant deux décennies, la cause réelle de mon trauma a été niée, ramenée à du fantasme, et mes symptômes ont simplement été étouffés sous médication.
Le virage par la science : Il y a 5 ans, tout a changé grâce à une prise en charge en neuropsychiatrie. En traitant le trauma pour ce qu'il est réellement — une réaction neurobiologique et un dysfonctionnement du traitement de l'information par le cerveau — ce suivi m'a enfin libéré.
En conclusion : La psychanalyse orthodoxe a trop souvent médicalisé le déni au détriment des faits. Il est crucial de privilégier les approches modernes et fondées sur la preuve (neurosciences, TCC, EMDR), qui ne cherchent pas à masquer la douleur, mais à réparer ce que le trauma a brisé.
If Psychology isn't so bad, why are psychologists so woke then, huh?
I'm a bit late to the party, but as an ACX grantee funded to look at reproducibility, I feel obligated to comment. Replications are very popular in psychology now, so we have a lot of data on reproducibility across different fields of psychology, and there is considerable variance. See https://metascienceobservatory.org/replications-database/by-discipline
Of course, none of this negates the main points of this post, which I think are valid! Too many people over-index on the replication crisis stuff.
So Psych is not doing so bad compared to most of the fields you measured.
I’m surprised Ed Psych replicates so well. That has usually come under fire.
Also, what is going on with medicine!? 😮
Scott notes that:
> The replication crisis primarily affected the sub-field of social psychology, and most centrally the sub-sub-field of social priming research - the study of how seemingly trivial interventions can have outsized effects on behavior.
I think that this touches on one of the things that makes this field ore vulnerable to bad research and also to being "controversial": There is a dichotomy or scale: The "simplified liberal" position is that people do what society tells them to do. The consequence of this is that human behavior is infinitely malleable. The "simplified conservative" position is that people do what they are incentivized to do, that the social context can cause them to do thing but only because that behavior is somehow coupled to something they *already* want. This suggests that there are distinct limits to how much human behavior can be modified.
This then couples social psychology research to the more ideological of our political divisions, which causes the general public to care about the results more and tempts the researchers to obtain the results that they emotionally prefer.
Scott - sorry but you are dead wrong on this one.
Psychology is divided into the "nothing to see here" crowd and the rest of us.
We are still going through the replication crisis and have yet to go through the citation crisis which means that, on average, more than half of what a psychologist might be saying is not based on replicable results.