My comment: this is a good essay, but I'm confused that the only take on Alzheimers I ever hear is "everyone important subscribes to the amyloid hypothesis, but this is an example of bad science and cracks are starting to show in the facade". If everyone important subscribes to it, why won't anyone defend it? Are they too embarrassed, and just waiting to collect a few more funding checks before quietly retiring? Or is this the thing where everyone wants to read sweeping theories about how The Arrogant Experts Are Wrong About Everything and nobody wants to hear those experts patiently explain the boring facts?
If any ACX readers are arrogant experts willing to stand up for amyloid, please send me an email and pitch me on a post.
My guess is that the issue here is the one Kuhn identifies, that a paradigm never goes away until you have an alternative. Say what you will about the amyloid plaque hypothesis, at least it’s a hypothesis. It has lots of problems, but it also has some scraps of evidence for it. If no one else has *any* hypothesis, then this one will naturally dominate the literature.
Sort of like how late 19th century astronomy had all sorts of hypotheses about why the planet Vulcan was so hard to observe in the way it interferes with the perihelion shift of Mercury, but no one saying “gravity is wrong”.
There is at least one serious alternative hypothesis, tau proteins. I doubt that this would fare much better for the level of scrutiny applied to the amyloid hypothesis here, but I am not sure that it is so much inferior either. (But I am happy to be corrected by experts.)
Ok -- I'll take the bait on this. Not to critique the core point of this review (that we should take results from genetic animal systems with great caution). But to partially defend the relevance of amyloid as a drug target in AD.*
1. There's a word not found in this review: Leqembi. That's the *second* FDA-approved anti-amyloid antibody. It’s not a cure. Indeed, it’s modestly effective and has significant side effects. But it’s proof of principle. We give patients cancer drugs that aren’t curative, but extend life. That's similar here. Patients who took the drug got maybe 3-6 months of ‘cognition extension.’ Not great, but it’s a start. If I could paste the graph from the approval documents, I would -- but you can find it here:
2. The Aβ*56 fraud (which is bad!) has basically nothing to do with why the amyloid hypothesis was adopted. You can find a fairly expert discussion in the comments here (https://www.alzforum.org/news/community-news/sylvain-lesne-who-found-av56-accused-image-manipulation). The fraud also isn't what made big pharmas invest (and they are still investing) in amyloid-directed therapies. Lesne first published in 2006. At that point Lilly was already *in the clinic* with a gamma secretase inhibitor. People didn't back these programs because of Lesne and Aβ*56 but because...
3. The human genetic support for the role of amyloid in AD is profound! I won't reiterate it all here, but just google up a review paper and you can see why everyone was so excited. (In brief, Downs’ syndrome patients and families with early-onset Alzheimer’s all had mutations in the amyloid pathway. The book "How Not to Study a Disease" by severe amyloid skeptic Karl Herrup has a fair and illuminating discussion of the scientific history.
So in short -- there was groupthink around the centrality of amyloid in AD** BUT, it's also the case that after many failed drugs, there's some emerging evidence that anti-amyloid therapies may help in humans. And there is hope that improved products may further enhance efficacy. (e.g., Roche's brain-shuttle enabled anti-amyloid antibody https://www.roche.com/media/releases/med-cor-2025-04-03).
*A bit of a cheat. The strong version of the Amyloid hypothesis is that Amyloid is the cause of everything. Here I defend a weaker thesis -- that there is reason to think amyloid is relevant to AD pathology and that anti-amyloid therapy has the potential to provide meaningful benefit.
**It is not uncommon to find an overemphasis on using familial/genetic forms of a disease as a model for sporadic. I have heard ALS drug developers make the same point about SOD1, and the pointlessness of the SOD1 mouse)
it’s pieces like this, and comments like this, that are why i can’t quit ACX. I’ve wanted this for ages and everything is either written down for dummies or would involve too much investment in fluency acquisition for mere curiosity (vs work, where I do it every so often for a new business pitch—I’m a biopharma brand strategist). thank you, seriously
This gets to my problems with this review. We're hearing one side of an argument in a scientific field with which most of us are unfamiliar. We don't know what the state of the discussion is, so it's difficult to read this essay in context.
Are there serious flaws in an important 30-year-old paper? How do these compare to the flaws in other, similar papers on different subjects? How much of this is reading back later discoveries and developments?
My general rule is that I don't read scientific papers in unfamiliar fields. Journals are meant to be conversations between active participants in cutting-edge research. Much of Ph.D. training is learning the background to participate in these conversations. Popular science writing, if done honestly and well, can summarize the current state of these discussions. But these writers need to be careful to include all sides, and not just rely on a single paper.
Multiple drugs that target amyloid plaque buildup are actively being prescribed to treat Alzheimer's disease. The science is compelling and clear that amyloid plaque removal does something, at least.
Is it unfortunate that these drugs don't do more? Yes. But to say the amyloid hypothesis is "wrong" is to ignore reality. We've already commercialized the hypothesis.
Bottom line: If the average reader of this comment was diagnosed with Alzheimer's disease tomorrow, your physician would probably prescribe you a drug that removes amyloid plaque.
Peruse this paper if you don't believe me. Donanemab, an anti amyloid-beta mAb, hardly slows down cognitive decline in patients. Sure it's better then placebo, but barely by anything, and certainly less than you'd expect if it was indeed the primary driver of the disease.
consider this study looking at discontinuation of Infliximab, an anti TNF-alpha mAb. From the theory that TNF-alpha plays a large role in Crohn's disease inflammation, does an mAb that blocks its activity do anything? Well check out figure 2A and notice that while none of the people receiving continuing Infliximab treatment had a symptomatic relapse, HALF OF THE PEOPLE without the treatment did!
"Consider the familiar structure of a scientific paper: Introduction (background and hypothesis), Methods, Results, Discussion, Conclusion. This format implies that the work followed a clean, sequential progression: scientists identified a gap in knowledge, formulated a causal explanation, designed definitive experiments to fill the gap, evaluated compelling results, and most of the time, confirmed their hypothesis.
Real lab work rarely follows such a clear path. Biological research is filled with what Medawar describes lovingly as “messing about”: false starts, starting in the middle, unexpected results, reformulated hypotheses, and intriguing accidental findings. The published paper ignores the mess in favour of the illusion of structure and discipline. It offers an ideal version of what might have happened rather than a confession of what did."
Sort of OT, but judicial reasoning is similar. On paper, the trial judge is supposed to determine the facts and then apply the law to those facts.
As a practical matter, in the real world, first the judge makes up their mind, then seeks factual and legal determinations that support the preordained conclusion.
The difference is that the judge has a lot more power than the scientist, at least unless the scientist totally makes shit up.
Sometimes the judge makes up their mind, but then the opinion "won't write" (facts and/or law don't support the snap judgement) and the judge changes their mind. Happens more than you would think.
"...messing about false starts..." Great line. But when you're working in a field where you don't control the environment or experiment at all? My graduate work was as an analytical geochemist. When I was working on core samples from the Deep Sea Drilling Project - you didn't have any kind of control except in what you chose to, or what you had the capability to, analyze for. Talk about holding your breath until the end where you had to figure out what justifiable conclusions you could draw. So you can end up with and present two types of conclusions - the ones the data fully supports, and the ones that you wish you had enough data to fully support. At that point, all you can be is honest. If you're very unlucky, or didn't fully consider the kind of data you might obtain, you could end up having to backtrack mightily or start entirely over. A dissertation is expected to produce a positive conclusion of some sort. In a paper published by an established researcher, you can get away with less certainty, which does not mean you should be less forthcoming about how well those conclusions are supported.
In the fields I've worked in, the general approach is to test for about 100 different conclusions, about 3 of which you'd like to prove, and over 50 of which you're pretty certain are right. Then you have 3 "fantastic" papers, and 50 "solid" papers. Of course, you're probably not going to get the statistical power to disprove those 50...
The drawback to all this is that you can get 10 different researchers testing for those 3 conclusions, never realizing that other people are doing the /exact same research/ and getting null-results (did not pass statistical significance).
This is my field (amyloid, GluT4, insulin signalling, metabolism and cognition) and I very much enjoyed the piece; I'm happy to say that I have been vocally pointing out flaws in the strong amyloid (and especially, in plaques-as-causative) hypotheses for quite a few years now :).
To Scott's Q below: at least part of the issue is that many, many billions of pharma dollars were spent on developing treatments (primarily antibodies) aimed at reducing amyloid load. So there is and was a huge sunk cost. There was also, at least for a while, a gate-keeping role of editors at major Alzheimer's journals and funding bodies.
Medawar has a good discussion of inductive in the old sense, and why it is problematic - there is no logic of scientific discovery (even if we relax criteria so that the evidence doesn’t have to *prove* the conclusion but just make it the *right* conclusion). But he and Platt then go on to endorse similarly too-systematic accounts of how science *should* work, whether it’s Popper’s idea that there is *no* confirmation and just attempted falsification, or the idea that some kind of “strong” test can account for all the alternatives.
A lot of people in the internet rationalist community build the same kind of flawed assumptions into their bayesianism, as though we could figure out what is the *correct* degree of belief to have in every hypothesis.
But there’s not. All Bayesianism gives us is an internal criterion of how to make our own decision making consistent bud we violate this, we run the risk of being necessarily self-defeating. But some people who violate these principles happen to get lucky, and there are many different ways to satisfy the Bayesian rules, some of which will be lucky and some of which will be unlucky in any given domain of investigation.
Many of the problems of the paper discussed in this post are the kind of understandable flaw that any given paper will have - someone with one set of priors will interpret things in light of those. But the bigger problem is in the field that followed it, that didn’t happen to include enough irrational contrariness to force these tests that, in retrospect, seem like clear ones to have worried about.
"Nice to see someone on this site quoting Medawar, who was, of course, a Popperian. I've tried in the past to get Scott--and also his mentor, Eliezer Yudkowsky--to take a proper look at Popper's theory of knowledge, including his critique of induction (including the Bayesian one they subscribe to). Sadly, they insist on dismissing Popper, not--as far as I can tell--on the basis of anything he actually said, but on the basis of a strawman version that appears in the secondary literature. Sigh."
If it’s not too much trouble, do you have pointers to where people like Scott have dismissed Popperian views?
I’m not a philosopher of science but I do have a passing/growing interest in the topic and I’d be curious what people in the rationalist community think of popper. Naively, I would’ve thought they’d like falsificationism. usually the critiques I read of Popperian view come from a perspective rooted in Lakatos or Feyerabend.
Scott dismissed Popper in an email message years ago, and I didn't save it. I don't know whether he's written Popper elsewhere. My discussion with Yudkowsky took place via X and is also lost. (To me, anyway. I'm not a very sophisticated user.) As I recall, however, he referred me to something he'd written at his Less Wrong website. Regarding Lakatos and Feyerbend, I don't agree with them, but at least they deal with what Popper actually wrote. Most of his critics, however, say something like, "No scientific theory can be definitively falsified, therefore Popper is wrong." That's a bad take for two reasons. First, Popper never claimed theories can be falsified in practice, he just suggested that one can distinguish between the empirical sciences and other kinds of enquiry by noting the the former works with statements that have empirical content, i.e., statements that have implications that can be tested experientially, and that can therefore be falsified in principle. The second and more important reason it's a bad take is that falsifiability isn't an important part of Popper's theory of knowledge. As I explain in the paper for which I provided a link, the fundamental elements of Popper's theory are falibilism, critical rationalism, and the distinction between objective and subjective knowledge.
I don't know whether Popper ever wrote about or even knew about the Bayesian approach to epistemology espoused by Scott and the rest of the neorationalists. He died in 1994. However, he wrote extensively about probability and explained why his critique of induction didn't just mean statements can't be shown to be true; it also means they can't be shown to be probable. He says so at various places in The Logic of Scientific Discovery, and there's an extensive discussion in the Postscript. My copies are at work, and I'm at home right now. I'll try to remember to send you more specific citations next week.
P.S. I just asked a friend who's also a Popperian, and he said, "He writes about probability in A World of Propensities, in the appendices to LSD, in the first volume of the Postscirpt Realism & the Aim of Science, to a lesser extent in the other 2 volumes, he has an interesting discussion in the appendices to World of Parmenides, Miller has an essay in the Cambridge Companion to Popper on his contributions to probability theory. There's lots of other places where he writes about it but those are the most lengthy treatments."
I'd have thought probability was always a matter of degree in some sense, but I'm just a provincial lawyer and not very numerate. You'll have to read Popper yourself, or ask someone more knowledgeable.
What would change about the analysis of the paper without an agenda? The author laid out a list of the evidence they'd expect to see in support of the original paper's claims--were those standards unreasonable, or did the author of the essay not fairly evaluate them?
If neither of those things are true, then the author's agenda could very well be "demonstrate proper paper-reading technique using a real example".
Use a REAL example. Pull Seneff's paper on COVID-19 mRNA vaccines-- "where the potential issues are with this massive "nearly untested" intervention." Given that her paper got decent look-see with the OWS staff, I'd say it's a decent "How do you start analyzing whether this intervention works or not."
If you don't think her paper's good enough (or think she's a crank in general), pull a different one. This was a big, splashy intervention. Surely someone else sat down and started saying "here's where this might go wrong, and how to identify if it does."
What is the difference between the paper mentioned in the essay and the one you mention that makes the latter more "REAL" than the former, other than the latter being linked to a more heavily politicized topic?
The Seneff paper is doing what this essay purports to do -- find the "what would we expect to see if this is going to crash and burn and destroy humanity." So it's not the amyloid paper, it's the "proper analysis" -- done as an a priori "here's what we know, and here's where this might kill us all" (sorry to sound all doom and gloom, but this is a massive public health intervention over "most of humanity" -- looking at worst case is part of the fun).
Unlike the above essay, Seneff is putting her money where her mouth is, BEFORE the mRNA vaccine gets widely tested. You want to find out where potential "we have a problem" points are? you should be able to describe them before the paper to be analyzed is even written.
I'd be a lot happier with someone who stood up and said, "before I sit down and look at this paper...", not 30 years later, "here's how to analyze this paper -properly-"
2. The paper is older, the dust has settled, unlikely to rankle so many feathers
3. The subject is more mature, so we know the answer
When learning math, I was always annoyed when the practice problems didn't have a neat answer (often they were only one small change from having a neat answer, which made me suspicious that the problem-writers had accidentally flipped a sign or something). How am I supposed to know I'm doing any of these correctly? I know real life isn't always neat, but this isn't real life, it's a theoretical exercise to practice the fundamentals before moving on to messy practical applications.
I hope you find the essay you're looking for, but I think this one is better for the audience.
I really needed to be more clear that the end of the second paragraph was referring to the math problems. I'm aware Alzheimer's is real life and that the field is still active; my point stands that it's more useful to pick a less-controversial topic and to pick apart older research when learning fundamentals.
Talking about COVID vaccines... I mean, good grief, the person to whom I was responding asked for a paper WITHOUT an agenda.
You can't do science if you already know the answer. Science is all about working with uncertainty.
You aren't getting the "a priori reasoning" in the essay above. You're missing out on the fails. You're getting an ex post facto "right and true" version of "how to attempt to flunk a paper."
Seneff walks you through the potential for an antifreeze allergy in a significant segment of the population. I'm pretty confident we didn't hit that failure mode of the mRNA vaccines.
How do you know that you're doing it correctly? You aren't, unless you're uncertain, and working with uncertainty. This isn't math, math gets things correct. Science just makes a better model -- and lo, if tomorrow the model is no longer accurate, we will build a new one (Yes, there's a model out there for "no dark matter, gravity is just different in the galactic arms" -- and yes, if you wanted to look at "how to disprove dark matter" you could look at that paper. But that's just running "one disprove" not Seneff's "here's ten reasons we shouldn't do this").
It took me a while to understand the introduction, because I assumed that he was talking about some specific paper. I thought it was a complaint about the rigor of some paper that had yet to be explained, and I guess I was supposed to click on the link to see which paper was being complained about and why it's surprising that a Nobel laureate would complain about it.
Perhaps inserting "[the entire concept of]" into the quote would help.
Who were the brave souls who first articulated these concerns and what do we know of what became of their concerns and their careers? In other words, science should only work with constant critical appraisal. It seems clear that this crityappraisal was either absent or smashed by powerful forces. That in itself is of course age old. But knowing it is age old should it be excused or should we learn from it?
This is an exceedingly important question to answer because it can be used to broadly determine if medicine is a science and whether its conclusions are trustworthy. If the careers of the scientists who question the amyloid hypothesis are healthy, then medicine is operating more or less like a science and can be treated as such. However, if the questioners were instead expelled because of their concerns, then medicine is not a science and should not be given the same level of trust.
Good. These are the right questions to be asking. Now, assume malevolence (devil's advocate if you must). The Pharma companies want to make money, so they're going to throw money at people whose ideas have potential therapeutics attached. So, yeah, I'm pretty sure you're going to see that the questioners did not get the shiny, big rewards -- be they patents($$$), status, or otherwise.
True. How much money do we need to waste before you decide that capitalism is the wrong way to approach medicine? (or: state the failure condition you'd accept for "we need a new solution here").
Is there a good step by step guide to reading scientific papers anywhere? This post has a good start at one, but having a checklist of things to think about would be pretty helpful.
As is, I'm confused about what it means to buy "Other". Naively one might think that a vote for "Other" would mean that some review other than the three currently in the market would win. But you're planning to add reviews as they are posted, and eventually all possible reviews should appear in the market. Doesn't this mean that "Other" can never possibly win?
The illustrative application of scientific rigor is what marks the best of this essay. I've been attempting to work on the 'detective' mode while reading papers for a bit, so I'm glad to read this well-written piece. Overall, I'd say this leaves the story a bit unfinished by it's own promise initially. This essay could've used a larger view along with this segment: on other evidence ignored, on more recent hypotheses that show promise. Scott's comment addresses another potential direction too.
Nice to see someone on this site quoting Medawar, who was, of course, a Popperian. I've tried in the past to get Scott--and also his mentor, Eliezer Yudkowsky--to take a proper look at Popper's theory of knowledge, including his critique of induction (including the Bayesian one they subscribe to). Sadly, they insist on dismissing Popper, not--as far as I can tell--on the basis of anything he actually said, but on the basis of a strawman version that appears in the secondary literature. Sigh.
I liked this review for its hands on review of the science, but I’m not really convinced that amyloids aren’t a predicate to Alzheimer’s. I’m not sure anyone holds the strong hypothesis at all these days, and critique of a 20 year old paper doesn’t seem like a persuasive way to argue against the weak one.
Having read the whole essay I'm still unsure what exactly this is a review of? Overly condensed scientific papers? The Alzheimers literature? Making transgenic mice? This feels like a regular essay with the word "review" slapped on it so that it can meet the criteria for a review competition.
>Through a painstaking process called sub-cloning, equal parts molecular biology and divination, they managed to insert into their mouse a human APP gene carrying the mutation found in families with high rates of early-onset Alzheimer's.
>You can design your construct perfectly on paper, but in truth, you solve the problem by tweaking reagents like an alchemist, trying to find the perfect brew to coax your foreign gene into the plasmid at high efficiency.
I guess I'm an alchemist then :) I didn't choose the name Metacelsus for nothing...
More seriously, I'm glad I'm not stuck with the molecular biology tools of 1995. Or worse, 1984, when researchers had to clone a gene by literally scraping pieces of mouse chromosome 17 off of glass slides using tiny needles. https://pubmed.ncbi.nlm.nih.gov/6697397/
My comment: this is a good essay, but I'm confused that the only take on Alzheimers I ever hear is "everyone important subscribes to the amyloid hypothesis, but this is an example of bad science and cracks are starting to show in the facade". If everyone important subscribes to it, why won't anyone defend it? Are they too embarrassed, and just waiting to collect a few more funding checks before quietly retiring? Or is this the thing where everyone wants to read sweeping theories about how The Arrogant Experts Are Wrong About Everything and nobody wants to hear those experts patiently explain the boring facts?
If any ACX readers are arrogant experts willing to stand up for amyloid, please send me an email and pitch me on a post.
My guess is that the issue here is the one Kuhn identifies, that a paradigm never goes away until you have an alternative. Say what you will about the amyloid plaque hypothesis, at least it’s a hypothesis. It has lots of problems, but it also has some scraps of evidence for it. If no one else has *any* hypothesis, then this one will naturally dominate the literature.
Sort of like how late 19th century astronomy had all sorts of hypotheses about why the planet Vulcan was so hard to observe in the way it interferes with the perihelion shift of Mercury, but no one saying “gravity is wrong”.
There is at least one serious alternative hypothesis, tau proteins. I doubt that this would fare much better for the level of scrutiny applied to the amyloid hypothesis here, but I am not sure that it is so much inferior either. (But I am happy to be corrected by experts.)
Ok -- I'll take the bait on this. Not to critique the core point of this review (that we should take results from genetic animal systems with great caution). But to partially defend the relevance of amyloid as a drug target in AD.*
1. There's a word not found in this review: Leqembi. That's the *second* FDA-approved anti-amyloid antibody. It’s not a cure. Indeed, it’s modestly effective and has significant side effects. But it’s proof of principle. We give patients cancer drugs that aren’t curative, but extend life. That's similar here. Patients who took the drug got maybe 3-6 months of ‘cognition extension.’ Not great, but it’s a start. If I could paste the graph from the approval documents, I would -- but you can find it here:
(https://www.leqembi.com/-/media/Files/Leqembi/Prescribing-Information.pdf?hash=106915a5-be7a-4bbc-8c8a-68b0a326d339)
2. The Aβ*56 fraud (which is bad!) has basically nothing to do with why the amyloid hypothesis was adopted. You can find a fairly expert discussion in the comments here (https://www.alzforum.org/news/community-news/sylvain-lesne-who-found-av56-accused-image-manipulation). The fraud also isn't what made big pharmas invest (and they are still investing) in amyloid-directed therapies. Lesne first published in 2006. At that point Lilly was already *in the clinic* with a gamma secretase inhibitor. People didn't back these programs because of Lesne and Aβ*56 but because...
3. The human genetic support for the role of amyloid in AD is profound! I won't reiterate it all here, but just google up a review paper and you can see why everyone was so excited. (In brief, Downs’ syndrome patients and families with early-onset Alzheimer’s all had mutations in the amyloid pathway. The book "How Not to Study a Disease" by severe amyloid skeptic Karl Herrup has a fair and illuminating discussion of the scientific history.
So in short -- there was groupthink around the centrality of amyloid in AD** BUT, it's also the case that after many failed drugs, there's some emerging evidence that anti-amyloid therapies may help in humans. And there is hope that improved products may further enhance efficacy. (e.g., Roche's brain-shuttle enabled anti-amyloid antibody https://www.roche.com/media/releases/med-cor-2025-04-03).
*A bit of a cheat. The strong version of the Amyloid hypothesis is that Amyloid is the cause of everything. Here I defend a weaker thesis -- that there is reason to think amyloid is relevant to AD pathology and that anti-amyloid therapy has the potential to provide meaningful benefit.
**It is not uncommon to find an overemphasis on using familial/genetic forms of a disease as a model for sporadic. I have heard ALS drug developers make the same point about SOD1, and the pointlessness of the SOD1 mouse)
it’s pieces like this, and comments like this, that are why i can’t quit ACX. I’ve wanted this for ages and everything is either written down for dummies or would involve too much investment in fluency acquisition for mere curiosity (vs work, where I do it every so often for a new business pitch—I’m a biopharma brand strategist). thank you, seriously
Thanks!
This gets to my problems with this review. We're hearing one side of an argument in a scientific field with which most of us are unfamiliar. We don't know what the state of the discussion is, so it's difficult to read this essay in context.
Are there serious flaws in an important 30-year-old paper? How do these compare to the flaws in other, similar papers on different subjects? How much of this is reading back later discoveries and developments?
My general rule is that I don't read scientific papers in unfamiliar fields. Journals are meant to be conversations between active participants in cutting-edge research. Much of Ph.D. training is learning the background to participate in these conversations. Popular science writing, if done honestly and well, can summarize the current state of these discussions. But these writers need to be careful to include all sides, and not just rely on a single paper.
Multiple drugs that target amyloid plaque buildup are actively being prescribed to treat Alzheimer's disease. The science is compelling and clear that amyloid plaque removal does something, at least.
Is it unfortunate that these drugs don't do more? Yes. But to say the amyloid hypothesis is "wrong" is to ignore reality. We've already commercialized the hypothesis.
Bottom line: If the average reader of this comment was diagnosed with Alzheimer's disease tomorrow, your physician would probably prescribe you a drug that removes amyloid plaque.
Yeah, and the drugs that are being prescribed *don't actually do much*.
https://jamanetwork.com/journals/jama/fullarticle/2807533
Peruse this paper if you don't believe me. Donanemab, an anti amyloid-beta mAb, hardly slows down cognitive decline in patients. Sure it's better then placebo, but barely by anything, and certainly less than you'd expect if it was indeed the primary driver of the disease.
https://evidence.nejm.org/doi/full/10.1056/EVIDoa2200061
consider this study looking at discontinuation of Infliximab, an anti TNF-alpha mAb. From the theory that TNF-alpha plays a large role in Crohn's disease inflammation, does an mAb that blocks its activity do anything? Well check out figure 2A and notice that while none of the people receiving continuing Infliximab treatment had a symptomatic relapse, HALF OF THE PEOPLE without the treatment did!
https://www.science.org/content/blog-post/faked-beta-amyloid-data-what-does-it-mean
if you want a more thorough breakdown of the failings of beta-amyloid to explain Alzheimer's I strongly recommend giving this article a read.
"Consider the familiar structure of a scientific paper: Introduction (background and hypothesis), Methods, Results, Discussion, Conclusion. This format implies that the work followed a clean, sequential progression: scientists identified a gap in knowledge, formulated a causal explanation, designed definitive experiments to fill the gap, evaluated compelling results, and most of the time, confirmed their hypothesis.
Real lab work rarely follows such a clear path. Biological research is filled with what Medawar describes lovingly as “messing about”: false starts, starting in the middle, unexpected results, reformulated hypotheses, and intriguing accidental findings. The published paper ignores the mess in favour of the illusion of structure and discipline. It offers an ideal version of what might have happened rather than a confession of what did."
Sort of OT, but judicial reasoning is similar. On paper, the trial judge is supposed to determine the facts and then apply the law to those facts.
As a practical matter, in the real world, first the judge makes up their mind, then seeks factual and legal determinations that support the preordained conclusion.
The difference is that the judge has a lot more power than the scientist, at least unless the scientist totally makes shit up.
Sometimes the judge makes up their mind, but then the opinion "won't write" (facts and/or law don't support the snap judgement) and the judge changes their mind. Happens more than you would think.
And sometimes the en banc demands that the defendant time-travel, despite the logistical difficulties involved.
"...messing about false starts..." Great line. But when you're working in a field where you don't control the environment or experiment at all? My graduate work was as an analytical geochemist. When I was working on core samples from the Deep Sea Drilling Project - you didn't have any kind of control except in what you chose to, or what you had the capability to, analyze for. Talk about holding your breath until the end where you had to figure out what justifiable conclusions you could draw. So you can end up with and present two types of conclusions - the ones the data fully supports, and the ones that you wish you had enough data to fully support. At that point, all you can be is honest. If you're very unlucky, or didn't fully consider the kind of data you might obtain, you could end up having to backtrack mightily or start entirely over. A dissertation is expected to produce a positive conclusion of some sort. In a paper published by an established researcher, you can get away with less certainty, which does not mean you should be less forthcoming about how well those conclusions are supported.
In the fields I've worked in, the general approach is to test for about 100 different conclusions, about 3 of which you'd like to prove, and over 50 of which you're pretty certain are right. Then you have 3 "fantastic" papers, and 50 "solid" papers. Of course, you're probably not going to get the statistical power to disprove those 50...
The drawback to all this is that you can get 10 different researchers testing for those 3 conclusions, never realizing that other people are doing the /exact same research/ and getting null-results (did not pass statistical significance).
This is my field (amyloid, GluT4, insulin signalling, metabolism and cognition) and I very much enjoyed the piece; I'm happy to say that I have been vocally pointing out flaws in the strong amyloid (and especially, in plaques-as-causative) hypotheses for quite a few years now :).
To Scott's Q below: at least part of the issue is that many, many billions of pharma dollars were spent on developing treatments (primarily antibodies) aimed at reducing amyloid load. So there is and was a huge sunk cost. There was also, at least for a while, a gate-keeping role of editors at major Alzheimer's journals and funding bodies.
Medawar has a good discussion of inductive in the old sense, and why it is problematic - there is no logic of scientific discovery (even if we relax criteria so that the evidence doesn’t have to *prove* the conclusion but just make it the *right* conclusion). But he and Platt then go on to endorse similarly too-systematic accounts of how science *should* work, whether it’s Popper’s idea that there is *no* confirmation and just attempted falsification, or the idea that some kind of “strong” test can account for all the alternatives.
A lot of people in the internet rationalist community build the same kind of flawed assumptions into their bayesianism, as though we could figure out what is the *correct* degree of belief to have in every hypothesis.
But there’s not. All Bayesianism gives us is an internal criterion of how to make our own decision making consistent bud we violate this, we run the risk of being necessarily self-defeating. But some people who violate these principles happen to get lucky, and there are many different ways to satisfy the Bayesian rules, some of which will be lucky and some of which will be unlucky in any given domain of investigation.
Many of the problems of the paper discussed in this post are the kind of understandable flaw that any given paper will have - someone with one set of priors will interpret things in light of those. But the bigger problem is in the field that followed it, that didn’t happen to include enough irrational contrariness to force these tests that, in retrospect, seem like clear ones to have worried about.
I posted the following elsewhere in the comments:
"Nice to see someone on this site quoting Medawar, who was, of course, a Popperian. I've tried in the past to get Scott--and also his mentor, Eliezer Yudkowsky--to take a proper look at Popper's theory of knowledge, including his critique of induction (including the Bayesian one they subscribe to). Sadly, they insist on dismissing Popper, not--as far as I can tell--on the basis of anything he actually said, but on the basis of a strawman version that appears in the secondary literature. Sigh."
Logic plays a role in scientific discovery, but not as a means of constructing or justifying conclusions. It is, instead, one of the tools we use to help us criticize that test hypotheses. See pp. 15-21 in https://docs.google.com/document/d/1CrtC9yvBPx06DDd7ULQ6--gDvMmpBJZO/edit?pli=1.
If it’s not too much trouble, do you have pointers to where people like Scott have dismissed Popperian views?
I’m not a philosopher of science but I do have a passing/growing interest in the topic and I’d be curious what people in the rationalist community think of popper. Naively, I would’ve thought they’d like falsificationism. usually the critiques I read of Popperian view come from a perspective rooted in Lakatos or Feyerabend.
Scott dismissed Popper in an email message years ago, and I didn't save it. I don't know whether he's written Popper elsewhere. My discussion with Yudkowsky took place via X and is also lost. (To me, anyway. I'm not a very sophisticated user.) As I recall, however, he referred me to something he'd written at his Less Wrong website. Regarding Lakatos and Feyerbend, I don't agree with them, but at least they deal with what Popper actually wrote. Most of his critics, however, say something like, "No scientific theory can be definitively falsified, therefore Popper is wrong." That's a bad take for two reasons. First, Popper never claimed theories can be falsified in practice, he just suggested that one can distinguish between the empirical sciences and other kinds of enquiry by noting the the former works with statements that have empirical content, i.e., statements that have implications that can be tested experientially, and that can therefore be falsified in principle. The second and more important reason it's a bad take is that falsifiability isn't an important part of Popper's theory of knowledge. As I explain in the paper for which I provided a link, the fundamental elements of Popper's theory are falibilism, critical rationalism, and the distinction between objective and subjective knowledge.
What is your Twitter account, in case others want to try searching for your tweet?
@JonGuze
Is this the conversation you were thinking of?
https://x.com/JonGuze/status/1701772371189752106
What did Popper say about Bayesianism?
I don't know whether Popper ever wrote about or even knew about the Bayesian approach to epistemology espoused by Scott and the rest of the neorationalists. He died in 1994. However, he wrote extensively about probability and explained why his critique of induction didn't just mean statements can't be shown to be true; it also means they can't be shown to be probable. He says so at various places in The Logic of Scientific Discovery, and there's an extensive discussion in the Postscript. My copies are at work, and I'm at home right now. I'll try to remember to send you more specific citations next week.
P.S. I just asked a friend who's also a Popperian, and he said, "He writes about probability in A World of Propensities, in the appendices to LSD, in the first volume of the Postscirpt Realism & the Aim of Science, to a lesser extent in the other 2 volumes, he has an interesting discussion in the appendices to World of Parmenides, Miller has an essay in the Cambridge Companion to Popper on his contributions to probability theory. There's lots of other places where he writes about it but those are the most lengthy treatments."
> his critique of induction didn't just mean statements can't be shown to be true; it also means they can't be shown to be probable
That sounds rather binary, whereas Bayesianism is about DEGREES of probability.
I'd have thought probability was always a matter of degree in some sense, but I'm just a provincial lawyer and not very numerate. You'll have to read Popper yourself, or ask someone more knowledgeable.
Scott, you might want to consider converting the footnotes to Substack's native format to make them easier to read.
I was going to, but I couldn't find footnote [1] and eventually gave up.
i enjoyed this post, but i think this was kind of funny:
> Fortunately, at the start of this analysis, we took the time to define the experimental standards needed to evaluate these claims.
No we didn't! We're re-reading a 30 year old paper with an agenda! This is the exact same thing that we're accusing scientific papers of doing.
What would change about the analysis of the paper without an agenda? The author laid out a list of the evidence they'd expect to see in support of the original paper's claims--were those standards unreasonable, or did the author of the essay not fairly evaluate them?
If neither of those things are true, then the author's agenda could very well be "demonstrate proper paper-reading technique using a real example".
Use a REAL example. Pull Seneff's paper on COVID-19 mRNA vaccines-- "where the potential issues are with this massive "nearly untested" intervention." Given that her paper got decent look-see with the OWS staff, I'd say it's a decent "How do you start analyzing whether this intervention works or not."
If you don't think her paper's good enough (or think she's a crank in general), pull a different one. This was a big, splashy intervention. Surely someone else sat down and started saying "here's where this might go wrong, and how to identify if it does."
What is the difference between the paper mentioned in the essay and the one you mention that makes the latter more "REAL" than the former, other than the latter being linked to a more heavily politicized topic?
The Seneff paper is doing what this essay purports to do -- find the "what would we expect to see if this is going to crash and burn and destroy humanity." So it's not the amyloid paper, it's the "proper analysis" -- done as an a priori "here's what we know, and here's where this might kill us all" (sorry to sound all doom and gloom, but this is a massive public health intervention over "most of humanity" -- looking at worst case is part of the fun).
Unlike the above essay, Seneff is putting her money where her mouth is, BEFORE the mRNA vaccine gets widely tested. You want to find out where potential "we have a problem" points are? you should be able to describe them before the paper to be analyzed is even written.
I'd be a lot happier with someone who stood up and said, "before I sit down and look at this paper...", not 30 years later, "here's how to analyze this paper -properly-"
I think the author made a better choice.
1. The topic is less polarized
2. The paper is older, the dust has settled, unlikely to rankle so many feathers
3. The subject is more mature, so we know the answer
When learning math, I was always annoyed when the practice problems didn't have a neat answer (often they were only one small change from having a neat answer, which made me suspicious that the problem-writers had accidentally flipped a sign or something). How am I supposed to know I'm doing any of these correctly? I know real life isn't always neat, but this isn't real life, it's a theoretical exercise to practice the fundamentals before moving on to messy practical applications.
I hope you find the essay you're looking for, but I think this one is better for the audience.
I really needed to be more clear that the end of the second paragraph was referring to the math problems. I'm aware Alzheimer's is real life and that the field is still active; my point stands that it's more useful to pick a less-controversial topic and to pick apart older research when learning fundamentals.
Talking about COVID vaccines... I mean, good grief, the person to whom I was responding asked for a paper WITHOUT an agenda.
You can't do science if you already know the answer. Science is all about working with uncertainty.
You aren't getting the "a priori reasoning" in the essay above. You're missing out on the fails. You're getting an ex post facto "right and true" version of "how to attempt to flunk a paper."
Seneff walks you through the potential for an antifreeze allergy in a significant segment of the population. I'm pretty confident we didn't hit that failure mode of the mRNA vaccines.
How do you know that you're doing it correctly? You aren't, unless you're uncertain, and working with uncertainty. This isn't math, math gets things correct. Science just makes a better model -- and lo, if tomorrow the model is no longer accurate, we will build a new one (Yes, there's a model out there for "no dark matter, gravity is just different in the galactic arms" -- and yes, if you wanted to look at "how to disprove dark matter" you could look at that paper. But that's just running "one disprove" not Seneff's "here's ten reasons we shouldn't do this").
It took me a while to understand the introduction, because I assumed that he was talking about some specific paper. I thought it was a complaint about the rigor of some paper that had yet to be explained, and I guess I was supposed to click on the link to see which paper was being complained about and why it's surprising that a Nobel laureate would complain about it.
Perhaps inserting "[the entire concept of]" into the quote would help.
Who were the brave souls who first articulated these concerns and what do we know of what became of their concerns and their careers? In other words, science should only work with constant critical appraisal. It seems clear that this crityappraisal was either absent or smashed by powerful forces. That in itself is of course age old. But knowing it is age old should it be excused or should we learn from it?
This is an exceedingly important question to answer because it can be used to broadly determine if medicine is a science and whether its conclusions are trustworthy. If the careers of the scientists who question the amyloid hypothesis are healthy, then medicine is operating more or less like a science and can be treated as such. However, if the questioners were instead expelled because of their concerns, then medicine is not a science and should not be given the same level of trust.
Good. These are the right questions to be asking. Now, assume malevolence (devil's advocate if you must). The Pharma companies want to make money, so they're going to throw money at people whose ideas have potential therapeutics attached. So, yeah, I'm pretty sure you're going to see that the questioners did not get the shiny, big rewards -- be they patents($$$), status, or otherwise.
The existence of failure modes doesn’t invalidate the entire modality.
True. How much money do we need to waste before you decide that capitalism is the wrong way to approach medicine? (or: state the failure condition you'd accept for "we need a new solution here").
Is there a good step by step guide to reading scientific papers anywhere? This post has a good start at one, but having a checklist of things to think about would be pretty helpful.
Read first, bet later: as always here is the manifold market for who will win the not-a-book review contest, now updated with this candidate!
https://manifold.markets/BayesianTom/who-will-win-acxs-everythingexceptb
Since we have a list of most of the finalists (from https://www.astralcodexten.com/p/open-thread-387), wouldn't it make sense to add them all to the market now?
As is, I'm confused about what it means to buy "Other". Naively one might think that a vote for "Other" would mean that some review other than the three currently in the market would win. But you're planning to add reviews as they are posted, and eventually all possible reviews should appear in the market. Doesn't this mean that "Other" can never possibly win?
The illustrative application of scientific rigor is what marks the best of this essay. I've been attempting to work on the 'detective' mode while reading papers for a bit, so I'm glad to read this well-written piece. Overall, I'd say this leaves the story a bit unfinished by it's own promise initially. This essay could've used a larger view along with this segment: on other evidence ignored, on more recent hypotheses that show promise. Scott's comment addresses another potential direction too.
Nice to see someone on this site quoting Medawar, who was, of course, a Popperian. I've tried in the past to get Scott--and also his mentor, Eliezer Yudkowsky--to take a proper look at Popper's theory of knowledge, including his critique of induction (including the Bayesian one they subscribe to). Sadly, they insist on dismissing Popper, not--as far as I can tell--on the basis of anything he actually said, but on the basis of a strawman version that appears in the secondary literature. Sigh.
I liked this review for its hands on review of the science, but I’m not really convinced that amyloids aren’t a predicate to Alzheimer’s. I’m not sure anyone holds the strong hypothesis at all these days, and critique of a 20 year old paper doesn’t seem like a persuasive way to argue against the weak one.
Having read the whole essay I'm still unsure what exactly this is a review of? Overly condensed scientific papers? The Alzheimers literature? Making transgenic mice? This feels like a regular essay with the word "review" slapped on it so that it can meet the criteria for a review competition.
>Through a painstaking process called sub-cloning, equal parts molecular biology and divination, they managed to insert into their mouse a human APP gene carrying the mutation found in families with high rates of early-onset Alzheimer's.
>You can design your construct perfectly on paper, but in truth, you solve the problem by tweaking reagents like an alchemist, trying to find the perfect brew to coax your foreign gene into the plasmid at high efficiency.
I guess I'm an alchemist then :) I didn't choose the name Metacelsus for nothing...
More seriously, I'm glad I'm not stuck with the molecular biology tools of 1995. Or worse, 1984, when researchers had to clone a gene by literally scraping pieces of mouse chromosome 17 off of glass slides using tiny needles. https://pubmed.ncbi.nlm.nih.gov/6697397/