> 22: Creating Intelligent Tutoring Systems (5/10)
... They are concerned about whether large language models will obsolete their product, and interested in talking to anyone who thinks they can predict LLM’s near-future performance - if this is you, contact James Koppel.
I am a bumpkin myself, but I know a couple smart people who might be interested. Is there some specific writeup or URL I can send them to get them up to speed on the main project -- "Creating Intelligent Tutoring Systems" ? I don't want to just send them to Koppel's main site, that would be too vague...
I'm very amused by '40: Typist For Saharon Shelah (7/10)'. Guess he just keeps getting nerdsniped by new ideas and typing things up in LaTeX properly is *such* a pain...
> In practice I’m not sure what I learned from reading them.
I think you can extract some insights:
* Politics-related proposals are unlikely to ever get done, regardless of how much funding they get.
* Projects with specific actionable goals are more likely to achieve progress than projects with vague feel-good mission statements.
* Teams perform better than individuals (probably because the existence of a team is correlated with existing progress having been made, though that's just my conjecture).
Reading about Kirschman's work reminds me of this bit from Ringworld:
> How strange, if new life evolves capable of living on the garbage?
> “That happened on Earth once,” said Louis Wu. “A yeast that could eat polyethylene. It was eating the plastic bags off the supermarket shelves. It’s dead now. We had to give up polyethylene.”
As someone who has spent plenty of time in academic and political settings, the optimism, flexibility, and open-minded experimentation your grants represent bring me great joy.
"4: Alice Evans’ Research On “The Great Gender Divergence” (?/10)
Dr. Evans has done over four months of research in Morocco, Italy, India, and Turkey. You can find some of her most recent thoughts at her blog here. Her book is still on track to be published from Princeton Press, more details tbd."
I still remain to be convinced that this is going to be of any use to anyone. It's not like this lady academic is starving in a garret with her inspirational work confined to a heap of notebooks under the bed because no-one will publish her, so why does she need a dig-out with an ACX grant?
The Princeton publishing blurb is also ignorant, at least in its effusive enthusiasm that nobody nowhere never did nothing before us did (this ties back in to the query raised on here about "Why learn history?" Well, for a start, so that you don't sound like a Princeton University Press intern):
"Something radical happened over the twentieth century: women entered the workforce and became political leaders across the world. This had never happened before, not in the entirety of human history."
Empress Matilda? Who dat?
Elizabeth I? Never heard of her
Catherine the Great? Is this a cosmetics brand?
Every damn woman who ever milked a cow and churned butter which was then sold at market? Don't be silly, that's not "work" so you never existed anyhow.
The most interesting bit I got out of these reports was "which grantees either didn't understand Scott's instructions that 5/10 was the expected amount of progress, or inflated their scores to match more standard out-of-10 rating systems".
Judging by comparison with the textual descriptions, I'd estimate at least a third of the grantees did this. I put much more probability on grantees' systematic incentives to inflate than on Scott's theory that things went systemically better than expected.
That being said, this whole thing remains super cool and I'm not nearly as surprised as Scott that the ACX exposure counts for a lot on its own!
Some of the reports make me wonder whether _two_ grades would be more informative: basically «help us do X to achieve Y» has «we did X as planned and with good quality» and «X helped with Y» parts. I.e. it looks like 6. did things in their control fine, but things out of control played out not so well this time.
And maybe in addition to the median 5/10, it's good to say something about spread: is «needed to win a fair coin toss, won/lost» 6/10 and 4/10?
Regarding Promote Economically Literate Climate Policy In US States comment "Overall they report frustration, as many of the legislators they worked with have been voted out or term-limited": Possible silver lining. I recall a Cato policy analysis from many years ago that reviewed congressional representative's voting records relative to time in office. The longer they are there the more they tend toward government run solutions and oppose private alternatives. If so, and assuming what they found at the federal level applies as well at the state level, the fact people have been voted out (and thus replaced by freshmen legislators) may in the end be helpful to their plans.
We did not get one of these grants, but we were listed on the applicants thread, and it helped us get some help with the project. Since then, we've partnered with a major law firm and are investigating PPP loan fraud. It's been a lot of work, but it's nice to have apply our analytics skills to something rewarding instead of helping a giant business optimize click-through rates or something. We can't talk about any cases until they are brought out from under seal but we're extremely happy with our traction and we've begun making inroads helping healthcare payers such as health insurance companies, unions, and large employers, prevent fraud.
Thank you very much for running this program: you wound up introducing us to our first angel investor. He really helped us understand how this could be more than a cute idea to punish some bad guys, and that has paved the way for the other angels which are following. We really appreciate that programs like this exist, and will report back at the next ACX grant update.
I have no other comment other than to say that I enjoyed reading this greatly, more so than the original grant announcement, and found this greatly energizing
I scanned through. There is good here, stupid here, smart, and bad. With so many grants and so many apparently legitimate agendas or motivations, I suppose it can’t be help. But it would seem the ACX grants organisation pick and parse a bit more in their presentation of the outcomes of the grantees. I believe that’s the word. In the interest of everyone I think. None of us are going to be able to take the time to outside of ASX look at each one of these 1 year result reports. ASX should be more judicious and less, and pick and parse through the language they’re using to present the reports to the community. A bit of philosophical grammar, evaluate modality. But thanks to ASX, it is appreciated -- the grants I mean.
Why is this only 8/10? It sounds like they succeeded in their goal; what would have resulted in them rating it 10/10? Were they hoping for a more prestigious school?
> The grant was to help make it financially easier for him to go on a long round of interviews [Anonymous] successfully got a job offer from a top school[...]
On the 6.5/10 average: might this not be simple publication bias? That is, people being less likely to send you progress reports if things aren't going as well?
The ratings are an interesting psych experiment in themselves. I read the instructions in the post and thought "yeah, that's not going to work as intended," and I think that was borne out by the ratings given.
If you've met your goals, it's psychologically extremely difficult to give a rating of 5/10 - I mean, 50% isn't even a passing grade! Hence the reports that say things like "we had some unexpected setbacks but achieved most of what we intended, 7/10" when that should probably be 4/10.
I think it's (mostly) not dishonest inflation, but people either finding it too hard to overcome their cognitive bias or simply misreading the instructions (like the comment from Isaac King).
Scott, I think if you do this again you should say "Rate your progress on a scale of 1 to 5 relative to what you expected. If things went even better than you expected, you can give out-of-range answers like 6/5 or 7/5." Logically equivalent, but it will make all the difference psychologically: I think you'd get an average rating of about 4.
For the ones that say "They have asked me not to discuss their progress yet." my hope would be that once progress updates are allowed they come with a *really* good explanation of why progress updates were withheld. I am personally a big fan of open science, and keeping information private feels like it means you are striving to achieve something other than "betterment of society/humanity" i.e., not particularly altruistic.
If these people can provide great rationale for keeping the information private (such as releasing of a status update would cause some kind of bias in their data collection) then we can all walk away thinking it was a wise decision.
If, on the other hand, they just wanted to keep updates private because they were waiting for a patent application or to secure their position as a "leader" or to get their work published in some reputable journal then I think future grants should have a stipulation that grant recipients *MUST* share status updates and not withhold details.
I got a grant by being forwarded to the LTFF for some concrete ideas on expanding on a text about forecasting[1]. I have done 4/6 of the things I promised I would do, the two missing ones being adding a literature review and extending the analysis to continuous questions. I give myself a 6/10.
To speak a bit more about how contingent Manifold's existence was on ACX Grants: I think there's like an 50% chance that me/James/Stephen would not have chosen prediction markets (among our other crypto ideas) if we weren't submitting a proposal ACX grants specifically; crypto was on our minds as you can see from our original application: https://docs.google.com/document/d/12DufTzcIaJxSQOxMYIHAwt7tGa5PAkIyhZOQ2JrmCfg/edit
I also think there's a 60% chance we wouldn't have continued with a prediction markets project if we hadn't won an ACX Grant and gotten the subsequent publicity through that announcement and Mantic Mondays. ACX readers formed the bulk of Manifold's initial adopters, and still represent our most vocal and engaged users.
We owe a lot to ACX and Scott in particular, and hope to pay back that generosity with useful predictions of all kinds!
They're planning to spend 2/3 of the tax money on tax credits for poor/mid incomes and eliminating sales tax on grocery store food. Then an additional $93M goes to rural economic development, airports and highways for some reason?
Only about 1/6 of it, $100M, goes to actually cleaning the air, which they say they'll spend on "cleaner school buses, electrifying lawn and garden equipment". So like a fleet of shiny new electric school buses and a mail-in rebate if you buy an electric lawnmower? And nothing about toxic dust from the Great Salt Lake drying up due to the drought? If you're going to spend $50M on rural economic development, shouldn't that at least be aimed at water conservation?
> The average project rated their progress at 6.5/10, where 5 was “meets expectations” (this average includes some private projects I didn’t fully list here). So either people systematically underestimated their chances when applying, or they’re systematically inflating their progress now to look good, or things systematically went better than expected.
No, none of those things. People are absolutely awful at assessing their progress. They think they're doing better than they are - they're not inflating their progress to look good to you, but they are probably inflating the progress to *themselves*, and **certainly** systematically ridiculously overoptimistic about how their progress compares to their problem. Which is normal. Wikipedia's planning fallacy example has average real-case estimate 33.9 days, best-case 27.4 days, worst-case 48.6 days; reality averaged 55.5 days, 70% of students taking longer than predicted. That's with a fairly narrow scope where they think they perfectly understand the problem, which is applicable for about zero of these grants.
"Do 90% of the work in the first nine days, and then the other 90% of the work on the tenth day." Halve the ratings and you'd have answers closer to the truth. Though some of them are still way too optimistic by that measure. Anyone who cites getting more money as progress, or is looking for more money now, halve their score again. Or just round down to 0, that's probably more accurate.
Re. this: "Trevor Klee...realized this would be too expensive to do in humans in the current funding environment, and has pivoted to getting his medication approved for a feline autoimmune disease as both a proof-of-concept and as a cheaper, faster way to start making revenue. ... He still anticipates eventually moving back to humans."
This is a very common trap. I don't know of a single case where it worked out for the humans. A company wants to develop a drug for humans, but FDA approval is too expensive, so they get it approved in dogs, horses, or cats, thinking they'll make enough money to push it through approval for humans. They never do; it's hard enough to keep a new pet-drug business above water. Then their patent expires, and the drug will never, ever be approved for humans, because nobody will pay for FDA approval of an unpatentable drug.
Until just a few years ago, this was almost acceptable, because you could go to Agway or a foreign website and buy the drug for your dog and take it yourself. But governments and industry have taken EXTREMELY aggressive steps in the past few years to prevent this.
An example is Adequan. It temporarily cures arthritis in humans. It cures arthritis in me! But it will never, ever be legal for humans in the US or Europe, because it was developed in the 1980s and the patent expired long ago. But it is approved for dogs, horses, and pigs in the US.
Until very recently, I could easily buy it over-the-counter in any country but America, or order it thru ebay or from any overseas veterinary website. But organizations have just taken many steps to prevent any humans from taking Adequan, including:
- turning it into a prescription-only drug in all Western nations
- (I presume) prosecuting online pharmacies that don't comply (as they all suddenly went out of business or stopped selling it)
- purging overseas pharmacies that sell it over-the-counter from search engine results
- purging pharmacies that sell it without prescription from TrustPilot listings
- creating an organization to rate the "safety" of online pharmacies, which actually rates their compliance with import regulations
- investigating ebay sales of Adequan to verify possession of a veterinary prescription
- banning ebay users who sell it to people who have no such prescription
- delisting Adequan sales on ebay entirely
- in my state, creating laws for dog licensing that forbid veterinarians from treating unlicensed dogs, and forbid licensing dogs without proof of legal ownership of the dog, including an investigation process to verify possession of alleged stray dogs, so that you can't even get Adequan by borrowing your neighbor's arthritic dog, licensing it, and taking it to the vet to get Adequan (I tried)
I can't think of any reason anyone might have to prevent people from taking Adequan, so I think it's just part of a general tightening of access to veterinary drugs, which has been going on for about the past 5 years. The drugs for which doing this makes some sense -- antibiotics -- were left for last, but the US govt. has finally gotten around to them, and IIRC sometime next year it will become impossible to buy any veterinary antibiotics without a prescription.
Besides just bias and inflation, one might have different scales (that is, completion could be 10 for someone and just a 7 for others, an expected benefit could one's 8 or another's 11).
Also note - I'm not sure if we can even avg these scores out in the first place. As in, if you get a 3 and 7, does that equal 2 5s? The 3 might be much worse compared to a 5 than the 7 is better.
Ignoring all that lets have some fun making some other markers for the scale.
Limited scale -
1 - Failure
3 - Delay
5 - Expected Progress
7 - Full Completion without issue
9 - Unexpected great success
From that, I redid the scores from my not that optimistic guesses based only on the descriptions.
1-10 6?8?336756
11+ 8?55623735
21+ ?2867?645?
31+ 6759?5?676
Avg non ? ~5.5
Note for 30 I changed to a ? cause it didn't really fit in the scale. Also on a few I graded actually a bit higher than what they gave themselves.
The avg for the visible ones above was 6.67 so if we assume the invisible ones were roughly similarly biased, then it's still just about a point difference. Doesn't mean there wasn't a lot of bias - the most common dif was a 2, but the next highest was 0, so it avg out.
Full (not to scale) scale
0 - Likely scammed/tricked. Not necessarily by the grantee, mind you.
1 - Canceled/ full failure and can't continue/ hit obstacle that would warrant a cancellation at any point.
2 - Significant delay or backtracking. Unknown or difficult to guess how major it is. (note that this could be worse than 1 in many cases. At least 1 might be quick)
3 - Stalled or "small" failures. Generally still able to see an end, but might have already failed to meet some goals/objectives.
4 - Minor obstacles that have caused some delay/deviation from estimates, but nothing that could count as failures.
5 - Free Space
6 - Smooth progress.
7 - Full success
8 - Went well enough that it expanded scope/target of initial project/task. (Note - might want to do 8->3 or something if expanding it caused more issues later on...)
My main takeaway is people suck at using numeric scales.
10/10 for achieving the first step out of a many step failure prone process? 8/10 for doing precisely and exactly what they said they'd do? You marked 5/10 as "as expected", what's up with all the insane grade inflation? I assume it's because nobody think of 5/10 as being "as expected", people don't want to think of themselves as failures (which 5/10 feels like), so they mark themselves up ridiculously?
> 22: Creating Intelligent Tutoring Systems (5/10)
... They are concerned about whether large language models will obsolete their product, and interested in talking to anyone who thinks they can predict LLM’s near-future performance - if this is you, contact James Koppel.
I am a bumpkin myself, but I know a couple smart people who might be interested. Is there some specific writeup or URL I can send them to get them up to speed on the main project -- "Creating Intelligent Tutoring Systems" ? I don't want to just send them to Koppel's main site, that would be too vague...
I'm very amused by '40: Typist For Saharon Shelah (7/10)'. Guess he just keeps getting nerdsniped by new ideas and typing things up in LaTeX properly is *such* a pain...
> In practice I’m not sure what I learned from reading them.
I think you can extract some insights:
* Politics-related proposals are unlikely to ever get done, regardless of how much funding they get.
* Projects with specific actionable goals are more likely to achieve progress than projects with vague feel-good mission statements.
* Teams perform better than individuals (probably because the existence of a team is correlated with existing progress having been made, though that's just my conjecture).
Really cool. Thank you for posting updates!
Reading about Kirschman's work reminds me of this bit from Ringworld:
> How strange, if new life evolves capable of living on the garbage?
> “That happened on Earth once,” said Louis Wu. “A yeast that could eat polyethylene. It was eating the plastic bags off the supermarket shelves. It’s dead now. We had to give up polyethylene.”
Let's hope it stays in vitro :P
This is the best end to the week
This post made me so happy.
As someone who has spent plenty of time in academic and political settings, the optimism, flexibility, and open-minded experimentation your grants represent bring me great joy.
"4: Alice Evans’ Research On “The Great Gender Divergence” (?/10)
Dr. Evans has done over four months of research in Morocco, Italy, India, and Turkey. You can find some of her most recent thoughts at her blog here. Her book is still on track to be published from Princeton Press, more details tbd."
I still remain to be convinced that this is going to be of any use to anyone. It's not like this lady academic is starving in a garret with her inspirational work confined to a heap of notebooks under the bed because no-one will publish her, so why does she need a dig-out with an ACX grant?
The Princeton publishing blurb is also ignorant, at least in its effusive enthusiasm that nobody nowhere never did nothing before us did (this ties back in to the query raised on here about "Why learn history?" Well, for a start, so that you don't sound like a Princeton University Press intern):
"Something radical happened over the twentieth century: women entered the workforce and became political leaders across the world. This had never happened before, not in the entirety of human history."
Empress Matilda? Who dat?
Elizabeth I? Never heard of her
Catherine the Great? Is this a cosmetics brand?
Every damn woman who ever milked a cow and churned butter which was then sold at market? Don't be silly, that's not "work" so you never existed anyhow.
The most interesting bit I got out of these reports was "which grantees either didn't understand Scott's instructions that 5/10 was the expected amount of progress, or inflated their scores to match more standard out-of-10 rating systems".
Judging by comparison with the textual descriptions, I'd estimate at least a third of the grantees did this. I put much more probability on grantees' systematic incentives to inflate than on Scott's theory that things went systemically better than expected.
That being said, this whole thing remains super cool and I'm not nearly as surprised as Scott that the ACX exposure counts for a lot on its own!
Some of the reports make me wonder whether _two_ grades would be more informative: basically «help us do X to achieve Y» has «we did X as planned and with good quality» and «X helped with Y» parts. I.e. it looks like 6. did things in their control fine, but things out of control played out not so well this time.
And maybe in addition to the median 5/10, it's good to say something about spread: is «needed to win a fair coin toss, won/lost» 6/10 and 4/10?
It looks a little strange to me that David Bahry apparently doesn't acknowledge the ACX funding in his academic works.
Regarding Promote Economically Literate Climate Policy In US States comment "Overall they report frustration, as many of the legislators they worked with have been voted out or term-limited": Possible silver lining. I recall a Cato policy analysis from many years ago that reviewed congressional representative's voting records relative to time in office. The longer they are there the more they tend toward government run solutions and oppose private alternatives. If so, and assuming what they found at the federal level applies as well at the state level, the fact people have been voted out (and thus replaced by freshmen legislators) may in the end be helpful to their plans.
We did not get one of these grants, but we were listed on the applicants thread, and it helped us get some help with the project. Since then, we've partnered with a major law firm and are investigating PPP loan fraud. It's been a lot of work, but it's nice to have apply our analytics skills to something rewarding instead of helping a giant business optimize click-through rates or something. We can't talk about any cases until they are brought out from under seal but we're extremely happy with our traction and we've begun making inroads helping healthcare payers such as health insurance companies, unions, and large employers, prevent fraud.
Thank you very much for running this program: you wound up introducing us to our first angel investor. He really helped us understand how this could be more than a cute idea to punish some bad guys, and that has paved the way for the other angels which are following. We really appreciate that programs like this exist, and will report back at the next ACX grant update.
I have no other comment other than to say that I enjoyed reading this greatly, more so than the original grant announcement, and found this greatly energizing
This is awesome, Scott. You are a force for good and an inspiration.
I am amused that it was hardest to detect progress on the neutrino grant.
I scanned through. There is good here, stupid here, smart, and bad. With so many grants and so many apparently legitimate agendas or motivations, I suppose it can’t be help. But it would seem the ACX grants organisation pick and parse a bit more in their presentation of the outcomes of the grantees. I believe that’s the word. In the interest of everyone I think. None of us are going to be able to take the time to outside of ASX look at each one of these 1 year result reports. ASX should be more judicious and less, and pick and parse through the language they’re using to present the reports to the community. A bit of philosophical grammar, evaluate modality. But thanks to ASX, it is appreciated -- the grants I mean.
> Help [Anonymous] Interview For A Professorship
Why is this only 8/10? It sounds like they succeeded in their goal; what would have resulted in them rating it 10/10? Were they hoping for a more prestigious school?
> The grant was to help make it financially easier for him to go on a long round of interviews [Anonymous] successfully got a job offer from a top school[...]
Typo, missing period after "interviews".
Great to read about all those promising things!
On the 6.5/10 average: might this not be simple publication bias? That is, people being less likely to send you progress reports if things aren't going as well?
The ratings are an interesting psych experiment in themselves. I read the instructions in the post and thought "yeah, that's not going to work as intended," and I think that was borne out by the ratings given.
If you've met your goals, it's psychologically extremely difficult to give a rating of 5/10 - I mean, 50% isn't even a passing grade! Hence the reports that say things like "we had some unexpected setbacks but achieved most of what we intended, 7/10" when that should probably be 4/10.
I think it's (mostly) not dishonest inflation, but people either finding it too hard to overcome their cognitive bias or simply misreading the instructions (like the comment from Isaac King).
Scott, I think if you do this again you should say "Rate your progress on a scale of 1 to 5 relative to what you expected. If things went even better than you expected, you can give out-of-range answers like 6/5 or 7/5." Logically equivalent, but it will make all the difference psychologically: I think you'd get an average rating of about 4.
For the ones that say "They have asked me not to discuss their progress yet." my hope would be that once progress updates are allowed they come with a *really* good explanation of why progress updates were withheld. I am personally a big fan of open science, and keeping information private feels like it means you are striving to achieve something other than "betterment of society/humanity" i.e., not particularly altruistic.
If these people can provide great rationale for keeping the information private (such as releasing of a status update would cause some kind of bias in their data collection) then we can all walk away thinking it was a wise decision.
If, on the other hand, they just wanted to keep updates private because they were waiting for a patent application or to secure their position as a "leader" or to get their work published in some reputable journal then I think future grants should have a stipulation that grant recipients *MUST* share status updates and not withhold details.
"Saharon is a prolific and influential Israeli mathematician, but many of his discoveries are hand-written in an unpublishable format"
Put me in mind of this video on the history of the Fast Fourier Transform - https://youtu.be/nmgFG7PUHfo
Spoiler: Gauss discovered the algorithm originally, but nobody noticed as his written description was too obscure.
I got a grant by being forwarded to the LTFF for some concrete ideas on expanding on a text about forecasting[1]. I have done 4/6 of the things I promised I would do, the two missing ones being adding a literature review and extending the analysis to continuous questions. I give myself a 6/10.
[1]: https://niplav.github.io/range_and_forecasting_accuracy.html
To speak a bit more about how contingent Manifold's existence was on ACX Grants: I think there's like an 50% chance that me/James/Stephen would not have chosen prediction markets (among our other crypto ideas) if we weren't submitting a proposal ACX grants specifically; crypto was on our minds as you can see from our original application: https://docs.google.com/document/d/12DufTzcIaJxSQOxMYIHAwt7tGa5PAkIyhZOQ2JrmCfg/edit
I also think there's a 60% chance we wouldn't have continued with a prediction markets project if we hadn't won an ACX Grant and gotten the subsequent publicity through that announcement and Mantic Mondays. ACX readers formed the bulk of Manifold's initial adopters, and still represent our most vocal and engaged users.
We owe a lot to ACX and Scott in particular, and hope to pay back that generosity with useful predictions of all kinds!
I want to like www.cleanthedarnair.org, but it looks way too greenwashed.
They're planning to spend 2/3 of the tax money on tax credits for poor/mid incomes and eliminating sales tax on grocery store food. Then an additional $93M goes to rural economic development, airports and highways for some reason?
Only about 1/6 of it, $100M, goes to actually cleaning the air, which they say they'll spend on "cleaner school buses, electrifying lawn and garden equipment". So like a fleet of shiny new electric school buses and a mail-in rebate if you buy an electric lawnmower? And nothing about toxic dust from the Great Salt Lake drying up due to the drought? If you're going to spend $50M on rural economic development, shouldn't that at least be aimed at water conservation?
> The average project rated their progress at 6.5/10, where 5 was “meets expectations” (this average includes some private projects I didn’t fully list here). So either people systematically underestimated their chances when applying, or they’re systematically inflating their progress now to look good, or things systematically went better than expected.
No, none of those things. People are absolutely awful at assessing their progress. They think they're doing better than they are - they're not inflating their progress to look good to you, but they are probably inflating the progress to *themselves*, and **certainly** systematically ridiculously overoptimistic about how their progress compares to their problem. Which is normal. Wikipedia's planning fallacy example has average real-case estimate 33.9 days, best-case 27.4 days, worst-case 48.6 days; reality averaged 55.5 days, 70% of students taking longer than predicted. That's with a fairly narrow scope where they think they perfectly understand the problem, which is applicable for about zero of these grants.
"Do 90% of the work in the first nine days, and then the other 90% of the work on the tenth day." Halve the ratings and you'd have answers closer to the truth. Though some of them are still way too optimistic by that measure. Anyone who cites getting more money as progress, or is looking for more money now, halve their score again. Or just round down to 0, that's probably more accurate.
Re. this: "Trevor Klee...realized this would be too expensive to do in humans in the current funding environment, and has pivoted to getting his medication approved for a feline autoimmune disease as both a proof-of-concept and as a cheaper, faster way to start making revenue. ... He still anticipates eventually moving back to humans."
This is a very common trap. I don't know of a single case where it worked out for the humans. A company wants to develop a drug for humans, but FDA approval is too expensive, so they get it approved in dogs, horses, or cats, thinking they'll make enough money to push it through approval for humans. They never do; it's hard enough to keep a new pet-drug business above water. Then their patent expires, and the drug will never, ever be approved for humans, because nobody will pay for FDA approval of an unpatentable drug.
Until just a few years ago, this was almost acceptable, because you could go to Agway or a foreign website and buy the drug for your dog and take it yourself. But governments and industry have taken EXTREMELY aggressive steps in the past few years to prevent this.
An example is Adequan. It temporarily cures arthritis in humans. It cures arthritis in me! But it will never, ever be legal for humans in the US or Europe, because it was developed in the 1980s and the patent expired long ago. But it is approved for dogs, horses, and pigs in the US.
Until very recently, I could easily buy it over-the-counter in any country but America, or order it thru ebay or from any overseas veterinary website. But organizations have just taken many steps to prevent any humans from taking Adequan, including:
- turning it into a prescription-only drug in all Western nations
- (I presume) prosecuting online pharmacies that don't comply (as they all suddenly went out of business or stopped selling it)
- purging overseas pharmacies that sell it over-the-counter from search engine results
- purging pharmacies that sell it without prescription from TrustPilot listings
- creating an organization to rate the "safety" of online pharmacies, which actually rates their compliance with import regulations
- investigating ebay sales of Adequan to verify possession of a veterinary prescription
- banning ebay users who sell it to people who have no such prescription
- delisting Adequan sales on ebay entirely
- in my state, creating laws for dog licensing that forbid veterinarians from treating unlicensed dogs, and forbid licensing dogs without proof of legal ownership of the dog, including an investigation process to verify possession of alleged stray dogs, so that you can't even get Adequan by borrowing your neighbor's arthritic dog, licensing it, and taking it to the vet to get Adequan (I tried)
I can't think of any reason anyone might have to prevent people from taking Adequan, so I think it's just part of a general tightening of access to veterinary drugs, which has been going on for about the past 5 years. The drugs for which doing this makes some sense -- antibiotics -- were left for last, but the US govt. has finally gotten around to them, and IIRC sometime next year it will become impossible to buy any veterinary antibiotics without a prescription.
Besides just bias and inflation, one might have different scales (that is, completion could be 10 for someone and just a 7 for others, an expected benefit could one's 8 or another's 11).
Also note - I'm not sure if we can even avg these scores out in the first place. As in, if you get a 3 and 7, does that equal 2 5s? The 3 might be much worse compared to a 5 than the 7 is better.
Ignoring all that lets have some fun making some other markers for the scale.
Limited scale -
1 - Failure
3 - Delay
5 - Expected Progress
7 - Full Completion without issue
9 - Unexpected great success
From that, I redid the scores from my not that optimistic guesses based only on the descriptions.
1-10 6?8?336756
11+ 8?55623735
21+ ?2867?645?
31+ 6759?5?676
Avg non ? ~5.5
Note for 30 I changed to a ? cause it didn't really fit in the scale. Also on a few I graded actually a bit higher than what they gave themselves.
The avg for the visible ones above was 6.67 so if we assume the invisible ones were roughly similarly biased, then it's still just about a point difference. Doesn't mean there wasn't a lot of bias - the most common dif was a 2, but the next highest was 0, so it avg out.
Full (not to scale) scale
0 - Likely scammed/tricked. Not necessarily by the grantee, mind you.
1 - Canceled/ full failure and can't continue/ hit obstacle that would warrant a cancellation at any point.
2 - Significant delay or backtracking. Unknown or difficult to guess how major it is. (note that this could be worse than 1 in many cases. At least 1 might be quick)
3 - Stalled or "small" failures. Generally still able to see an end, but might have already failed to meet some goals/objectives.
4 - Minor obstacles that have caused some delay/deviation from estimates, but nothing that could count as failures.
5 - Free Space
6 - Smooth progress.
7 - Full success
8 - Went well enough that it expanded scope/target of initial project/task. (Note - might want to do 8->3 or something if expanding it caused more issues later on...)
9 - Success + recognition/major achievement/discovery/etc.
10 - Made world news. In a good way. Hopefully.
Thank you so much to Scott and all the ACX readers who have supported Legal Impact for Chickens!
My main takeaway is people suck at using numeric scales.
10/10 for achieving the first step out of a many step failure prone process? 8/10 for doing precisely and exactly what they said they'd do? You marked 5/10 as "as expected", what's up with all the insane grade inflation? I assume it's because nobody think of 5/10 as being "as expected", people don't want to think of themselves as failures (which 5/10 feels like), so they mark themselves up ridiculously?