> 22: Creating Intelligent Tutoring Systems (5/10)
... They are concerned about whether large language models will obsolete their product, and interested in talking to anyone who thinks they can predict LLM’s near-future performance - if this is you, contact James Koppel.
I am a bumpkin myself, but I know a couple smart people who might be interested. Is there some specific writeup or URL I can send them to get them up to speed on the main project -- "Creating Intelligent Tutoring Systems" ? I don't want to just send them to Koppel's main site, that would be too vague...
My guess is that they panicked on June/July when Google disclosed Minerva.
Although it may look very impressive, I think that this is one of the cases where they reached the LLMs show their limits.
-the dataset for Math/Physics problem is too small, so they should have been able to get almost all of it
-the marginal returns are heavy
-it's one of those cases where you need more automated reasoning and a top down approach to problem (it's important to stress out a few things pedagogically and the LLM's can be quite bad at that).
Moreover I am a bit skeptical about using LLMs to do this stuff for a number of things. If you want to sell the product:
-you want to have something that allows the student to cheat at test;
-you want to have something that helps the teacher (sort of like a super-book).
I take Caplan side regarding education. Unless you have a very excellent product (and not something merely promising) the Institutions and your possible clients are going to dismiss you.
At the start of the year, an AI expert interested in tutoring told me "Anything you do will be wiped out by a few years of gains in LLMs, so the best thing you can do is collect training data." A few weeks ago, I got some insider info that makes me inclined to think he's right.
As for the skepticism about LLM's capabilities: you're making a lot of assumptions about what the tutoring would look like. Key term to search for is "interaction plateau hypothesis." Why2-ATLAS does work exactly like you describe except fancier, and it's only been shown to produce marginally better learning outcomes than much dumber systems.
> it's important to stress out a few things pedagogically
We're actually pretty good at this just with our canned feedback database. :)
BTW, I am an automated reasoning guy (never published in IJCAR or CADE, but am pretty well-versed in that literature), so it is quite painful to admit the supremacy of neural over deductive techniques.
> the Institutions and your possible clients are going to dismiss you.
The main planned client is my own training company Mirdin, so we're covered there. Institutions were never on the table as possible customers. Some of my former coauthors and other academic colleagues will be free users of the open source version, furthering the altruistic aims of the grant. If we do productize it, likely other customers will be other small upskilling businesses like my own, although a few SAT tutors have also reached out.
IDK I have just a hard time seeing how it is helpful. What "work" is the LLM doing? It cannot really assess how responsive or engaged the tutee is, nor motivate them if not.
I don't know I find good tutor is a mix of feeding people excuses and rationalizations and guilty trips and good/bad examples, and blaming others, and blaming their past selves, and much more emotional manipulation all predicated on their moment-to-moment emotional states.
The material isn't the issue, getting people to fire on all cylinders is. How is a LLM helping with that?
I'll be waiting for the insider info to get public then! As of now I remain a skeptical of large improvements of LLMs in this regards, but I'll be happy to be proven wrong. May be not so happy since I am currently working for a competitor, so I'll need to be looking for another job ;)
Well I don’t even understand how it is supposed to be helpful. For most professional an academic tutoring I have done the issue isn’t having an algorithm. There already is an algorithm, RTFM. A lot of the professional tutoring there is like an 800 page manual and oodles of fact sheets and guides and training materials that are totally sufficient. What the tutees lack is the emotional intelligence to actually go through that.
So they need a Sherpa to hold them accountable and cajole, encourage, condense, harass them into learning and applying themselves. It is rare that they are actually too stupid to get it, they just need the right motivators and someone to hold their hand and tell them it will be ok if you just try hard and listen to me.
> am a bumpkin myself, but I know a couple smart people who might be interested. Is there some specific writeup or URL I can send them to get them up to speed on the main project
Send me your contact info and I'll share the original grant proposal.
I'm very amused by '40: Typist For Saharon Shelah (7/10)'. Guess he just keeps getting nerdsniped by new ideas and typing things up in LaTeX properly is *such* a pain...
I would say «writing up the complete article around the core of the proof» is such a pain, and a typist can at least remove one part of the pain. Not all of it, but I guess it's about pushing things below the pain threshold!
Plus if you have someone periodically asking you "so, what else do you have for me to type up?", doing something that would result in an affirmative answer rather than blushing with "nope, I am squandering the grant money" is an incentive in itself.
Shelah is actually one of the most prolific mathematicians of all time. According to the database of Erdös numbers, as of 2004, "There are four authors with more than 700 papers: Paul Erdös with 1416 (he actually wrote more papers than that, but these are just the ones covered by Math Reviews), Drumi Bainov with 823, SAHARON SHELAH with 760, and Leonard Carlitz with 730."
A friend of mine who collaborated with Shelah described the process, which sounds a lot like what I've heard for collaborating with Erdös - you have an idea you're working on in one of the fields he works on; you arrange a meeting and have a conversation where the two of you bounce ideas off each other and he helps you solve the problem; then you go home and write it up and get to put his name on it too.
Alternative / complementary hypothesis: he's 77 (born 1945), so ergonomics reasons alone may already justify hiring a typist; plus on average, older people are less used to computers, or more comfortable with handwriting. Though given how incredibly prolific he is, the latter hypothesis seems rather unlikely.
> In practice I’m not sure what I learned from reading them.
I think you can extract some insights:
* Politics-related proposals are unlikely to ever get done, regardless of how much funding they get.
* Projects with specific actionable goals are more likely to achieve progress than projects with vague feel-good mission statements.
* Teams perform better than individuals (probably because the existence of a team is correlated with existing progress having been made, though that's just my conjecture).
Reading about Kirschman's work reminds me of this bit from Ringworld:
> How strange, if new life evolves capable of living on the garbage?
> “That happened on Earth once,” said Louis Wu. “A yeast that could eat polyethylene. It was eating the plastic bags off the supermarket shelves. It’s dead now. We had to give up polyethylene.”
Are you kidding? All plastic being obliterated and becoming an untenable material would be *fantastic*, the only real casualty would be e.g. people with stomas or who actually need the oxygen masks on the airplane (unrelated information request: have those masks ever saved any lives during all of human history? I realize now that this actually seems unlikely): places where we made bad materials decisions.
Most durable goods would become much more labor-intensive to make and thus more expensive. Looking around my room, so many devices have parts where it makes all sense to make them from plastic.
Do we want to have to machine Lego bricks out of metal? Would I have to throw away a perfectly good laptop because the keys rotted? Are plastic ends on USB cables or plastic headphones bad materials decisions? Should we start making piano keys from ivory again? Plastics are a boon for humanity, even if they're overused, doing *completely* without them would suck.
"Do we want to have to machine Lego bricks out of metal? Would I have to throw away a perfectly good laptop because the keys rotted? Are plastic ends on USB cables or plastic headphones bad materials decisions? Should we start making piano keys from ivory again? Plastics are a boon for humanity, even if they're overused, doing *completely* without them would suck."
No, give the children wood blocks to build with. No, make the keys out of hardwood or machine them out of aluminium as well. Yes, make Bakelite ends and wrap the cables in those knitted cloth sleeves. Yes, absolutely, large-scale elephant and rhino breeding programs would only be to the good for everybody, including local industry in Africa (and also, high-end piano keys aren't made of plastic *now*: they're wood with a thin plastic varnish on top, or in some cases resin). No, plastic is like asbestos: some incredibly useful properties, but just too dangerous to use.
I *am* tempted to write a dismissive reply, but let's try to be constructive instead: Price Matters.
The capitalization is because it's arguably the most meaningful metric in existence, from both a humanistic and philosophical point of view. Expressed in work-hours, the main difference between a contemporary citizen and somebody 500 years ago is that it took 2-3 work hours to take a hot bath, while now it takes on the order of minutes. All the functions in a smartphone could have been duplicated 20-30 years ago... with the price of a lifetime of work. Kings had half the convenience and healthcare that somebody with minimum wage gets nowadays. All of this was achieved by price reduction.
I'm not sure what benefit you expect to get from rotting plastics. Some 5% decrease in landfill usage? Feeling good because 1% less turtles are killed by 6 pack rings, while 99% of turtle deaths continue because of other reasons?
But I can tell you what you're going to pay for it - moving civilization back in time a few decades. Plastics are incredibly efficient for doing more with less, probably more than every other invention this side of fire.
You can say that it's worth it, to you. I'm not going to challenge your primary goals - I mean, they're primary for a reason. But I am going to say that what naturally follows from them is that you prefer a world without humans, or possibly a world with humans in a state of a primitive society with no internet and no healthcare. With which I unsurprisingly disagree.
Or maybe you just made a simple scope mistake and never even thought about price, and how it keeps rippling through until it affect everything else. I'd guess this is more likely - your thought process was probably similar to "wood can be used instead of plastic for the first 2-3 applications I can think of, so wood is a good overall replacement for plastic". Which is... such an incredibly common mistake I can't even fault you for it.
"I'm not sure what benefit you expect to get from rotting plastics. Some 5% decrease in landfill usage? Feeling good because 1% less turtles are killed by 6 pack rings, while 99% of turtle deaths continue because of other reasons?"
I'd just love to not have any microplastics in my bloodstream. I *do* like turtles, though.
"But I can tell you what you're going to pay for it - moving civilization back in time a few decades."
I agree this isn't ideal – ideally I would like to move civilization back in time a few centuries. But it's a start anyway. *Technologically* speaking, though, I'd be fine with us just eschewing bad materials, like we did with radium, asbestos and DDT. All of which came with efficiency losses. You soak them because of the negative consequences of continued use. If you don't believe climate change is a total canard, the same concept applies to fossil fuels.
"But I am going to say that what naturally follows from them is that you prefer a world without humans, or possibly a world with humans in a state of a primitive society with no internet and no healthcare. "
Idiot hyperbole. Give me the culture of the year 1600 and whatever technology is possible without filling my blood with xenopolymers and I'm as happy as a clam. (Uh, a clam that hasn't been filtering plastics out of the seawater.)
"I'd guess this is more likely - your thought process was probably similar to 'wood can be used instead of plastic for the first 2-3 applications I can think of, so wood is a good overall replacement for plastic'."
If you're going to speculate contemptuously about my mindset, I'd prefer it if you were at least able to keep a whole two of my comments on the topic in working memory. How do you decode "the only real casualty would be e.g. people with stomas or who actually need the oxygen masks on the airplane" as "wood can be used for the first 2-3 applications I can think of so it's a good overall replacement for plastic"? And I said that *first*.
But... microplastics are something we're not even sure have any health effects. Having plastic packaging on the other hand is helping people not die of food poisoning. I'm pretty sure you'd live a shorter, more painful life in the 1600s.
If I correctly understand what you want, then we can get it only by moving forward. London has a lot better air now than 150 years go. Sweden and Switzerland are cleaner and greener than China, because they can afford it. When China grows enough it will also afford moving to cleaner tech and investing into reforestation, or whatever they decide they want to do with the extra wealth.
And moving forward means using plastics more, not less - at least for now.
What material would you use for aortic and other vascular grafts, surgical (implantable) mesh, contact lenses, or the weight-bearing surfaces of artificial joints?
Hm, yeah, plastic-eating microbes do seem like they could have a large downside. Is this likely to turn out like gain-of-function research where we should have noticed earlier that it's not worth the risk?
Exactly, and those microbes have had about 450 million years since the invention of lignin to learn how to eat it. There's no reason to expect bacteria to be more effective at eating plastic, outside of sci-fi stories.
It would be nice to be able to look at discarded shards of plastic in the dirt and feel the same way about them as I do about wood chips: well, they'll rot away in a year or two.
Less nice if I have to treat all plastics the way I treat wood now: keep it dry, seal it to prevent rot, etc.
Sure, but wood also expands and warps if it absorbs water. Makes it a bit trickier than if rot was the only problem, as my poor cutting board can attest.
I wonder what conditions the particular bacteria being studied need in order to grow? Beetle guts would be very specific.
As someone who has spent plenty of time in academic and political settings, the optimism, flexibility, and open-minded experimentation your grants represent bring me great joy.
"4: Alice Evans’ Research On “The Great Gender Divergence” (?/10)
Dr. Evans has done over four months of research in Morocco, Italy, India, and Turkey. You can find some of her most recent thoughts at her blog here. Her book is still on track to be published from Princeton Press, more details tbd."
I still remain to be convinced that this is going to be of any use to anyone. It's not like this lady academic is starving in a garret with her inspirational work confined to a heap of notebooks under the bed because no-one will publish her, so why does she need a dig-out with an ACX grant?
The Princeton publishing blurb is also ignorant, at least in its effusive enthusiasm that nobody nowhere never did nothing before us did (this ties back in to the query raised on here about "Why learn history?" Well, for a start, so that you don't sound like a Princeton University Press intern):
"Something radical happened over the twentieth century: women entered the workforce and became political leaders across the world. This had never happened before, not in the entirety of human history."
Empress Matilda? Who dat?
Elizabeth I? Never heard of her
Catherine the Great? Is this a cosmetics brand?
Every damn woman who ever milked a cow and churned butter which was then sold at market? Don't be silly, that's not "work" so you never existed anyhow.
Well, there is measure and degree in hereditary, some inherited … rather actively (as in, personally gave the mutinuous orders to the sympathetic military regiments, leading to removal of the previous legally recognised monarch in favour of herself, and usually the previous monarch's death).
I guess Jeanne d'Arc counts as definitely non-hereditary political leader, although given the highly-hereditary context she had to have her muppet king.
(Of course for the entirety of human history one would need to look at cultures not inheriting from Mediterran Antiquity, too, but even without that)
I agree with you on that, as per my response to drosophilist, but the PUP didn't make that distinction.
Granted, exaggeration for the sake of presenting all the geese as swans ('this book will tell you about things never before seen in the entire history of humanity!') is par for the course for any marketing, but for a university press it's poor.
I also wonder about women who managed to become leaders at a local and national scale before the 20th century - it would have been difficult for them to become political leaders when politics as we practice it were not known, but the PUP said nothing about elections - but that would require me to go read some feminist histories.
Wouldn't that require you to eliminate from any relative comparison almost all male leaders up until the 20th century? Not many republics between Rome and 1776.
Heads of state were mostly hereditary, but what about high-level officials such as ministers appointed by the ruler, or city leaders elected by the local patricians? Those were almost always men, as far as I can tell.
Conversely, this is the one grant I most want to succeed in some tangibly useful way, because it's the most dubious-seeming one. That'd be the biggest possible update to my heuristics regarding social science research and funding distribution.
Granted, there will always be the opportunity-cost question of "are we sure the money couldn't have come from elsewhere?" (while granting that Scott's grant decisions are entirely up to him, and at the least I doubt she's taking the money for granted) . It's a bit different giving someone a hand up if they're already partway up the ladder...
I looked at Evans' blog & CV a little bit and I have a guess. There are parts of the effective altruism ecosystem that are working on improving outcomes of the type of work NGOs and governments have been doing - development economics for example. Gender has a multivalent relevance (hee hee - I'll imitate Princeton for just a moment) so anyway it will come up, it will play roles in EA development projects, and when EA sets gender victory goals, EA will want to have research to back up what the best form of gender equality is. I'm guessing here but I think Dr. Evans has taken this on.
Her PhD is in geography, BA philosophy. Here is a person who probably didn't have to digest all the waves of feminist critical theory before being allowed to do anthropological research. This is the sociological intellect escaping the stranglehold of the 20th century. Maybe?
The rest of the book blurb - "The Great Gender Divergence will explain the causes of Europe’s “precocious equality,” how East Asia and Latin America caught up, why gender equity in the Middle East, South Asia, and Sub-Saharan Africa has lagged behind, and why South-East Asia was always ahead. Evans offers a comparative history of how societies come to support gender equality and why this varies around the world, telling a story about geography, economic growth, strong states, and militant activism." That's a strong dive there at the end. Militant activism. That blurb is basically pre-pandemic though.
It will be interesting to see what answers come up. "What Gender Equality should we construct? Global survey of options" might be a better title. We will find out shortly, I guess, how much liberation is good for us. I mean. I have not gone back to check how much money this grant was for.
I think Scott supports gender equality in general and so if this line of investigation went somewhere - if there were some recipe that could be lifted out from some social arrangements - this would be knowledge he would be glad to have supported obtaining. Since "make it fifty-fifty!" is weird.
I ordered a book from a used book site recently and it turned out to be by a different writer with the same name - it was very cheap and followed a list of books I needed to get, so I didn't really check & just added it for curiousity's sake, and it turns out to be a treatise against sociobiology and for historicity, that human societies are primarily shaped by their collective choices rather than biological constraints. If this guy's making a solid point, whatever Evans discovers will be useful for community development only insofar as the target society adopts it.
One of her blog posts from the summer is about misunderstanding patriarchy, though. I haven't read it yet, but here's someone doing self-evaluation in addition to geography-based gender research. It checks the EA boxes.
Her website says she wants to supervise PhDs on (list), including: Why have female employment, representation and activism boomed in traditionally Catholic Ireland and Spain but not Italy?
---I don't know if you were actively seeking your next PhD, but hey.
"when EA sets gender victory goals, EA will want to have research to back up what the best form of gender equality is"
May I be taken from this world before that happens. If EA is going to meddle in "not enough women in Whatever", then at best it'll work out like their politics meddling (spent a ton of money, candidate was beaten by the obvious shoo-in) and at worst it'll make things worse (hm, we managed to get 60% more women in Whatever but somehow that hasn't resulted in the earthly paradise, and minority communities are still struggling with poverty, crime, illiteracy and family breakdown. But twelve of the women we got appointed to cushy sinecures have bought million-dollar second and third houses, so that's nice! https://nymag.com/intelligencer/2022/04/black-lives-matter-6-million-dollar-house.html)
I'm going to say this: what promotes gender equity (and what the hell does that phrase even mean? women now in the workforce doing what were traditionally men's jobs? women in the home having their work recognised? 50/50 men and women in every field of employment?) is, if I believe all the reports and studies that have been pouring out about this for years, is
(1) Education
(2) Contraception
(3) Employment
And how do you get those things? You need your society to be:
(A) Secularised
(B) Industrialised
There you go, and I didn't even need $20,000 of a grant to tell you that. I'll wait and see what Dr. Evans' book is like, but I'd be hugely surprised if she decides "Um, gender equity happens when all women are the Angel In The House".
Mostly my objections are that this is an established academic with an established career who has had no problems getting published before, not a new person starting out or a project with no other sources of funding. But it's Scott's money and if he wants to give it away to whomever he likes, he can. He can do a KLF and set bricks of it on fire, too, if he likes.
(I do think the mobile slaughterhouses one is a good idea and deserving of a grant).
Yes, I have to say I was less than impressed with the whole "I rejected the idea that religion might have an effect on social culture and practices" article. It's good that she can acknowledge her blind spots, but if she's playing catch-up with what has been a commonplace in the field, then how much original research is she doing and how much is her book just going to be another coffee-table production, to be displayed but not read?
If you're curious why activism is included, it's relevant to her theory of communist vs non-communist states. Communist states crushed traditional patriarchal cultures and were officially gender egalitarian, but she finds that today ex-communist states are still less egalitarian than capitalist ones. Her view is that communism crushed independent activism, so there wasn't really a women's movement in those countries.
Oh that’s interesting. I was wondering if the communism crushes patriarchal culture -> gender egalitarian dynamic would persist.
I was also wondering if a culture would be more egalitarian if it had experienced x centuries of colonization or outside rule during which time men & women were roughly equal status as peasants, and then if the culture regained self-rule it might be more egalitarian.
I think the presence of activism is an interesting choice of metric as well. I can’t decide if it indicates less oppression (otherwise it would be crushed) or more oppression (conflict can’t be solved adequately without protest).
My theory would be the really simple one that general wide-spread middle-class wealth seems to be at the heart of approximately all social progress, and Communism was rather bad at wealth generation. People can't invest energy in politics if they're too busy trying to survive. (And oil sheikhs and other nobility both can afford to be ridiculously wasteful because they're *too* rich and have a very strong incentive to avoid rocking the boat, just in case. It's the middle class that pushes for change)
Deiseach, I love your witty comments, but I disagree with you here. There is a world of difference between "a few high-born women, like Hatshepsut and Elizabeth I, held power" and "women are routinely elected members of Congress/Parliament and heads of state, in addition to having positions of power and influence in industry and science."
Likewise, yes of course milking cows and churning butter is hard work, but the modern world is unique in having women work *outside the home, for money, in large numbers*.
I agree that hereditary and non-hereditary leaders are different.
What PUP means is that "in the 20th century, women went into the industrial workforce in large numbers, and ran for political office, for the first time".
That is not at all the same as saying that "women had never before been part of the workforce or leaders in the entirety of human history".
Women working outside the home, for money? Happened in the 19th century down the mines, for one instance:
"A significant minority of wives, however, worked from sheer necessity. In the 1840s, in Lancashire alone, a survey of 412 cotton factories found that just over half of the 116,300 workers were female. Around 10,700 of them were married women. These women were not the sole breadwinners: over 9,200 of the husbands in this survey had regular work. Only 821 husbands were unemployed, undermining some critics’ claim that women worked, while their spouses boozed in the nearest pub."
That's before we even begin to touch women's work outside of Europe:
"Who can find a virtuous woman? for her price is far above rubies.
She is like the merchants' ships; she bringeth her food from afar.
She considereth a field, and buyeth it: with the fruit of her hands she planteth a vineyard
She maketh fine linen, and selleth it; and delivereth girdles unto the merchant."
But of course, this is a publisher's blurb, which means it is dumb as mud.
Is it a difference with importance? Is a country better governed for having 50% female MPs or CEOs versus 10% or 100% or 0%? Are women as a class more or less happy? Are men as a class more or less happy? Are families more functional, is the market more efficient, is crime higher or lower, are more or fewer people addicted to heroin?
I wouldn't pretend to know the answers to any of these questions, but it seems to me that answering them definitively is way more important than simply viewing a percentage of women (or minorities, or left- or right-handed people, or people with/without tattoos) in Profession X that is close to their proportion in the population as a big win to be celebrated all by itself. There's a certain insect mindlessness to the drive to have every job or role in society have a representation of sex and skin color and other such surface ephemera that matches that of the entire population that is discouraging in its failure to ascertain whether there is any actual measureable value in the result.
I mean, an AI maximizing paperclips comes to mind.
Alice Evans is not dumb and is entirely aware of women working on farms, her research makes a big distinction between that and going away from home to make money.
If that's your definition of "workforce", the vast majority of men don't qualify either prior to industrialisation (and post industrialisation many women did - industries like weaving stayed very female dominated while being among the very first to industrialise).
The shift, very broadly, is about the level of *recognition* and *pay* women get for their work, and of a few specific occupations removing legal barriers to entry, but for most of human history women have worked alongside men in broadly similar occupations. (Eg. many (though obviously not all) medieval guilds let widows inherit their late husband's business - they'd learned the trade over the years of their marriage - even though they didn't let women train in the field independently)
I did not say Dr. Evans was dumb, I said PUP's blurb was dumb. I would certainly hope Dr. Evans' book was finer-grained, e.g. "In the 20th century, globally women entered the industrial workforce and politics in a way unmatched before".
"Working on farms" and "going away from home to make money" were not mutually exclusive, it seems you have not heard of hiring fairs:
Hmm, that's a little discouraging. That's an activity that very few people of either sex did up until late Industrial Revolution. Prior to that the majority of any population worked on a farm, kept animals, or did small-scale local artisanry -- smiths, weavers, carpenters, et cetera. Money in the sense of gold and silver wouldn't even change hands that often among this class, a great deal would happen as a result of a complex web of favor exchanges and barter.
Historians have historically focused on the easier to study people with what they saw as more complex and interesting lives, namely the nobility and extremely wealthy, who might well have had wide choices to make in professions, and for whom money income was a major fact of life. But I thought it was recognized some time ago that this is an unfortunate distortion of history, as it studies only a tiny elite fraction of humanity, and gives little insight into what everyday life was like for most.
It's discouraging that (from what you say here) Dr. Evans would appear to continuesthis trend. It would be much more interesting to look into the roles of men and women in, say, the households of peasant farmers and small artisans over the past millenium or so. That would tell us a lot more about what life was actually like for the typical schmo (male or female).
The most interesting bit I got out of these reports was "which grantees either didn't understand Scott's instructions that 5/10 was the expected amount of progress, or inflated their scores to match more standard out-of-10 rating systems".
Judging by comparison with the textual descriptions, I'd estimate at least a third of the grantees did this. I put much more probability on grantees' systematic incentives to inflate than on Scott's theory that things went systemically better than expected.
That being said, this whole thing remains super cool and I'm not nearly as surprised as Scott that the ACX exposure counts for a lot on its own!
In fairness, if you received a grant for getting a job, and then you get the job thanks to the grant, is that 5/10 because that was the expected outcome? And if so, what would the 10/10 score represent? You got a better job than the one you wanted, *and also* a porn star mistress who was attracted to you specifically because of the size of your grant? "Hoped for job, got job, 10/10 success" seems a lot more intuitive.
I feel like the majority of these ratings are actually using a 4-to-10 scale where "as expected" is around a 7 or 8. I think the only rating below 4 was the guy who hasn't started yet, and the updates with 7s generally don't mention anything that seems like obvious good fortune.
#30 says their project has been de-prioritized and they hope to get back to it eventually, but gave a rating of 6/10.
#22 says their original plan won't work, their timeline has ballooned, and they're worried their work might soon become obsolete, but gave a rating of 5/10.
(Apologies for picking on you; you're just the most clear-cut examples of an unsurprising pattern.)
Yeah, I don't think it's feasible to tell people "a 5 indicates that you're doing as well as expected"; almost no one will actually rate themselves that way. To actually be doing as well as you expected, on a major project, is (given the natural human tendency towards optimism) a great big success! No one who feels like they're succeeding is going to give themselves a 5 out of 10: a 5 out of 10 is at best average and at worst an F.
Some of the reports make me wonder whether _two_ grades would be more informative: basically «help us do X to achieve Y» has «we did X as planned and with good quality» and «X helped with Y» parts. I.e. it looks like 6. did things in their control fine, but things out of control played out not so well this time.
And maybe in addition to the median 5/10, it's good to say something about spread: is «needed to win a fair coin toss, won/lost» 6/10 and 4/10?
I didn't ask him to and I imagine that would just anger the people who hate my politics, so I'm glad he didn't. In general everyone should feel free to deny ACX funding if it helps them in any way.
Well, in some cases academic ethics (or whatever the correct term may be) dictates that a researcher makes all their funding public, whether the grant giver insists on it or not?
(I assume in medicine you cannot just omit the fact that your cancer research was sponsored by Philips Morris, even if Philips Morris does not insist on being acknowledged?)
I am not claiming that this is the case here (the preprint is in Biology, not medicine, and does not include a statement of financial interest or whatever...)
Regarding Promote Economically Literate Climate Policy In US States comment "Overall they report frustration, as many of the legislators they worked with have been voted out or term-limited": Possible silver lining. I recall a Cato policy analysis from many years ago that reviewed congressional representative's voting records relative to time in office. The longer they are there the more they tend toward government run solutions and oppose private alternatives. If so, and assuming what they found at the federal level applies as well at the state level, the fact people have been voted out (and thus replaced by freshmen legislators) may in the end be helpful to their plans.
Did they find a difference between second-term legislators who eventually serve ten terms and second-term legislators who get only end up having three terms, or a difference between second-term legislators who eventually serve ten terms and ninth-term legislators who eventually serve ten terms, or a difference between current second-term legislators and current ninth-term legislators? The description sounds like the latter, but I would want to know about the difference between the first two, which identifies how much is selection and how much is causation.
We did not get one of these grants, but we were listed on the applicants thread, and it helped us get some help with the project. Since then, we've partnered with a major law firm and are investigating PPP loan fraud. It's been a lot of work, but it's nice to have apply our analytics skills to something rewarding instead of helping a giant business optimize click-through rates or something. We can't talk about any cases until they are brought out from under seal but we're extremely happy with our traction and we've begun making inroads helping healthcare payers such as health insurance companies, unions, and large employers, prevent fraud.
Thank you very much for running this program: you wound up introducing us to our first angel investor. He really helped us understand how this could be more than a cute idea to punish some bad guys, and that has paved the way for the other angels which are following. We really appreciate that programs like this exist, and will report back at the next ACX grant update.
I have no other comment other than to say that I enjoyed reading this greatly, more so than the original grant announcement, and found this greatly energizing
I don't know if that one worked, or if it was "take the money and run". If it was a dodge, then one out of forty grants isn't a bad outcome, I suppose, since you do have to anticipate that at least some applicants are not wholly in good faith.
If SD does turn up with a full slate of reports and data, I will apologise to them 😀
Sorry - I just completely missed Scott's email about this. I emailed him my progress as soon as I saw this post and it's updated now. I can't share specifics publicly at my advisor's request but the thesis will be available online after it's complete.
Why is this only 8/10? It sounds like they succeeded in their goal; what would have resulted in them rating it 10/10? Were they hoping for a more prestigious school?
Since 5/10 is "things going as expected", 8/10 looks like everything went really well (perhaps faster than expected, or actually a more prestigious school than thought). I understand 10/10 on this scale to be extreme overachievement.
> The grant was to help make it financially easier for him to go on a long round of interviews [Anonymous] successfully got a job offer from a top school[...]
On the 6.5/10 average: might this not be simple publication bias? That is, people being less likely to send you progress reports if things aren't going as well?
The ratings are an interesting psych experiment in themselves. I read the instructions in the post and thought "yeah, that's not going to work as intended," and I think that was borne out by the ratings given.
If you've met your goals, it's psychologically extremely difficult to give a rating of 5/10 - I mean, 50% isn't even a passing grade! Hence the reports that say things like "we had some unexpected setbacks but achieved most of what we intended, 7/10" when that should probably be 4/10.
I think it's (mostly) not dishonest inflation, but people either finding it too hard to overcome their cognitive bias or simply misreading the instructions (like the comment from Isaac King).
Scott, I think if you do this again you should say "Rate your progress on a scale of 1 to 5 relative to what you expected. If things went even better than you expected, you can give out-of-range answers like 6/5 or 7/5." Logically equivalent, but it will make all the difference psychologically: I think you'd get an average rating of about 4.
What's more, probably some grantees used the scale as you intended, some used it the way Isaac read it (10 is fully meeting expectations), and some did something in between, where they felt torn between the instructions and the natural way to interpret a 1-10 scale and plumped for a score in between the two - so it's not meaningful to compare scores between grantees or do any comparative statistics like "teams scored higher than individuals".
For the ones that say "They have asked me not to discuss their progress yet." my hope would be that once progress updates are allowed they come with a *really* good explanation of why progress updates were withheld. I am personally a big fan of open science, and keeping information private feels like it means you are striving to achieve something other than "betterment of society/humanity" i.e., not particularly altruistic.
If these people can provide great rationale for keeping the information private (such as releasing of a status update would cause some kind of bias in their data collection) then we can all walk away thinking it was a wise decision.
If, on the other hand, they just wanted to keep updates private because they were waiting for a patent application or to secure their position as a "leader" or to get their work published in some reputable journal then I think future grants should have a stipulation that grant recipients *MUST* share status updates and not withhold details.
A lot of the withheld ones were political and thus their success evaluation will be radically different in a week when the midterm results come in. I do agree that Scott should make a supplemental post sharing all the updates that were waiting till after the mid-term elections, within a week or two of said elections being over.
I got a grant by being forwarded to the LTFF for some concrete ideas on expanding on a text about forecasting[1]. I have done 4/6 of the things I promised I would do, the two missing ones being adding a literature review and extending the analysis to continuous questions. I give myself a 6/10.
To speak a bit more about how contingent Manifold's existence was on ACX Grants: I think there's like an 50% chance that me/James/Stephen would not have chosen prediction markets (among our other crypto ideas) if we weren't submitting a proposal ACX grants specifically; crypto was on our minds as you can see from our original application: https://docs.google.com/document/d/12DufTzcIaJxSQOxMYIHAwt7tGa5PAkIyhZOQ2JrmCfg/edit
I also think there's a 60% chance we wouldn't have continued with a prediction markets project if we hadn't won an ACX Grant and gotten the subsequent publicity through that announcement and Mantic Mondays. ACX readers formed the bulk of Manifold's initial adopters, and still represent our most vocal and engaged users.
We owe a lot to ACX and Scott in particular, and hope to pay back that generosity with useful predictions of all kinds!
They're planning to spend 2/3 of the tax money on tax credits for poor/mid incomes and eliminating sales tax on grocery store food. Then an additional $93M goes to rural economic development, airports and highways for some reason?
Only about 1/6 of it, $100M, goes to actually cleaning the air, which they say they'll spend on "cleaner school buses, electrifying lawn and garden equipment". So like a fleet of shiny new electric school buses and a mail-in rebate if you buy an electric lawnmower? And nothing about toxic dust from the Great Salt Lake drying up due to the drought? If you're going to spend $50M on rural economic development, shouldn't that at least be aimed at water conservation?
> The average project rated their progress at 6.5/10, where 5 was “meets expectations” (this average includes some private projects I didn’t fully list here). So either people systematically underestimated their chances when applying, or they’re systematically inflating their progress now to look good, or things systematically went better than expected.
No, none of those things. People are absolutely awful at assessing their progress. They think they're doing better than they are - they're not inflating their progress to look good to you, but they are probably inflating the progress to *themselves*, and **certainly** systematically ridiculously overoptimistic about how their progress compares to their problem. Which is normal. Wikipedia's planning fallacy example has average real-case estimate 33.9 days, best-case 27.4 days, worst-case 48.6 days; reality averaged 55.5 days, 70% of students taking longer than predicted. That's with a fairly narrow scope where they think they perfectly understand the problem, which is applicable for about zero of these grants.
"Do 90% of the work in the first nine days, and then the other 90% of the work on the tenth day." Halve the ratings and you'd have answers closer to the truth. Though some of them are still way too optimistic by that measure. Anyone who cites getting more money as progress, or is looking for more money now, halve their score again. Or just round down to 0, that's probably more accurate.
Re. this: "Trevor Klee...realized this would be too expensive to do in humans in the current funding environment, and has pivoted to getting his medication approved for a feline autoimmune disease as both a proof-of-concept and as a cheaper, faster way to start making revenue. ... He still anticipates eventually moving back to humans."
This is a very common trap. I don't know of a single case where it worked out for the humans. A company wants to develop a drug for humans, but FDA approval is too expensive, so they get it approved in dogs, horses, or cats, thinking they'll make enough money to push it through approval for humans. They never do; it's hard enough to keep a new pet-drug business above water. Then their patent expires, and the drug will never, ever be approved for humans, because nobody will pay for FDA approval of an unpatentable drug.
Until just a few years ago, this was almost acceptable, because you could go to Agway or a foreign website and buy the drug for your dog and take it yourself. But governments and industry have taken EXTREMELY aggressive steps in the past few years to prevent this.
An example is Adequan. It temporarily cures arthritis in humans. It cures arthritis in me! But it will never, ever be legal for humans in the US or Europe, because it was developed in the 1980s and the patent expired long ago. But it is approved for dogs, horses, and pigs in the US.
Until very recently, I could easily buy it over-the-counter in any country but America, or order it thru ebay or from any overseas veterinary website. But organizations have just taken many steps to prevent any humans from taking Adequan, including:
- turning it into a prescription-only drug in all Western nations
- (I presume) prosecuting online pharmacies that don't comply (as they all suddenly went out of business or stopped selling it)
- purging overseas pharmacies that sell it over-the-counter from search engine results
- purging pharmacies that sell it without prescription from TrustPilot listings
- creating an organization to rate the "safety" of online pharmacies, which actually rates their compliance with import regulations
- investigating ebay sales of Adequan to verify possession of a veterinary prescription
- banning ebay users who sell it to people who have no such prescription
- delisting Adequan sales on ebay entirely
- in my state, creating laws for dog licensing that forbid veterinarians from treating unlicensed dogs, and forbid licensing dogs without proof of legal ownership of the dog, including an investigation process to verify possession of alleged stray dogs, so that you can't even get Adequan by borrowing your neighbor's arthritic dog, licensing it, and taking it to the vet to get Adequan (I tried)
I can't think of any reason anyone might have to prevent people from taking Adequan, so I think it's just part of a general tightening of access to veterinary drugs, which has been going on for about the past 5 years. The drugs for which doing this makes some sense -- antibiotics -- were left for last, but the US govt. has finally gotten around to them, and IIRC sometime next year it will become impossible to buy any veterinary antibiotics without a prescription.
Ivermectin was first approved for use in cattle, horses and sheep. So it (getting a medicine approved for vetinary use before getting approval for its use in humans) has been done.
It's now on the WHO list of essential medicines, IIRC, so pretty successful. I have no idea about the economics of the case, though.
I would be very interested to hear why this strategy worked for Ivermectin (and possibly others?) but not for other things. Maybe there is something useful we can glean from its story to try to reproduce it with other therapies?
I generally agree with Phil's concern about animal-first drugs suffering the same problem as repurposed drugs and would love to see a solution to either problem!
I would be interested too. I don't work in the industry; I just recall TV commercials for Ivermectin as an anthelminthic for sheep in NZ in the early 1980s.* ("Kills roundworm, lungworm, and pinworm too!") Wikipedia corroborates my memory.
I only recently learned that it was approved for use in humans, when the controversy about it cropped up two(?) years ago. I was amazed and amused.
* TV commercials in NZ are, or at least were then, bizarre to residents of other countries. A good fraction of ads was devoted to agricultural products and services - quad bikes, fertilizer additives, spraying equipment, lime spreading, etc.
The Wikipedia page seems to have some clues. It was the "bestselling veterinary medicine in the world", indicating that Merck did not have any problem in this case with what Phil called "keep[ing] a new pet-drug business above water." Critically, it was adopted by the livestock industry, and not just pet owners. Then, it was approved for human use in collaboration with the WHO, and Merck offered to "donate all ivermectin needed to eradicate river blindness ... ". So it seems that the motivation for human licensing was at least partly for publicity, and perhaps even from a desire to do good in the world. The inventors went on to win (half of?) the Nobel Prize in 2015 for Ivermectin.
So, it seems the formula is a) have a miracle drug b) with important agribuisiness application c) which also treats crippling illness in photogenic children in the developing world. Perhaps the process for drug approval in the 1980s was also a bit less expensive?
There's also two incentives for getting a drug approved for a rare disease. The Orphan Drug Designation gives a company some special tax breaks for the research costs plus added years on its patent.
There was also another rare disease incentive that provided a drug company that got one approved a "pass" for accelerated review on some future drug. This 'pass' could be used by the drug company or sold to another one. It's only been done a few times I believe since the FDA is actually pretty good about doing reviews quickly and granting accelerated status when the drug is in a critical area.
I think the part where the human use is more charity than profit motivated makes a huge difference, because it means being out of patent or nearly so isn't a drawback.
I didn't know that about Ivermectin, but I see it was developed in the 1970s. Here's a report on a study by Tufts on how drug development costs in the US rose from an (average? median? geometric average? doesn't say) of (if I'm reading the bar graph right) about $170 million in the 1970s, to nearly $2,600 million in 2000-2010: https://cen.acs.org/articles/92/web/2014/11/Tufts-Study-Finds-Big-Rise.html
They don't mention any costs in drug development other than doing safety studies, opportunity costs, and the costs of drugs that failed (which are factored into those prices); so apparently the *research* costs are negligible. AS USUAL, the journalist didn't say who wrote the article, or where or when they published it, or what its title was; so I can give no citation.
Anyway, Ivermectin would have had a relatively low cost of approval compared to drugs today.
"An example is Adequan. It temporarily cures arthritis in humans. It cures arthritis in me! But it will never, ever be legal for humans in the US or Europe, because it was developed in the 1980s and the patent expired long ago. But it is approved for dogs, horses, and pigs in the US."
Unanswered question: Given the massive amounts insurance spends for arthritis treatments, why wouldn't a coalition of payers simply fund the studies? If they work, then you have a nice, inexpensive drug that works in humans and it will be produced as a generic drug?
You have to continue taking Adequan for the rest of your life, because it doesn't really regrow cartilage. Apparently it lays down something that acts like cartilage, but quickly wears away. (It's not a drug in the usual sense; it doesn't interact chemically at all AFAIK.) It's a lot cheaper for insurance companies to tell people with arthritis to take Naproxen, which is what doctors usually do.
Besides just bias and inflation, one might have different scales (that is, completion could be 10 for someone and just a 7 for others, an expected benefit could one's 8 or another's 11).
Also note - I'm not sure if we can even avg these scores out in the first place. As in, if you get a 3 and 7, does that equal 2 5s? The 3 might be much worse compared to a 5 than the 7 is better.
Ignoring all that lets have some fun making some other markers for the scale.
Limited scale -
1 - Failure
3 - Delay
5 - Expected Progress
7 - Full Completion without issue
9 - Unexpected great success
From that, I redid the scores from my not that optimistic guesses based only on the descriptions.
1-10 6?8?336756
11+ 8?55623735
21+ ?2867?645?
31+ 6759?5?676
Avg non ? ~5.5
Note for 30 I changed to a ? cause it didn't really fit in the scale. Also on a few I graded actually a bit higher than what they gave themselves.
The avg for the visible ones above was 6.67 so if we assume the invisible ones were roughly similarly biased, then it's still just about a point difference. Doesn't mean there wasn't a lot of bias - the most common dif was a 2, but the next highest was 0, so it avg out.
Full (not to scale) scale
0 - Likely scammed/tricked. Not necessarily by the grantee, mind you.
1 - Canceled/ full failure and can't continue/ hit obstacle that would warrant a cancellation at any point.
2 - Significant delay or backtracking. Unknown or difficult to guess how major it is. (note that this could be worse than 1 in many cases. At least 1 might be quick)
3 - Stalled or "small" failures. Generally still able to see an end, but might have already failed to meet some goals/objectives.
4 - Minor obstacles that have caused some delay/deviation from estimates, but nothing that could count as failures.
5 - Free Space
6 - Smooth progress.
7 - Full success
8 - Went well enough that it expanded scope/target of initial project/task. (Note - might want to do 8->3 or something if expanding it caused more issues later on...)
My main takeaway is people suck at using numeric scales.
10/10 for achieving the first step out of a many step failure prone process? 8/10 for doing precisely and exactly what they said they'd do? You marked 5/10 as "as expected", what's up with all the insane grade inflation? I assume it's because nobody think of 5/10 as being "as expected", people don't want to think of themselves as failures (which 5/10 feels like), so they mark themselves up ridiculously?
I also get particular pleasure from these!
> 22: Creating Intelligent Tutoring Systems (5/10)
... They are concerned about whether large language models will obsolete their product, and interested in talking to anyone who thinks they can predict LLM’s near-future performance - if this is you, contact James Koppel.
I am a bumpkin myself, but I know a couple smart people who might be interested. Is there some specific writeup or URL I can send them to get them up to speed on the main project -- "Creating Intelligent Tutoring Systems" ? I don't want to just send them to Koppel's main site, that would be too vague...
Tutoring in what? LLM are nowhere near tutoring on almost any subject.
My guess is that they panicked on June/July when Google disclosed Minerva.
Although it may look very impressive, I think that this is one of the cases where they reached the LLMs show their limits.
-the dataset for Math/Physics problem is too small, so they should have been able to get almost all of it
-the marginal returns are heavy
-it's one of those cases where you need more automated reasoning and a top down approach to problem (it's important to stress out a few things pedagogically and the LLM's can be quite bad at that).
Moreover I am a bit skeptical about using LLMs to do this stuff for a number of things. If you want to sell the product:
-you want to have something that allows the student to cheat at test;
-you want to have something that helps the teacher (sort of like a super-book).
I take Caplan side regarding education. Unless you have a very excellent product (and not something merely promising) the Institutions and your possible clients are going to dismiss you.
At the start of the year, an AI expert interested in tutoring told me "Anything you do will be wiped out by a few years of gains in LLMs, so the best thing you can do is collect training data." A few weeks ago, I got some insider info that makes me inclined to think he's right.
As for the skepticism about LLM's capabilities: you're making a lot of assumptions about what the tutoring would look like. Key term to search for is "interaction plateau hypothesis." Why2-ATLAS does work exactly like you describe except fancier, and it's only been shown to produce marginally better learning outcomes than much dumber systems.
> it's important to stress out a few things pedagogically
We're actually pretty good at this just with our canned feedback database. :)
BTW, I am an automated reasoning guy (never published in IJCAR or CADE, but am pretty well-versed in that literature), so it is quite painful to admit the supremacy of neural over deductive techniques.
> the Institutions and your possible clients are going to dismiss you.
The main planned client is my own training company Mirdin, so we're covered there. Institutions were never on the table as possible customers. Some of my former coauthors and other academic colleagues will be free users of the open source version, furthering the altruistic aims of the grant. If we do productize it, likely other customers will be other small upskilling businesses like my own, although a few SAT tutors have also reached out.
IDK I have just a hard time seeing how it is helpful. What "work" is the LLM doing? It cannot really assess how responsive or engaged the tutee is, nor motivate them if not.
I don't know I find good tutor is a mix of feeding people excuses and rationalizations and guilty trips and good/bad examples, and blaming others, and blaming their past selves, and much more emotional manipulation all predicated on their moment-to-moment emotional states.
The material isn't the issue, getting people to fire on all cylinders is. How is a LLM helping with that?
I'll be waiting for the insider info to get public then! As of now I remain a skeptical of large improvements of LLMs in this regards, but I'll be happy to be proven wrong. May be not so happy since I am currently working for a competitor, so I'll need to be looking for another job ;)
Well I don’t even understand how it is supposed to be helpful. For most professional an academic tutoring I have done the issue isn’t having an algorithm. There already is an algorithm, RTFM. A lot of the professional tutoring there is like an 800 page manual and oodles of fact sheets and guides and training materials that are totally sufficient. What the tutees lack is the emotional intelligence to actually go through that.
So they need a Sherpa to hold them accountable and cajole, encourage, condense, harass them into learning and applying themselves. It is rare that they are actually too stupid to get it, they just need the right motivators and someone to hold their hand and tell them it will be ok if you just try hard and listen to me.
How is the LLM doing that?
> am a bumpkin myself, but I know a couple smart people who might be interested. Is there some specific writeup or URL I can send them to get them up to speed on the main project
Send me your contact info and I'll share the original grant proposal.
I'm very amused by '40: Typist For Saharon Shelah (7/10)'. Guess he just keeps getting nerdsniped by new ideas and typing things up in LaTeX properly is *such* a pain...
I would say «writing up the complete article around the core of the proof» is such a pain, and a typist can at least remove one part of the pain. Not all of it, but I guess it's about pushing things below the pain threshold!
Plus if you have someone periodically asking you "so, what else do you have for me to type up?", doing something that would result in an affirmative answer rather than blushing with "nope, I am squandering the grant money" is an incentive in itself.
Shelah is actually one of the most prolific mathematicians of all time. According to the database of Erdös numbers, as of 2004, "There are four authors with more than 700 papers: Paul Erdös with 1416 (he actually wrote more papers than that, but these are just the ones covered by Math Reviews), Drumi Bainov with 823, SAHARON SHELAH with 760, and Leonard Carlitz with 730."
https://www.oakland.edu/enp/trivia/
At this point, Shelah is up to at least 1140 in that same database:
https://mathscinet.ams.org/mathscinet/MRAuthorID/160185
A friend of mine who collaborated with Shelah described the process, which sounds a lot like what I've heard for collaborating with Erdös - you have an idea you're working on in one of the fields he works on; you arrange a meeting and have a conversation where the two of you bounce ideas off each other and he helps you solve the problem; then you go home and write it up and get to put his name on it too.
Alternative / complementary hypothesis: he's 77 (born 1945), so ergonomics reasons alone may already justify hiring a typist; plus on average, older people are less used to computers, or more comfortable with handwriting. Though given how incredibly prolific he is, the latter hypothesis seems rather unlikely.
> In practice I’m not sure what I learned from reading them.
I think you can extract some insights:
* Politics-related proposals are unlikely to ever get done, regardless of how much funding they get.
* Projects with specific actionable goals are more likely to achieve progress than projects with vague feel-good mission statements.
* Teams perform better than individuals (probably because the existence of a team is correlated with existing progress having been made, though that's just my conjecture).
I'm curious which ones you're using as evidence here (email me at scott@slatestarcodex.com if you don't want to say so openly).
Really cool. Thank you for posting updates!
Reading about Kirschman's work reminds me of this bit from Ringworld:
> How strange, if new life evolves capable of living on the garbage?
> “That happened on Earth once,” said Louis Wu. “A yeast that could eat polyethylene. It was eating the plastic bags off the supermarket shelves. It’s dead now. We had to give up polyethylene.”
Let's hope it stays in vitro :P
"Let's hope it stays in vitro :P"
Are you kidding? All plastic being obliterated and becoming an untenable material would be *fantastic*, the only real casualty would be e.g. people with stomas or who actually need the oxygen masks on the airplane (unrelated information request: have those masks ever saved any lives during all of human history? I realize now that this actually seems unlikely): places where we made bad materials decisions.
Most durable goods would become much more labor-intensive to make and thus more expensive. Looking around my room, so many devices have parts where it makes all sense to make them from plastic.
Do we want to have to machine Lego bricks out of metal? Would I have to throw away a perfectly good laptop because the keys rotted? Are plastic ends on USB cables or plastic headphones bad materials decisions? Should we start making piano keys from ivory again? Plastics are a boon for humanity, even if they're overused, doing *completely* without them would suck.
"Do we want to have to machine Lego bricks out of metal? Would I have to throw away a perfectly good laptop because the keys rotted? Are plastic ends on USB cables or plastic headphones bad materials decisions? Should we start making piano keys from ivory again? Plastics are a boon for humanity, even if they're overused, doing *completely* without them would suck."
No, give the children wood blocks to build with. No, make the keys out of hardwood or machine them out of aluminium as well. Yes, make Bakelite ends and wrap the cables in those knitted cloth sleeves. Yes, absolutely, large-scale elephant and rhino breeding programs would only be to the good for everybody, including local industry in Africa (and also, high-end piano keys aren't made of plastic *now*: they're wood with a thin plastic varnish on top, or in some cases resin). No, plastic is like asbestos: some incredibly useful properties, but just too dangerous to use.
This is silly and would be crazy expensive.
I *am* tempted to write a dismissive reply, but let's try to be constructive instead: Price Matters.
The capitalization is because it's arguably the most meaningful metric in existence, from both a humanistic and philosophical point of view. Expressed in work-hours, the main difference between a contemporary citizen and somebody 500 years ago is that it took 2-3 work hours to take a hot bath, while now it takes on the order of minutes. All the functions in a smartphone could have been duplicated 20-30 years ago... with the price of a lifetime of work. Kings had half the convenience and healthcare that somebody with minimum wage gets nowadays. All of this was achieved by price reduction.
I'm not sure what benefit you expect to get from rotting plastics. Some 5% decrease in landfill usage? Feeling good because 1% less turtles are killed by 6 pack rings, while 99% of turtle deaths continue because of other reasons?
But I can tell you what you're going to pay for it - moving civilization back in time a few decades. Plastics are incredibly efficient for doing more with less, probably more than every other invention this side of fire.
You can say that it's worth it, to you. I'm not going to challenge your primary goals - I mean, they're primary for a reason. But I am going to say that what naturally follows from them is that you prefer a world without humans, or possibly a world with humans in a state of a primitive society with no internet and no healthcare. With which I unsurprisingly disagree.
Or maybe you just made a simple scope mistake and never even thought about price, and how it keeps rippling through until it affect everything else. I'd guess this is more likely - your thought process was probably similar to "wood can be used instead of plastic for the first 2-3 applications I can think of, so wood is a good overall replacement for plastic". Which is... such an incredibly common mistake I can't even fault you for it.
"I'm not sure what benefit you expect to get from rotting plastics. Some 5% decrease in landfill usage? Feeling good because 1% less turtles are killed by 6 pack rings, while 99% of turtle deaths continue because of other reasons?"
I'd just love to not have any microplastics in my bloodstream. I *do* like turtles, though.
"But I can tell you what you're going to pay for it - moving civilization back in time a few decades."
I agree this isn't ideal – ideally I would like to move civilization back in time a few centuries. But it's a start anyway. *Technologically* speaking, though, I'd be fine with us just eschewing bad materials, like we did with radium, asbestos and DDT. All of which came with efficiency losses. You soak them because of the negative consequences of continued use. If you don't believe climate change is a total canard, the same concept applies to fossil fuels.
"But I am going to say that what naturally follows from them is that you prefer a world without humans, or possibly a world with humans in a state of a primitive society with no internet and no healthcare. "
Idiot hyperbole. Give me the culture of the year 1600 and whatever technology is possible without filling my blood with xenopolymers and I'm as happy as a clam. (Uh, a clam that hasn't been filtering plastics out of the seawater.)
"I'd guess this is more likely - your thought process was probably similar to 'wood can be used instead of plastic for the first 2-3 applications I can think of, so wood is a good overall replacement for plastic'."
If you're going to speculate contemptuously about my mindset, I'd prefer it if you were at least able to keep a whole two of my comments on the topic in working memory. How do you decode "the only real casualty would be e.g. people with stomas or who actually need the oxygen masks on the airplane" as "wood can be used for the first 2-3 applications I can think of so it's a good overall replacement for plastic"? And I said that *first*.
But... microplastics are something we're not even sure have any health effects. Having plastic packaging on the other hand is helping people not die of food poisoning. I'm pretty sure you'd live a shorter, more painful life in the 1600s.
If I correctly understand what you want, then we can get it only by moving forward. London has a lot better air now than 150 years go. Sweden and Switzerland are cleaner and greener than China, because they can afford it. When China grows enough it will also afford moving to cleaner tech and investing into reforestation, or whatever they decide they want to do with the extra wealth.
And moving forward means using plastics more, not less - at least for now.
My seven-year-old son would have some words with you if you tried to replace his LEGO sets with wooden blocks. :)
What material would you use for aortic and other vascular grafts, surgical (implantable) mesh, contact lenses, or the weight-bearing surfaces of artificial joints?
Hm, yeah, plastic-eating microbes do seem like they could have a large downside. Is this likely to turn out like gain-of-function research where we should have noticed earlier that it's not worth the risk?
There are microbes right now that eat wood, but we still use wood for a lot of things.
Exactly, and those microbes have had about 450 million years since the invention of lignin to learn how to eat it. There's no reason to expect bacteria to be more effective at eating plastic, outside of sci-fi stories.
It would be nice to be able to look at discarded shards of plastic in the dirt and feel the same way about them as I do about wood chips: well, they'll rot away in a year or two.
Less nice if I have to treat all plastics the way I treat wood now: keep it dry, seal it to prevent rot, etc.
Sure, but wood also expands and warps if it absorbs water. Makes it a bit trickier than if rot was the only problem, as my poor cutting board can attest.
I wonder what conditions the particular bacteria being studied need in order to grow? Beetle guts would be very specific.
Probably the ocean is the main place I'd want something that can digest plastic to live. I don't see any more feasible way to clean up stuff like https://en.wikipedia.org/wiki/Great_Pacific_garbage_patch
If you want to have your heartstrings pulled, see https://jenny-jinya.tumblr.com/post/190440590650/short-comic-while-i-work-on-something-bigger-on
This is the best end to the week
This post made me so happy.
As someone who has spent plenty of time in academic and political settings, the optimism, flexibility, and open-minded experimentation your grants represent bring me great joy.
"4: Alice Evans’ Research On “The Great Gender Divergence” (?/10)
Dr. Evans has done over four months of research in Morocco, Italy, India, and Turkey. You can find some of her most recent thoughts at her blog here. Her book is still on track to be published from Princeton Press, more details tbd."
I still remain to be convinced that this is going to be of any use to anyone. It's not like this lady academic is starving in a garret with her inspirational work confined to a heap of notebooks under the bed because no-one will publish her, so why does she need a dig-out with an ACX grant?
The Princeton publishing blurb is also ignorant, at least in its effusive enthusiasm that nobody nowhere never did nothing before us did (this ties back in to the query raised on here about "Why learn history?" Well, for a start, so that you don't sound like a Princeton University Press intern):
"Something radical happened over the twentieth century: women entered the workforce and became political leaders across the world. This had never happened before, not in the entirety of human history."
Empress Matilda? Who dat?
Elizabeth I? Never heard of her
Catherine the Great? Is this a cosmetics brand?
Every damn woman who ever milked a cow and churned butter which was then sold at market? Don't be silly, that's not "work" so you never existed anyhow.
I think it's fair to differentiate hereditary from non-hereditary female leaders.
Well, there is measure and degree in hereditary, some inherited … rather actively (as in, personally gave the mutinuous orders to the sympathetic military regiments, leading to removal of the previous legally recognised monarch in favour of herself, and usually the previous monarch's death).
I guess Jeanne d'Arc counts as definitely non-hereditary political leader, although given the highly-hereditary context she had to have her muppet king.
(Of course for the entirety of human history one would need to look at cultures not inheriting from Mediterran Antiquity, too, but even without that)
I agree with you on that, as per my response to drosophilist, but the PUP didn't make that distinction.
Granted, exaggeration for the sake of presenting all the geese as swans ('this book will tell you about things never before seen in the entire history of humanity!') is par for the course for any marketing, but for a university press it's poor.
I also wonder about women who managed to become leaders at a local and national scale before the 20th century - it would have been difficult for them to become political leaders when politics as we practice it were not known, but the PUP said nothing about elections - but that would require me to go read some feminist histories.
Ah, the sacrifices one is called upon to make!
Wouldn't that require you to eliminate from any relative comparison almost all male leaders up until the 20th century? Not many republics between Rome and 1776.
Heads of state were mostly hereditary, but what about high-level officials such as ministers appointed by the ruler, or city leaders elected by the local patricians? Those were almost always men, as far as I can tell.
Conversely, this is the one grant I most want to succeed in some tangibly useful way, because it's the most dubious-seeming one. That'd be the biggest possible update to my heuristics regarding social science research and funding distribution.
Granted, there will always be the opportunity-cost question of "are we sure the money couldn't have come from elsewhere?" (while granting that Scott's grant decisions are entirely up to him, and at the least I doubt she's taking the money for granted) . It's a bit different giving someone a hand up if they're already partway up the ladder...
"I doubt she's taking the money for granted"
But it *was* granted. How else should she take it?
Well punned. I salute you, sir.
For granite is the solid alternative.
I looked at Evans' blog & CV a little bit and I have a guess. There are parts of the effective altruism ecosystem that are working on improving outcomes of the type of work NGOs and governments have been doing - development economics for example. Gender has a multivalent relevance (hee hee - I'll imitate Princeton for just a moment) so anyway it will come up, it will play roles in EA development projects, and when EA sets gender victory goals, EA will want to have research to back up what the best form of gender equality is. I'm guessing here but I think Dr. Evans has taken this on.
Her PhD is in geography, BA philosophy. Here is a person who probably didn't have to digest all the waves of feminist critical theory before being allowed to do anthropological research. This is the sociological intellect escaping the stranglehold of the 20th century. Maybe?
The rest of the book blurb - "The Great Gender Divergence will explain the causes of Europe’s “precocious equality,” how East Asia and Latin America caught up, why gender equity in the Middle East, South Asia, and Sub-Saharan Africa has lagged behind, and why South-East Asia was always ahead. Evans offers a comparative history of how societies come to support gender equality and why this varies around the world, telling a story about geography, economic growth, strong states, and militant activism." That's a strong dive there at the end. Militant activism. That blurb is basically pre-pandemic though.
It will be interesting to see what answers come up. "What Gender Equality should we construct? Global survey of options" might be a better title. We will find out shortly, I guess, how much liberation is good for us. I mean. I have not gone back to check how much money this grant was for.
I think Scott supports gender equality in general and so if this line of investigation went somewhere - if there were some recipe that could be lifted out from some social arrangements - this would be knowledge he would be glad to have supported obtaining. Since "make it fifty-fifty!" is weird.
I ordered a book from a used book site recently and it turned out to be by a different writer with the same name - it was very cheap and followed a list of books I needed to get, so I didn't really check & just added it for curiousity's sake, and it turns out to be a treatise against sociobiology and for historicity, that human societies are primarily shaped by their collective choices rather than biological constraints. If this guy's making a solid point, whatever Evans discovers will be useful for community development only insofar as the target society adopts it.
One of her blog posts from the summer is about misunderstanding patriarchy, though. I haven't read it yet, but here's someone doing self-evaluation in addition to geography-based gender research. It checks the EA boxes.
Her website says she wants to supervise PhDs on (list), including: Why have female employment, representation and activism boomed in traditionally Catholic Ireland and Spain but not Italy?
---I don't know if you were actively seeking your next PhD, but hey.
"when EA sets gender victory goals, EA will want to have research to back up what the best form of gender equality is"
May I be taken from this world before that happens. If EA is going to meddle in "not enough women in Whatever", then at best it'll work out like their politics meddling (spent a ton of money, candidate was beaten by the obvious shoo-in) and at worst it'll make things worse (hm, we managed to get 60% more women in Whatever but somehow that hasn't resulted in the earthly paradise, and minority communities are still struggling with poverty, crime, illiteracy and family breakdown. But twelve of the women we got appointed to cushy sinecures have bought million-dollar second and third houses, so that's nice! https://nymag.com/intelligencer/2022/04/black-lives-matter-6-million-dollar-house.html)
I'm going to say this: what promotes gender equity (and what the hell does that phrase even mean? women now in the workforce doing what were traditionally men's jobs? women in the home having their work recognised? 50/50 men and women in every field of employment?) is, if I believe all the reports and studies that have been pouring out about this for years, is
(1) Education
(2) Contraception
(3) Employment
And how do you get those things? You need your society to be:
(A) Secularised
(B) Industrialised
There you go, and I didn't even need $20,000 of a grant to tell you that. I'll wait and see what Dr. Evans' book is like, but I'd be hugely surprised if she decides "Um, gender equity happens when all women are the Angel In The House".
https://en.wikipedia.org/wiki/The_Angel_in_the_House
Mostly my objections are that this is an established academic with an established career who has had no problems getting published before, not a new person starting out or a project with no other sources of funding. But it's Scott's money and if he wants to give it away to whomever he likes, he can. He can do a KLF and set bricks of it on fire, too, if he likes.
(I do think the mobile slaughterhouses one is a good idea and deserving of a grant).
Alice Evans actually does know more than your random internet feminist, and has a post on things she knows now that she didn't before:
https://www.draliceevans.com/post/3-things-i-got-wrong-about-patriarchy
And what she still doesn't know:
https://www.draliceevans.com/post/what-don-t-we-know-about-patriarchy
Yes, I have to say I was less than impressed with the whole "I rejected the idea that religion might have an effect on social culture and practices" article. It's good that she can acknowledge her blind spots, but if she's playing catch-up with what has been a commonplace in the field, then how much original research is she doing and how much is her book just going to be another coffee-table production, to be displayed but not read?
If you're curious why activism is included, it's relevant to her theory of communist vs non-communist states. Communist states crushed traditional patriarchal cultures and were officially gender egalitarian, but she finds that today ex-communist states are still less egalitarian than capitalist ones. Her view is that communism crushed independent activism, so there wasn't really a women's movement in those countries.
Oh that’s interesting. I was wondering if the communism crushes patriarchal culture -> gender egalitarian dynamic would persist.
I was also wondering if a culture would be more egalitarian if it had experienced x centuries of colonization or outside rule during which time men & women were roughly equal status as peasants, and then if the culture regained self-rule it might be more egalitarian.
I think the presence of activism is an interesting choice of metric as well. I can’t decide if it indicates less oppression (otherwise it would be crushed) or more oppression (conflict can’t be solved adequately without protest).
Her post focusing on that is here:
https://www.draliceevans.com/post/did-communism-smash-the-patriarchy
My theory would be the really simple one that general wide-spread middle-class wealth seems to be at the heart of approximately all social progress, and Communism was rather bad at wealth generation. People can't invest energy in politics if they're too busy trying to survive. (And oil sheikhs and other nobility both can afford to be ridiculously wasteful because they're *too* rich and have a very strong incentive to avoid rocking the boat, just in case. It's the middle class that pushes for change)
Deiseach, I love your witty comments, but I disagree with you here. There is a world of difference between "a few high-born women, like Hatshepsut and Elizabeth I, held power" and "women are routinely elected members of Congress/Parliament and heads of state, in addition to having positions of power and influence in industry and science."
Likewise, yes of course milking cows and churning butter is hard work, but the modern world is unique in having women work *outside the home, for money, in large numbers*.
I agree that hereditary and non-hereditary leaders are different.
What PUP means is that "in the 20th century, women went into the industrial workforce in large numbers, and ran for political office, for the first time".
That is not at all the same as saying that "women had never before been part of the workforce or leaders in the entirety of human history".
Women working outside the home, for money? Happened in the 19th century down the mines, for one instance:
https://blog.dol.gov/2022/03/22/a-brief-history-of-women-in-mining
https://www.historyextra.com/period/victorian/the-scandal-of-female-miners-in-19th-century-britain/
Women working in cotton mills:
https://www.historytoday.com/working-woman%E2%80%99s-place
"A significant minority of wives, however, worked from sheer necessity. In the 1840s, in Lancashire alone, a survey of 412 cotton factories found that just over half of the 116,300 workers were female. Around 10,700 of them were married women. These women were not the sole breadwinners: over 9,200 of the husbands in this survey had regular work. Only 821 husbands were unemployed, undermining some critics’ claim that women worked, while their spouses boozed in the nearest pub."
That's before we even begin to touch women's work outside of Europe:
"Who can find a virtuous woman? for her price is far above rubies.
She is like the merchants' ships; she bringeth her food from afar.
She considereth a field, and buyeth it: with the fruit of her hands she planteth a vineyard
She maketh fine linen, and selleth it; and delivereth girdles unto the merchant."
But of course, this is a publisher's blurb, which means it is dumb as mud.
Is it a difference with importance? Is a country better governed for having 50% female MPs or CEOs versus 10% or 100% or 0%? Are women as a class more or less happy? Are men as a class more or less happy? Are families more functional, is the market more efficient, is crime higher or lower, are more or fewer people addicted to heroin?
I wouldn't pretend to know the answers to any of these questions, but it seems to me that answering them definitively is way more important than simply viewing a percentage of women (or minorities, or left- or right-handed people, or people with/without tattoos) in Profession X that is close to their proportion in the population as a big win to be celebrated all by itself. There's a certain insect mindlessness to the drive to have every job or role in society have a representation of sex and skin color and other such surface ephemera that matches that of the entire population that is discouraging in its failure to ascertain whether there is any actual measureable value in the result.
I mean, an AI maximizing paperclips comes to mind.
Alice Evans is not dumb and is entirely aware of women working on farms, her research makes a big distinction between that and going away from home to make money.
If that's your definition of "workforce", the vast majority of men don't qualify either prior to industrialisation (and post industrialisation many women did - industries like weaving stayed very female dominated while being among the very first to industrialise).
The shift, very broadly, is about the level of *recognition* and *pay* women get for their work, and of a few specific occupations removing legal barriers to entry, but for most of human history women have worked alongside men in broadly similar occupations. (Eg. many (though obviously not all) medieval guilds let widows inherit their late husband's business - they'd learned the trade over the years of their marriage - even though they didn't let women train in the field independently)
I did not say Dr. Evans was dumb, I said PUP's blurb was dumb. I would certainly hope Dr. Evans' book was finer-grained, e.g. "In the 20th century, globally women entered the industrial workforce and politics in a way unmatched before".
"Working on farms" and "going away from home to make money" were not mutually exclusive, it seems you have not heard of hiring fairs:
https://en.wikipedia.org/wiki/Hiring_and_mop_fairs
Potato harvesting:
https://blog.nls.uk/politics-publicity-and-potatoes-scotlands-tattie-howkin-films/
Hop harvesting:
https://en.wikipedia.org/wiki/Hops#Migrant_labor_and_social_impact
Or the great British project for sending orphaned and poor children to Canada, Australia, New Zealand and South Africa as servants and farm labourers:
https://en.wikipedia.org/wiki/Home_Children
Hmm, that's a little discouraging. That's an activity that very few people of either sex did up until late Industrial Revolution. Prior to that the majority of any population worked on a farm, kept animals, or did small-scale local artisanry -- smiths, weavers, carpenters, et cetera. Money in the sense of gold and silver wouldn't even change hands that often among this class, a great deal would happen as a result of a complex web of favor exchanges and barter.
Historians have historically focused on the easier to study people with what they saw as more complex and interesting lives, namely the nobility and extremely wealthy, who might well have had wide choices to make in professions, and for whom money income was a major fact of life. But I thought it was recognized some time ago that this is an unfortunate distortion of history, as it studies only a tiny elite fraction of humanity, and gives little insight into what everyday life was like for most.
It's discouraging that (from what you say here) Dr. Evans would appear to continuesthis trend. It would be much more interesting to look into the roles of men and women in, say, the households of peasant farmers and small artisans over the past millenium or so. That would tell us a lot more about what life was actually like for the typical schmo (male or female).
The most interesting bit I got out of these reports was "which grantees either didn't understand Scott's instructions that 5/10 was the expected amount of progress, or inflated their scores to match more standard out-of-10 rating systems".
Judging by comparison with the textual descriptions, I'd estimate at least a third of the grantees did this. I put much more probability on grantees' systematic incentives to inflate than on Scott's theory that things went systemically better than expected.
That being said, this whole thing remains super cool and I'm not nearly as surprised as Scott that the ACX exposure counts for a lot on its own!
In fairness, if you received a grant for getting a job, and then you get the job thanks to the grant, is that 5/10 because that was the expected outcome? And if so, what would the 10/10 score represent? You got a better job than the one you wanted, *and also* a porn star mistress who was attracted to you specifically because of the size of your grant? "Hoped for job, got job, 10/10 success" seems a lot more intuitive.
I feel like the majority of these ratings are actually using a 4-to-10 scale where "as expected" is around a 7 or 8. I think the only rating below 4 was the guy who hasn't started yet, and the updates with 7s generally don't mention anything that seems like obvious good fortune.
#30 says their project has been de-prioritized and they hope to get back to it eventually, but gave a rating of 6/10.
#22 says their original plan won't work, their timeline has ballooned, and they're worried their work might soon become obsolete, but gave a rating of 5/10.
(Apologies for picking on you; you're just the most clear-cut examples of an unsurprising pattern.)
Yeah, I don't think it's feasible to tell people "a 5 indicates that you're doing as well as expected"; almost no one will actually rate themselves that way. To actually be doing as well as you expected, on a major project, is (given the natural human tendency towards optimism) a great big success! No one who feels like they're succeeding is going to give themselves a 5 out of 10: a 5 out of 10 is at best average and at worst an F.
Some of the reports make me wonder whether _two_ grades would be more informative: basically «help us do X to achieve Y» has «we did X as planned and with good quality» and «X helped with Y» parts. I.e. it looks like 6. did things in their control fine, but things out of control played out not so well this time.
And maybe in addition to the median 5/10, it's good to say something about spread: is «needed to win a fair coin toss, won/lost» 6/10 and 4/10?
It looks a little strange to me that David Bahry apparently doesn't acknowledge the ACX funding in his academic works.
I didn't ask him to and I imagine that would just anger the people who hate my politics, so I'm glad he didn't. In general everyone should feel free to deny ACX funding if it helps them in any way.
...does this apply to any other funding someone might find problematic?
Well, in some cases academic ethics (or whatever the correct term may be) dictates that a researcher makes all their funding public, whether the grant giver insists on it or not?
(I assume in medicine you cannot just omit the fact that your cancer research was sponsored by Philips Morris, even if Philips Morris does not insist on being acknowledged?)
I am not claiming that this is the case here (the preprint is in Biology, not medicine, and does not include a statement of financial interest or whatever...)
Regarding Promote Economically Literate Climate Policy In US States comment "Overall they report frustration, as many of the legislators they worked with have been voted out or term-limited": Possible silver lining. I recall a Cato policy analysis from many years ago that reviewed congressional representative's voting records relative to time in office. The longer they are there the more they tend toward government run solutions and oppose private alternatives. If so, and assuming what they found at the federal level applies as well at the state level, the fact people have been voted out (and thus replaced by freshmen legislators) may in the end be helpful to their plans.
Did they find a difference between second-term legislators who eventually serve ten terms and second-term legislators who get only end up having three terms, or a difference between second-term legislators who eventually serve ten terms and ninth-term legislators who eventually serve ten terms, or a difference between current second-term legislators and current ninth-term legislators? The description sounds like the latter, but I would want to know about the difference between the first two, which identifies how much is selection and how much is causation.
Well it helps you identify selection, it’s not a silver bullet.
We did not get one of these grants, but we were listed on the applicants thread, and it helped us get some help with the project. Since then, we've partnered with a major law firm and are investigating PPP loan fraud. It's been a lot of work, but it's nice to have apply our analytics skills to something rewarding instead of helping a giant business optimize click-through rates or something. We can't talk about any cases until they are brought out from under seal but we're extremely happy with our traction and we've begun making inroads helping healthcare payers such as health insurance companies, unions, and large employers, prevent fraud.
Thank you very much for running this program: you wound up introducing us to our first angel investor. He really helped us understand how this could be more than a cute idea to punish some bad guys, and that has paved the way for the other angels which are following. We really appreciate that programs like this exist, and will report back at the next ACX grant update.
I would greatly enjoy if Scott wrote an anti-portfolio update on applicants he most regrets not funding in another year's time
I have no other comment other than to say that I enjoyed reading this greatly, more so than the original grant announcement, and found this greatly energizing
This is awesome, Scott. You are a force for good and an inspiration.
I am amused that it was hardest to detect progress on the neutrino grant.
I don't know if that one worked, or if it was "take the money and run". If it was a dodge, then one out of forty grants isn't a bad outcome, I suppose, since you do have to anticipate that at least some applicants are not wholly in good faith.
If SD does turn up with a full slate of reports and data, I will apologise to them 😀
Sorry - I just completely missed Scott's email about this. I emailed him my progress as soon as I saw this post and it's updated now. I can't share specifics publicly at my advisor's request but the thesis will be available online after it's complete.
> Help [Anonymous] Interview For A Professorship
Why is this only 8/10? It sounds like they succeeded in their goal; what would have resulted in them rating it 10/10? Were they hoping for a more prestigious school?
Since 5/10 is "things going as expected", 8/10 looks like everything went really well (perhaps faster than expected, or actually a more prestigious school than thought). I understand 10/10 on this scale to be extreme overachievement.
> The grant was to help make it financially easier for him to go on a long round of interviews [Anonymous] successfully got a job offer from a top school[...]
Typo, missing period after "interviews".
Great to read about all those promising things!
On the 6.5/10 average: might this not be simple publication bias? That is, people being less likely to send you progress reports if things aren't going as well?
The ratings are an interesting psych experiment in themselves. I read the instructions in the post and thought "yeah, that's not going to work as intended," and I think that was borne out by the ratings given.
If you've met your goals, it's psychologically extremely difficult to give a rating of 5/10 - I mean, 50% isn't even a passing grade! Hence the reports that say things like "we had some unexpected setbacks but achieved most of what we intended, 7/10" when that should probably be 4/10.
I think it's (mostly) not dishonest inflation, but people either finding it too hard to overcome their cognitive bias or simply misreading the instructions (like the comment from Isaac King).
Scott, I think if you do this again you should say "Rate your progress on a scale of 1 to 5 relative to what you expected. If things went even better than you expected, you can give out-of-range answers like 6/5 or 7/5." Logically equivalent, but it will make all the difference psychologically: I think you'd get an average rating of about 4.
What's more, probably some grantees used the scale as you intended, some used it the way Isaac read it (10 is fully meeting expectations), and some did something in between, where they felt torn between the instructions and the natural way to interpret a 1-10 scale and plumped for a score in between the two - so it's not meaningful to compare scores between grantees or do any comparative statistics like "teams scored higher than individuals".
For the ones that say "They have asked me not to discuss their progress yet." my hope would be that once progress updates are allowed they come with a *really* good explanation of why progress updates were withheld. I am personally a big fan of open science, and keeping information private feels like it means you are striving to achieve something other than "betterment of society/humanity" i.e., not particularly altruistic.
If these people can provide great rationale for keeping the information private (such as releasing of a status update would cause some kind of bias in their data collection) then we can all walk away thinking it was a wise decision.
If, on the other hand, they just wanted to keep updates private because they were waiting for a patent application or to secure their position as a "leader" or to get their work published in some reputable journal then I think future grants should have a stipulation that grant recipients *MUST* share status updates and not withhold details.
A lot of the withheld ones were political and thus their success evaluation will be radically different in a week when the midterm results come in. I do agree that Scott should make a supplemental post sharing all the updates that were waiting till after the mid-term elections, within a week or two of said elections being over.
"Saharon is a prolific and influential Israeli mathematician, but many of his discoveries are hand-written in an unpublishable format"
Put me in mind of this video on the history of the Fast Fourier Transform - https://youtu.be/nmgFG7PUHfo
Spoiler: Gauss discovered the algorithm originally, but nobody noticed as his written description was too obscure.
I got a grant by being forwarded to the LTFF for some concrete ideas on expanding on a text about forecasting[1]. I have done 4/6 of the things I promised I would do, the two missing ones being adding a literature review and extending the analysis to continuous questions. I give myself a 6/10.
[1]: https://niplav.github.io/range_and_forecasting_accuracy.html
To speak a bit more about how contingent Manifold's existence was on ACX Grants: I think there's like an 50% chance that me/James/Stephen would not have chosen prediction markets (among our other crypto ideas) if we weren't submitting a proposal ACX grants specifically; crypto was on our minds as you can see from our original application: https://docs.google.com/document/d/12DufTzcIaJxSQOxMYIHAwt7tGa5PAkIyhZOQ2JrmCfg/edit
I also think there's a 60% chance we wouldn't have continued with a prediction markets project if we hadn't won an ACX Grant and gotten the subsequent publicity through that announcement and Mantic Mondays. ACX readers formed the bulk of Manifold's initial adopters, and still represent our most vocal and engaged users.
We owe a lot to ACX and Scott in particular, and hope to pay back that generosity with useful predictions of all kinds!
I want to like www.cleanthedarnair.org, but it looks way too greenwashed.
They're planning to spend 2/3 of the tax money on tax credits for poor/mid incomes and eliminating sales tax on grocery store food. Then an additional $93M goes to rural economic development, airports and highways for some reason?
Only about 1/6 of it, $100M, goes to actually cleaning the air, which they say they'll spend on "cleaner school buses, electrifying lawn and garden equipment". So like a fleet of shiny new electric school buses and a mail-in rebate if you buy an electric lawnmower? And nothing about toxic dust from the Great Salt Lake drying up due to the drought? If you're going to spend $50M on rural economic development, shouldn't that at least be aimed at water conservation?
> The average project rated their progress at 6.5/10, where 5 was “meets expectations” (this average includes some private projects I didn’t fully list here). So either people systematically underestimated their chances when applying, or they’re systematically inflating their progress now to look good, or things systematically went better than expected.
No, none of those things. People are absolutely awful at assessing their progress. They think they're doing better than they are - they're not inflating their progress to look good to you, but they are probably inflating the progress to *themselves*, and **certainly** systematically ridiculously overoptimistic about how their progress compares to their problem. Which is normal. Wikipedia's planning fallacy example has average real-case estimate 33.9 days, best-case 27.4 days, worst-case 48.6 days; reality averaged 55.5 days, 70% of students taking longer than predicted. That's with a fairly narrow scope where they think they perfectly understand the problem, which is applicable for about zero of these grants.
"Do 90% of the work in the first nine days, and then the other 90% of the work on the tenth day." Halve the ratings and you'd have answers closer to the truth. Though some of them are still way too optimistic by that measure. Anyone who cites getting more money as progress, or is looking for more money now, halve their score again. Or just round down to 0, that's probably more accurate.
Re. this: "Trevor Klee...realized this would be too expensive to do in humans in the current funding environment, and has pivoted to getting his medication approved for a feline autoimmune disease as both a proof-of-concept and as a cheaper, faster way to start making revenue. ... He still anticipates eventually moving back to humans."
This is a very common trap. I don't know of a single case where it worked out for the humans. A company wants to develop a drug for humans, but FDA approval is too expensive, so they get it approved in dogs, horses, or cats, thinking they'll make enough money to push it through approval for humans. They never do; it's hard enough to keep a new pet-drug business above water. Then their patent expires, and the drug will never, ever be approved for humans, because nobody will pay for FDA approval of an unpatentable drug.
Until just a few years ago, this was almost acceptable, because you could go to Agway or a foreign website and buy the drug for your dog and take it yourself. But governments and industry have taken EXTREMELY aggressive steps in the past few years to prevent this.
An example is Adequan. It temporarily cures arthritis in humans. It cures arthritis in me! But it will never, ever be legal for humans in the US or Europe, because it was developed in the 1980s and the patent expired long ago. But it is approved for dogs, horses, and pigs in the US.
Until very recently, I could easily buy it over-the-counter in any country but America, or order it thru ebay or from any overseas veterinary website. But organizations have just taken many steps to prevent any humans from taking Adequan, including:
- turning it into a prescription-only drug in all Western nations
- (I presume) prosecuting online pharmacies that don't comply (as they all suddenly went out of business or stopped selling it)
- purging overseas pharmacies that sell it over-the-counter from search engine results
- purging pharmacies that sell it without prescription from TrustPilot listings
- creating an organization to rate the "safety" of online pharmacies, which actually rates their compliance with import regulations
- investigating ebay sales of Adequan to verify possession of a veterinary prescription
- banning ebay users who sell it to people who have no such prescription
- delisting Adequan sales on ebay entirely
- in my state, creating laws for dog licensing that forbid veterinarians from treating unlicensed dogs, and forbid licensing dogs without proof of legal ownership of the dog, including an investigation process to verify possession of alleged stray dogs, so that you can't even get Adequan by borrowing your neighbor's arthritic dog, licensing it, and taking it to the vet to get Adequan (I tried)
I can't think of any reason anyone might have to prevent people from taking Adequan, so I think it's just part of a general tightening of access to veterinary drugs, which has been going on for about the past 5 years. The drugs for which doing this makes some sense -- antibiotics -- were left for last, but the US govt. has finally gotten around to them, and IIRC sometime next year it will become impossible to buy any veterinary antibiotics without a prescription.
Ivermectin was first approved for use in cattle, horses and sheep. So it (getting a medicine approved for vetinary use before getting approval for its use in humans) has been done.
It's now on the WHO list of essential medicines, IIRC, so pretty successful. I have no idea about the economics of the case, though.
I would be very interested to hear why this strategy worked for Ivermectin (and possibly others?) but not for other things. Maybe there is something useful we can glean from its story to try to reproduce it with other therapies?
I generally agree with Phil's concern about animal-first drugs suffering the same problem as repurposed drugs and would love to see a solution to either problem!
I would be interested too. I don't work in the industry; I just recall TV commercials for Ivermectin as an anthelminthic for sheep in NZ in the early 1980s.* ("Kills roundworm, lungworm, and pinworm too!") Wikipedia corroborates my memory.
I only recently learned that it was approved for use in humans, when the controversy about it cropped up two(?) years ago. I was amazed and amused.
* TV commercials in NZ are, or at least were then, bizarre to residents of other countries. A good fraction of ads was devoted to agricultural products and services - quad bikes, fertilizer additives, spraying equipment, lime spreading, etc.
The Wikipedia page seems to have some clues. It was the "bestselling veterinary medicine in the world", indicating that Merck did not have any problem in this case with what Phil called "keep[ing] a new pet-drug business above water." Critically, it was adopted by the livestock industry, and not just pet owners. Then, it was approved for human use in collaboration with the WHO, and Merck offered to "donate all ivermectin needed to eradicate river blindness ... ". So it seems that the motivation for human licensing was at least partly for publicity, and perhaps even from a desire to do good in the world. The inventors went on to win (half of?) the Nobel Prize in 2015 for Ivermectin.
So, it seems the formula is a) have a miracle drug b) with important agribuisiness application c) which also treats crippling illness in photogenic children in the developing world. Perhaps the process for drug approval in the 1980s was also a bit less expensive?
Yes, it would be good to have some more examples. I'm confident they exist, but perhaps "doing it a long time ago" is key for success.
There's also two incentives for getting a drug approved for a rare disease. The Orphan Drug Designation gives a company some special tax breaks for the research costs plus added years on its patent.
There was also another rare disease incentive that provided a drug company that got one approved a "pass" for accelerated review on some future drug. This 'pass' could be used by the drug company or sold to another one. It's only been done a few times I believe since the FDA is actually pretty good about doing reviews quickly and granting accelerated status when the drug is in a critical area.
I think the part where the human use is more charity than profit motivated makes a huge difference, because it means being out of patent or nearly so isn't a drawback.
I didn't know that about Ivermectin, but I see it was developed in the 1970s. Here's a report on a study by Tufts on how drug development costs in the US rose from an (average? median? geometric average? doesn't say) of (if I'm reading the bar graph right) about $170 million in the 1970s, to nearly $2,600 million in 2000-2010: https://cen.acs.org/articles/92/web/2014/11/Tufts-Study-Finds-Big-Rise.html
There's a version of the same chart here, this time with the bar graph values labelled: https://www.statista.com/statistics/265054/pharmaceuticals-cost-of-development-in-the-us/
They don't mention any costs in drug development other than doing safety studies, opportunity costs, and the costs of drugs that failed (which are factored into those prices); so apparently the *research* costs are negligible. AS USUAL, the journalist didn't say who wrote the article, or where or when they published it, or what its title was; so I can give no citation.
Anyway, Ivermectin would have had a relatively low cost of approval compared to drugs today.
So I guess the solution for future drugs is to develop them in the past before medical regulations got out of control? 😬
"An example is Adequan. It temporarily cures arthritis in humans. It cures arthritis in me! But it will never, ever be legal for humans in the US or Europe, because it was developed in the 1980s and the patent expired long ago. But it is approved for dogs, horses, and pigs in the US."
Unanswered question: Given the massive amounts insurance spends for arthritis treatments, why wouldn't a coalition of payers simply fund the studies? If they work, then you have a nice, inexpensive drug that works in humans and it will be produced as a generic drug?
You have to continue taking Adequan for the rest of your life, because it doesn't really regrow cartilage. Apparently it lays down something that acts like cartilage, but quickly wears away. (It's not a drug in the usual sense; it doesn't interact chemically at all AFAIK.) It's a lot cheaper for insurance companies to tell people with arthritis to take Naproxen, which is what doctors usually do.
Besides just bias and inflation, one might have different scales (that is, completion could be 10 for someone and just a 7 for others, an expected benefit could one's 8 or another's 11).
Also note - I'm not sure if we can even avg these scores out in the first place. As in, if you get a 3 and 7, does that equal 2 5s? The 3 might be much worse compared to a 5 than the 7 is better.
Ignoring all that lets have some fun making some other markers for the scale.
Limited scale -
1 - Failure
3 - Delay
5 - Expected Progress
7 - Full Completion without issue
9 - Unexpected great success
From that, I redid the scores from my not that optimistic guesses based only on the descriptions.
1-10 6?8?336756
11+ 8?55623735
21+ ?2867?645?
31+ 6759?5?676
Avg non ? ~5.5
Note for 30 I changed to a ? cause it didn't really fit in the scale. Also on a few I graded actually a bit higher than what they gave themselves.
The avg for the visible ones above was 6.67 so if we assume the invisible ones were roughly similarly biased, then it's still just about a point difference. Doesn't mean there wasn't a lot of bias - the most common dif was a 2, but the next highest was 0, so it avg out.
Full (not to scale) scale
0 - Likely scammed/tricked. Not necessarily by the grantee, mind you.
1 - Canceled/ full failure and can't continue/ hit obstacle that would warrant a cancellation at any point.
2 - Significant delay or backtracking. Unknown or difficult to guess how major it is. (note that this could be worse than 1 in many cases. At least 1 might be quick)
3 - Stalled or "small" failures. Generally still able to see an end, but might have already failed to meet some goals/objectives.
4 - Minor obstacles that have caused some delay/deviation from estimates, but nothing that could count as failures.
5 - Free Space
6 - Smooth progress.
7 - Full success
8 - Went well enough that it expanded scope/target of initial project/task. (Note - might want to do 8->3 or something if expanding it caused more issues later on...)
9 - Success + recognition/major achievement/discovery/etc.
10 - Made world news. In a good way. Hopefully.
Thank you so much to Scott and all the ACX readers who have supported Legal Impact for Chickens!
My main takeaway is people suck at using numeric scales.
10/10 for achieving the first step out of a many step failure prone process? 8/10 for doing precisely and exactly what they said they'd do? You marked 5/10 as "as expected", what's up with all the insane grade inflation? I assume it's because nobody think of 5/10 as being "as expected", people don't want to think of themselves as failures (which 5/10 feels like), so they mark themselves up ridiculously?