Overall, I share Scott's view that this round went fine but was not a resounding success (I might have rated it a 6/10 fwiw), and have many thoughts about how to make our future rounds better. Chief among them: open up trading to nonaccredited investors via mana/charity dollars; do more frequent retro evaluations; provide more feedback and support for grantees.
Beyond the large ACX Grants round at the end of the year, we plan on a few other experiments with certs including one sponsored by Manifold specifically for Manifold community projects, and perhaps an essay contest with ChinaTalk backed by impact certs. If you'd like to sponsor an impact cert-based contest for your own org (for-profit or charity), reach out to austin@manifund.org!
And of course, a huge thanks to Scott for believing in impact markets, working with us to figure out how they could be set up, promoting this effort and funding the whole thing! None of this would have happened without Scott pushing for it~
- I think the $8k invested in subsidizing Polymarket markets skews the metrics of this round a lot; of that, $7.4k or 93% was invested by a family member of the founder, which I'd argue is much closer to "founder keeping equity" than "external investor deciding the project was good". Removing that brings the total invested to $26k invested for ~$21-$31k of impact, which seems pretty reasonable.
- I'm not sure what Marcus is smoking but imo $50k for figma mocks and some backend glue code is crazy. I could see a slightly better case if they'd actually shipped a working prototype, and a much better case if people were actually using the thing.
- I am a bit disappointed by the investor's choice to pass on some projects, eg Daniel Reeves's small-group futarchy proposal: https://manifund.org/projects/implement-a-prognootling-app. Perhaps I should have funded this myself when it wasn't on track to hit it's funding bar...
- In general, the results do update me towards "for early stage projects, a YC-like entity paying a lot of attention to project selection may be better than a competitive marketplace system". But I think the accredited investor filter may have also cut out a lot of good capital allocation, and I'm hopeful that we can figure out a solution here.
I try much more to evaluate the worth of the result as opposed to what I think it would cost to construct what was done. This is why I gave a lot of projects a score of $0. This is well recognized in the for-profit space where some companies are valued 100x or more of what VCs invested in them and often 0 but this doesn't seem to be the case in the non-profit world (yet).
For Devansh, I think he successfully accomplished step 1 of creating a viable and useful impact certificate market that solves some problems (only one impact cert per project, people actually coming in and retroactively funding the work, etc.). If successful, I think this is worth millions of dollars in improved outcomes in the non-profit space and thus getting past the first few obstacles makes it worth a lot more. Call this a 5-10% chance of the worth of the project being $>1M
Appreciate you sharing your thinking! My brain finds "100% chance of $50k" ludicrous but "5% chance of $1m" kinda reasonable, actually. Though, as Scott alluded to, it seems like Devansh's project is still mostly predicated on future impact, as opposed to current results -- a bit contrary to the goals of doing a retroactive impact analysis.
"If the project is partway finished, judge it based on its current state and what that suggests about its future results. For example, if a project is 95% done, that's less good than it being 100% done (because it still might fall apart later) but still good evidence that it will eventually be completed, so probably you should give it a score slightly lower than if it was completely done."
I took this to imply that I should judge it based on the current progress as well. If I were to be more clear, I think it is like 10% of the way to something worth several million but I discount that a lot due to risk, time discounting, etc.
I still struggle to understand how to balance the impact / earnings point here.
It seems very unlikely that these markets will return better (or even comparable with) simple capitalistic markets if I wanted to make money.
If I wanted to donate to charity / have a direct impact, my marginal tax rate is c.50% so I can essentially double my money by donating directly to a charity (this isn't quite doubling, as the value of the services funded by my taxes is a lot higher than zero, albeit a lot lower than e.g. malaria treatment).
This is, I acknowledge, a general critique of impact investing.
This seems exclusively useful for (and maybe this is in fact the entire point, albeit I see the potential scale here as small)
1) getting something off the ground that wouldn't reasonably qualify as a charity but which you think is socially useful; or
2) Harnessing investor time/expertise to assess projects and outsourcing that from grantmakers.
This tax rate issue is a good point. Is there anything that can be done here? Maybe not soon, but is there a path toward eventually getting impact markets categorized such that I could deduct such investments from my taxes?
Probably there' some tension here between the investment vehicle side and the charity side. But I dunno, it seems like it would be a pretty pro-social outcome if impact markets were a tax-advantaged way of making money over capital markets.
The issue with tax exemption is that it would require the government to opine on what constitutes 'impact', which is currently much broader than what constitutes charity; the net effect may actually be to sideline the novel, interesting ideas that impact markets are supposed to enable.
I think no. Suppose that you break even on an impact market, so you end up with exactly as much money as you started with. How should this affect your taxes?
If you profit on an impact market, I think this is normal capital gains. It would be neat if Congress changed that, but that sounds like the kind of thing that would take a twenty-year, multi-million dollar lobbying campaign.
The part I've been thinking about more is the final oracular funder's perspective. I think if these are charities, then they don't pay taxes anyway and it doesn't matter, but I'm not sure.
My first stab would be be, you would get a tax deduction equal to the amount you invested, and then capital gains tax on the money you get back. So it would effectively convert your tax on that money from your normal tax rate, to a capital gains tax rate. Yes, this is a hack to lower your tax rates, but it seems to be within the bounds of the sort of thing we already have? Like capital gains taxes existing at all, or charity deductions existing at all; both are ways of lowering your effective tax rate as long as you do something "virtuous" (investment/charity).
You can imagine exploits here, if someone spins up a fake charity or fake impact valuations. But I think societies already have some good experience prosecuting tax fraud, so like many legal areas, this probably doesn't need to be mathematically foolproof.
Note that my first-stab proposal above has the property that it encourages impact investing regardless of how good the investor is at predicting impact; the idea is that even if you gave the money to a zero-impact grant, you were doing it charitably, so the government should give you the deduction. Maybe we'd instead want a system that incentivizes *impactful* charitable giving? I'm not sure though; there's no impact-test for existing charitable donations, and you get tax benefits from investment capital *losses*.
Anyway, I agree this is probably mostly theorycrafting. Although it'd be nice to think our society could figure out improvements to the current tax code that encourage good things, the fact that prediction markets are still struggling with legality gives me little hope for impact markets getting any legal benefits.
I think the tax implications here are worse than you suggest. Suppose there's a charity which will do what it says with certainty: for example, some homeless shelter that always shelters the same number of homeless. You, a rich person, invest 100% of your money in impact certificates for that shelter; at the end of the year, funders pay you back exactly the same amount you put in, eg all your money. Now you don't have to pay any taxes. Why wouldn't everybody do this, so that the government couldn't collect any tax at all?
I assume by "all your money" you mean "all your income", not "all your preexisting wealth"? (Since wealth is not currently taxed anyway.)
In my model you would pay capital gains tax on the money you got back. So you've basically converted income tax into capital gains tax. Given that rich people's income is ~all capital gains already, this seems fine?
Sorry, I didn't understand that part. I still think this is problematic. Capital gains tax is ~20%, income tax is ~35% at high brackets. There are many people who make high incomes who would like to pay 20% instead of 35% on those high incomes - is there anything that would stop them?
It seems like the requirement for the founder to set the price is difficult here, and lead directly or indirectly to some of the failures here. Notably, Max ended up selling 100% of his project at what was (in hindsight) too low of a price, and the Base Rate Times didn't raise any money because the founder overestimated what investors/funders would be willing to pay.
In practice, the way VC markets work is closer to a fixed VC/founder equity ratio, with the amount of equity sold typically close to 10% per round across a wide variety of businesses. The _price_ of that equity is set by some combination of market demand for the equity and the expected costs of the startup over the next ~2 years until another fundraising is needed.
This is a result of some apparently-irrational behaviors, such as founders being reluctant to raise valuation even in the face of high investor demand, and investors being reluctant to buy "too much" equity for fear of reducing the incentives on the founder to succeed.
I wonder whether similar effects would be useful in an impact market. For example, if every participant was always required to distribute 75% of the equity for their project and keep 25% as "incentive", and the opening auction solely served to set the price for that 75%, which the founder has the binary choice to take or not.
I wouldn't describe the behaviors as irrational. There are a lot of risks to raising the valuation (down rounds and cap table resets). It's also harder to hire employees if you have an inflated valuation (since part of the promise is that they will see a growth in their equity). You already pointed to the problem from investor side about reducing the founder incentive (this is the biggest investor risk).
The last part of what you said is pretty close to what happened: the auction solely served to set the initial price, and the founder had the binary choice to take it or not—they just set their bar for taking it in advance, as a minimum valuation, and the Base Rate Times guy set it very high.
It's possible though that the auction didn't price set very efficiently because there weren't very many investors and they weren't that profit-maximizing, though as Austin said in his comment, we have ideas about how to do better next time on that.
I do think having a default (or forced) equity option is good though, and we're interested in doing that for future rounds! Insofar as it's in our control. Also thinking something like 75/25.
> the fact that we overall disagreed with funding decisions is pretty damning.
I feel there is an obvious solution here: have the funders write up a short initial impression on each project. This reduces risk for investors and lets them focus on the value they're actually bringing: predicting outcomes rather than predicting what judges will think about outcomes.
Like, much waste could have been avoided if something like this had been written *beforehand*:
> Polymarket - We have trouble figuring out why this would be better than the many other real money prediction markets that happen on Polymarket all the time, and we feel like we already have good data on how real-money markets compare to play-money ones. This is something we don't really want.
Presto, $8k not wasted.
[ Edit: you might respond "but this would discourage projects where we don't see the benefits" - and the answer is that this is only true if investors think you wouldn't see the benefits after the fact - in which case you, almost definitionally, don't want it funded anyway. ]
I don't think that's definitionally true—it's possible for an investor to worry they won't see the benefit after the fact, when actually they would see the benefit.
Sure, its possible that these proposed paragraphs would, for a specific project, cause investors to more poorly guess the outcome, but it is (imo) completely unreasonable to expect these proposed paragraphs would make investors guess worse in expected value.
It's like if you're guessing the sum of 3 dice rolls. You pretty much always would prefer to know 2 dice rolls over just 1, but there are specific situations where knowing 1 would cause you to guess better (e.g. if the dice rolls are [6, 1, 6], knowing 1 dice would cause you to guess 13 while knowing the first two dice would cause you to guess 10.5).
Oh, yes, I think the paragraphs definitely help—I think they actively counteract the possible issue where investors are uncertain/wrong about how the funders will evaluate.
What you're saying makes sense. Two reasons we're not doing that:
- First, one advantage of impact markets is supposed to be less use of grantmaker time, which is a big limiting resource in some charitable spaces. That's less of an issue here with eighteen projects. But ACX Grants got 500 projects last time. Writing something up on all of those would take more time than just judging them the traditional way!
- Second, even though this round was "predict Scott and Austin and a few other people's preferences", I'm hoping that a real impact market would have many potential buyers. Investors would use their pre-existing knowledge of what those buyers wanted rather than have each of the (potentially dozens) of end users weigh in on each project. At the limit, the set of potential buyers would just the population, and you could fund projects that you thought would be good according to anyone's values.
I think this point about multiple buyers is very important perhaps critical. On any time horizon buyouts are going to be subject to preferences of the grant maker. Investors can learn about those preferences but there is always going to be risk created from the mismatch. In markets, risk has to be rewarded so the grant maker has to pay investors extra for the risk of not knowing what the grant maker wants and that is dead weight loss. It doesnt achieve anything because its information the grant maker already had, but sacrificed through the process.
I dont think the solution is just knowing the grant makers preferences better. The solution is diversification. Diversifiable risk is not rewarded according to portfolio theory. So with enough buyers, the investor just asks is this good according to me, and gets enough buyers with preferences that approach or average out to their own, and the dead weight loss goes away.
Just let proposers or investors pay you (or other judges) a fee in exchange for your initial impression of the project.
I think that neatly solves both your objections.
Edit: To elaborate, the willingness to pay the fee should presumably be roughly equal to the expected value of the information to the investor. Hence, having a fee should roughly ensure you are only writing pre-paragraphs when doing so is most valuable.
I think the only way to make impact markets work would be to explicitly define what the criteria that will be used to evaluate the results will be in advance (e.g. prize money for inventing X or at minimum best at doing Y), otherwise your just doing normal charitable giving with a delay. Also I doubt a market will have significant impact (better than if you just gave the money on day 1) with less than a 5 year time horizon.
It seems undesirable for the final retroactive funding decision to have any sort of "future growth" component. The investors are already unicorn hunting, so the final decision should be based on concrete facts as much as possible. Otherwise, there'll be compounding forecasting error, plus the game becomes "predict which of these the judges will predict will grow the most", optimizing for hype over substance.
I agree with some version of this. Perhaps a longer time delay is necessary. That a panel of rationalists concluded that talking to people about rationalism was smashingly impactful without having to refer to final outcomes is parody.
This current round seems like a bet on what Scott and his friends will think is a good idea on an only slightly less apriori basis than the investors. On that basis it doesnt seem better than just letting Scott dole out the money. Well it saves Scott some due diligence I guess. He evaluates it after the class was conducted so filtered out potential hucksters. I can see why Scott would like that given his narrative of his grant program. As a long time consumer of great Scott output I can appreciate that, but it has less potential as a new paradigm.
I judged that getting a class of sixty people x 1 year was a very impactful result compared to what would have been my expectation when it was proposed (getting a few people to show up once, then drift off.)
As I said above, this is closer to selling during a Series B than selling during an IPO, but I wanted to run a test round in a short enough amount of time to learn something, and having to wait until all the educated people graduated and got careers and used their education wasn't compatible with the time schedule here.
Okay, I will re-evaluate under the possibility of multiple exit points.
A question under consideration is for project X, which takes a long time to succeed, but in the end everyone agrees is great, is there a reliable exit for its investors? I think maybe not. And that here only one project generated positive return (before adjustment) is a red flag.
If there is an early exit phase, we might imagine that for most projects we only learn if they got off the ground. Say the failure to launch is 50%. If the investors have perfect information about grant maker preference but no information about launch success, half the projects generate 2x, half generate 0% (+ returns for risk premia). If they have perfect information about launch success, then they all generate 1x. Here the donor is paying the investors to discover launch probability.
Now suppose one project also shows evidence not just of launch but a big success, then what happens? If the grant makers funds are limited, then the competition is relative and this project absorbs all the capital. In capital markets the best project might absorb funding from the "best" backers, but the investors in other projects retain their equity and the project can shop around for more backers of which there seem to be an indefinite supply. If the market functions, any project with positive EV can get funding. With charity it seems funds have to be finite, so the competition will always be relative.
What I think this means is that any tendency to have early exits biases projects towards those expected to show faster results. If half of projects are expected to show results before exit point, then investors only choose those, regardless of the EV in others because those are the ones likely to absorb capital in a relative competition. Investors would assess the grant size of each exit point and basket accordingly. If the launch is for covid solutions, that is all fine. If its for x-risk solutions, then you definitely want to weight the capital towards exit points much later. I think you saw a version of this above, where one panelist was super excited about the long term benefit of a project, but there was nothing yet to show for it. Investors would shy away from any project likely to resemble that.
Using capital markets as a comparator, I now agree that whenever your exit point is chosen you have to evaluate both on realized benefit (income) and expected value (growth) just as any equity is valued, though later exit points will have more income to assess.
This assumes we dont reach a point where enough charity donors just buy up old investor certificates and that this is anticipated, and that these types of projects rely on a pre commitment from donors.
(I didnt mean to be too snarky about the high assessment for that particular project, just it does look funny. But if a rationalist blogger didnt assign high value to spreading rationalist ideas what is he even doing)
Hypothesis of a bias toward proselytizing existing memes, potentially at the expense of other good work, seems disappointingly plausible with this data.
I sort of agree. This was a test run, part of the point of the test run was to iterate quickly, and part of "quickly" means that we couldn't wait for all the projects to bear final results. This was especially true for the "educate new EAs" project, where we know approximately how many new EAs were educated but not whether, say, some of them become professional biologists and invent a new vaccine twenty years later.
So I think I think of this as like an investor who buys in a seed round and sells in Series B (maybe this isn't a thing real investors do, I don't know). We bought completely new projects that were 100% potential. Now we're selling projects that are sort of developed, but which still have to be evaluated on some combination of results and potential. One day when there's a real impact market I hope it will last long enough to assess final results, and then people can trade beforehand in expectation of those results.
> But in reality: A lot of the retro eval is just “did they actually deliver the thing they said they wanted to do”…
None of the deliverables that I had promised (code, models, data, paper) are out yet, mostly due to my personal failure to execute. Nuño is one of the most excellent people I have ever worked with, but I kept getting busy or distracted with other things instead of getting anything done.
I totally overestimated how much bandwidth $540 can buy, and underestimated how strongly I would lean towards perfectionist paralysis in the absence of short feedback loops, which lead to me holding on to work-in-progress far longer than reasonable because I wanted to find time to polish artifacts before sharing them with anyone (even close collaborators). Naturally I never found time to do this and momentum crawled to a halt before my unreasonable quality thresholds were met.
Overall my update is towards accepting that I have a tendency to become excited about something, sign up for it, do a burst of great work in the very beginning, proceed to become excited about something else, and run off towards that while leaving the previous thing unfinished.
In the moment this can feel like you're continuously getting great work done, but when you look at the bigger picture, it's spread out too thinly to have much of an impact unless you are lucky enough to come across a problem which can be solved in the short interval of time where you actually work on it.
Since I am an ambitious person, I lean towards selecting important problems, most (but not all) of which are hard. The strategy that has brought me this far in my career is gambling on some percentage of problems actually being easy. So the majority of projects do not survive but I make up for it with the sheer bulk of participating in everything. My capital is primarily concentrated in intellectual resources, not financial assets, and being prolific allows me to diversify the investment portfolio. The victory condition is outlier success in the long-tailed distribution of impact. In other words, I instinctively sacrifice consistent performance in exchange for seeking upside risk, exposing myself to a wider range of outcomes.
Although I cannot speak to the optimality of this greedy r-selected strategy with respect to my incentive structure given the chaotic environment of technical AI safety research, on a personal level, I hate being like this. I would happily take a pill that made me more k-selected with projects I choose to work on (note that this is not the same as being more "focused"), because I would strongly prefer to at least be able to make credible commitments to myself — let alone to my collaborators.
One fundamental property that I struggle with is this determination and optimistic delusion that prevents me from quitting things long past the point others would have. Perhaps this is my way of preserving optionality, or an unwillingness to confront the shortness of life, but I genuinely believe I'll get back to things later and no amount of evidence to the contrary ever changes my mind. Even faced with an unambiguous calibration curve showing me that I'm overconfident, I continue to make the same error.
I don't believe this necessarily paints me in a good light, but my intention here is to be as honest as possible, which means highlighting that lying to yourself is also a form of deceptive behavior. Generally speaking, deception is unethical.
I will use a computer analogy because computers are what I know. An operating system lies when presenting a contiguous address space to a running process. In reality, the physical memory is fragmented. The operating system's lies make a scarce resource appear more abundant. Almost all modern programs expect to be able to allocate or deallocate memory freely, and tend to fail in uncontrolled ways when that expectation is not met. This rate of failure rises sharply when a system's operating load is dangerously close to its peak capacity.
Clearly, since my intuitive sensors report misleading information, I'm closer to the process than I am to the operating system. But I don't see that as an acceptable explanation for the significant delay. In fact, this absence of reliability is not something that is isolated to this particular endeavor. To my chagrin, missed deadlines are almost a routine event. I am aware of the differences between the patterns I follow and the instructions provided to a safety critical real-time embedded system, but awareness does not equate to automacity. It will take further reinforcement before that gap closes.
Taking the outside view, this problem has a simple solution. Impose firmer constraints. Enforce rate limits. Set lower standards. Explicitly drop things to free up spare capacity. Prioritize. But a portion of me resists, arguing that overly rigid structure will rob me of a certain spark. Freedom, learning, exploration, creativity, spontaneity.
I think doing this was a neat experiment and I had good reasons to compromise on its success at the plot points where I did. It's on the backburner for now, but I will still finish it. Not out of duty or obligation or pressure or stress but because I still find it interesting to indulge in and this is a missing piece in a broader goal I'm working towards.
It's a delicate balance to strike. I don't want to get into the sleazy salesman's habit of exaggerating words without backing them up with action. That would place me past the point of distrust. I am quite limited in many aspects, and earnest attempts do not weigh the same as results. Part of the motivation behind wanting better predictive capabilities is to be able to better manage the division of my energy across opportunities competing for my attention. Thankfully, I can do that using the faculties I already have. Sleep, exercise, write faster, and test earlier.
> Investors spent $33,220 to produce what we judged as $21,125 - $31,125 in charitable value, making the market inefficient. I don’t know exactly how to think about this: I wouldn’t want us to pay the investors much more than they invested, or I’d feel like we were being ripped off.
This is the mistake that leads people to support communism.
Your point of view is wrong and harmful. What you _should_ want is to end up paying the investors much, much more than they invested. You already decided what the project output is worth to you; you should see that it is a better world if $21,000 of charitable value costs $1,000 of monetary value to produce than if it costs $20,500. But your stated preference is for the second of those worlds.
(In slightly more detail, your position alllmost makes sense in the light of your stated commitment to spend no less than $20,000. In that scenario, an outcome you might want to ward off is that someone signs up, invests no money (good!), produces no results (bad!), and then you pay him $20,000, which is a lot more than he invested and is also a waste.
Except that you don't actually avoid that outcome with your rule. Where one person can sign up for free money for no effort, so can tens of thousands of others. When 40,000 people sign up for this deal, you end up paying each of them at most 50 cents, which is not much more than they invested and is therefore OK with you. The waste is still there, it's just as bad as ever, but your rule didn't see it.)
I think you make an interesting point, but I read the whole situation differently.
If the investors are grossly undervaluing the projects, then they're doing a bad job ~of predicting~. It's certainly good for the world—more value is being created than expected, which is great! But it's bad for the hypothesis that "investors can smartly allocate grant funds"—under-predicting is just as bad as over-predicting when the question is the efficiency of this prediction market, right?
And if grant investors are bad at predicting, then Scott is better off with old-fashioned grant-giving strategies. If Scott can decide as well as any "expert" which grants should be funded, there's no need for all this complexity.
In other words, Scott is getting "ripped off", not because extra value was created, but because if the experts aren't actually predicting anything he could've just been the original investor himself. (At least that's my interpretation. Just a random reader, definitely not an expert in this stuff.)
I think you're somewhat right, but that the more complicated factor is that it depends on the counterfactual.
I think most of these projects would have happened anyway even without the impact market. We just would have paid exactly what they cost, in the form of a grant. I agree that eventually if impact markets are to be useful, it will be because they encourage new projects, and that will be because we pay for them. But I think we're still at the "paying a little extra for projects that would have existed anyway" part of the growth cycle.
Next time around I'd like to see a less self-licking selection of projects, though. While I can understand the logic behind the idea that the most effectively altruistic thing you can do with your money is to raise awareness of effective altruism so that more people donate more money which we can use to further raise awareness of effective altruism... it still makes me uneasy.
It would be nice to see another experiment done with a more limited scope to exclude these sorts of self-licking projects.
Yeah, I was the dissenting judge for the Maryland EA project. No knock on it, I think it's great that it exists but also that it falls outside the scope of forecasting grants, and vaguely feel like it won a bunch of funding because other judges pattern-matched it to "oh this is what OpenPhil would be willing to pay".
I'm not sure what a good remedy here is though -- in practice having good judges and aggregation mechanism is pretty tricky and will massively influence the final evals, and I think Scott's proposal of "pick 5 reasonably well informed people and take the median" was pretty good given the time constraints. And there's also an argument like "all of this is Scott's money anyways, so it's really up to him to allocate it however he likes". If you want to see different projects and are in a financial position to put up your own retroactive bounty pool, we'd be happy to set up an impact cert round for you!
What I'd like to do next time is get the EA Infrastructure Fund (a charity which exists to help promote EA) on board, so that all good help-EA-expand impact certs will be bought by a charity that's clearly earmarked for that purpose and nobody will be surprised.
I remember that when I was reading the descriptions of the projects, a lot of them seemed to (implicitly) go along the lines of "I was going to do this good thing anyway, but now that these markets are open, I might as well get paid by telling people it requires $5000."
Overall I feel this was a win. I'm glad the organizers feel like there was room for improvement, but as an investor, I was really glad I participated. I don't give almost anything to charity normally, but the format of this as a market, where I had to exercise my personal best judgement as to returns, act as an advisor to projects when desired, and potentially even make money, helped pull me in. I was very OK with not making any money---indeed, when I saw the unadjusted table first while scrolling through this article, my thought was "Oh well! At least I have some validation that I know how to pick more-impactful projects".
OPTIC seems like a huge success to me, and it was indeed great connecting with Saul and the team. The fact that they were proactive about reaching out to their investors for advice and feedback was a really good sign; I would encourage more projects to do that. (I think if I were a real VC-style impact investor, I would myself reach out and schedule meetings with my investees. But I'm a busy guy. So big kudos to them for taking up that part of the investor-investee relationship!) As part of that relationship, I was also happy that I was able to connect them with my coworker at Google who maintains our internal prediction market; he was able to take a train up to Boston and give a talk there: https://www.youtube.com/watch?v=pYYuMLQjpWY. This kind of incentives-aligned networking with your investor seems good!
My main criteria for picking were:
* Would Scott like this? In particular, this led me to the Kelly-optimal bets tool, and the superforecaster predictions. The latter, in particular, seemed to play well with a lot of the themes we see on ACX, comparing superforecasters with prediction markets and other prediction technologies. I was also interested in https://manifund.org/projects/predicting-replication-in-academic-economics from that aspect, but I wasn't fully convinced by the answers to my queries in the comments there.
* Would this have a big impact if successful? From this perspective OPTIC and Crystal Ballin' seemed high-leverage. The risk here is whether the work actually gets done; I tried to suss this out with Crystal Ballin' but it seems to have only partially panned out. Still, I remain hopeful for them, albeit not on the original timescale.
If people are interested in what kind of projects I investigated, even if I didn't invest, check out https://manifund.org/Domenic for my comment history.
The biggest areas for improvement, I feel, are in communication, both around the whole structure and for newbie investors/investees in particular.
Because you're dealing with something novel, it helps if the rules are really clear. Who exactly are the judges? It would be good to know their backgrounds and preferences ahead of time. Are they supposed to be judging actual realized impact at the 6-month mark, or future potential impact, or...? What exactly is the deadline? What exactly is the funding pot? ("Between $20,000 and $40,000" is a huge range!)
And for newbie investees/investors, I think there are just some basic market mechanics that could serve to be more signposted. E.g., the fact that we collectively bet $45k on a $20k-$40k pot should have generated big warning signs or something. The fact that a project asking for $20k (between 50% and 100% of the total pot) was unlikely to get funded should have been clearer. (Both of these could be addressed in the UI, ideally! Otherwise, putting it in some sort of pre-reading would work.) You could also try to help investors understand better what sort of things to watch out for, e.g. impact/tractability/neglectedness (seems like this would have caught the Polymarket case), investee track record (what many of my questions were focused on), over-ambitious vague projects, etc.
Finally, I told Austin this a while ago, but... where are my actual impact CERTIFICATES!? I want something pretty to print out and brag about to my friends! I want something where selling it back to Scott feels like parting with a small piece of my charitable soul! I want something where I stare at my 66.12% share for OPTIC, maybe in pie-chart form, and think... how could I get the other 33.88% percent?
Thanks for the thoughts, Domenic! I'm glad our round nudged you to think a bit about charity allocations, and congrats on the successful OPTIC bid (I recall putting in an investment offer myself but I think you edged me out :P)
Also really appreciate the feedback - will think about UI improvements to make these market mechanics clearer for next time. A big reason to do ACX Minigrants at all was to discover where these issues are, so we can fix them before trying more ambitious rounds. And fair point on the actual shiny CERTIFICATES as well!
I know that this sort of post is less 'sexy' and seems to draw less appreciation than most others, so I just wanted to say that I think this is a fascinating and potentially very useful experiment, and I hope it keeps going and you keep posting about it here.
As one of the investors (invested $2k, now worth $1489), and someone who has raised a total of $85M from VCs in the real markets, I think you are way too pessimistic about this outcome. I feel like it was a huge success (9/10).
- you got $31k in value created for $31k without having to take any risk yourselves. Some projects did better than you expected! And you get to pay them more! Some did worse and you didn't have to pay them at all! Incredible! (Aside: this benefit to you is why it's ok for investors to make big returns; taking on risk and investing in a wide field is a highly valuable activity.)
- "we are skeptical of investors choices" is not a failure, it's a success! Think about how much investment goes into stupid startups that you are skeptical of; that's just how it goes. Sometimes though, you are wrong, and it's good that other people have different opinions, and we get Airbnb. As long as all the good stuff gets funded, it doesn't matter how much bad stuff other people waste money on. Relatedly...
- I would argue everything happened correctly with Base Rate Times -- the founder stated that they would not do the project unless they had a certain grant (valuation) that neither we nor you would have funded at that level (and in fact was not worth in the end). The failure here was the founder's pricing of their idea, stated preferences vs revealed preferences (i.e. do it anyways), or misunderstanding of how to correctly keep 100% equity but still participate in the round. In a real open impact market they would have gotten $6k at the end anyways, and it's only because of the constraints of this simulated market that they didn't (the requirement to raise any money). Plenty of startups are bootstrapped and then exit, and this is good.
- the real VC market has the exact same dynamic (actually spookily similar) of not only a power law for startup exits, but also a power law for VC returns -- most lose money, some roughly break even, but the top ones make insane returns. Everyone knows this and wants to participate anyways because _they_ think they can outperform the crowd.
- I don't think you should worry about overall investor returns at all in fact, because a) we were considering this charity in the first place and willingly overspent given that (I aimed to invest at EV rather than aiming to actually make a return, and didn't pay attention to total amount invested by others at all -- reasonable given the bidding structure), and b) the market will adjust naturally based on payouts, especially as the big losers one round forgo the next round. This is all natural and correct. If you keep paying out at the end, investors will find a balance where they do on average make money.
- Even overpaying for securities you and Austin especially like is reasonable! This happens all the time IRL for exits; it only takes one entity thinking it's worth it to buy a company at a valuation no one else believes in. IRL the top five buyers don't pool their money and take the median valuation -- the highest valuer just buys it. You two are not being irrational to shell out more for things you think are worth it.
- You're selling yourself short on the market structure. This was actually quite investor friendly. Usually only the highest-paying investor gets to participate in a startup, they have to wait 5 years for any payout, and they have no idea if in 5 years the startup will IPO into a bear market. It is weird for the total funding not to reflect the actual value creation, but actually you did adjust for this by having a floating total in the 20k-40k range, and I think you did a good job valuing stuff. In a real market the dynamic would be similar I'd think, where charities can buy impact certificates OR make grants, and do more of one vs the other depending on how good the certificates are (thereby adjusting the total end payout). (I agree that the round fails for long term projects like the golf ball one, though; that maybe needs 5 years.)
The one and only reason this wasn't a perfect 10 imo is that BRT/AI treaty didn't get an award at the end, because they were ineligible. In the real market, bootstrapped companies are just as eligible for exits.
Thanks, it's great to hear the perspective from real investors (which I was only LARPing as) and your reflections make me feel much better about some of our market design choices. I do wish we had made BRT eligible at the end, and I definitely plan on changing this for future rounds (eg the one that Manifold natively sponsors)
As Austin mentioned, he wants to try something much more ambitious and real-stock-market-like. I'm much more cautious and he's going to humor me for the rest of the year, but next year I'll humor him and we'll try something bigger (hopefully not with my money or reputation on the line)
The cynic in me can't help but feel conflicted about the only project that the judges valued as a standout success being "educate impressionable young people on our values". (A less charitable verb would be "indoctrinate"). Pretty much every talking point about wokeness in academia being a liberal brainwashing camp could be used here with very little modification.
Like, people already accuse EA of being a cult that spends more money on "paying middle class intellectuals to think about how to do charity" than actual charity, and having "gain new adherents to the philosophy" be your highest ROI in a fundraising contest is... not great optics? Especially when the goal is to sell the "impact" to corporations for pure-optics-reasons.
I don't even disagree with the ROI assessment ($300 is really low, and $10k isn't unreasonable), but if the goal is to sell EA benefits to non-EAs, again from a purely optics-position (which matter more to non-EAs than actual outcomes) that seems to miss the mark.
Yeah, and as a vaguely-rat-adjacent non-EA, when I met some of these kids I was not super impressed with how educated they had been. It felt like they'd learned to love the right sources/buzzwords/topics, but not particularly how to think any better.
(If anything, I would say they learned how to think *worse,* as the more they believe in any curriculum as gospel the less they are thinking for *themselves.*)
Of course from the EA perspective they're a win because some of them will become good EAs, I'm sure, (at least for some years if not for a lifetime), but from a 'raising the sanity waterline' perspective - which is what one of them said to me the education they were getting was all about - well, I don't think it turns out that's compatible with teaching people what the right answers to questions are.
You're urging me to distort a financial market for optics reasons! That would break the moral fabric that binds the universe together!
FWIW, I don't really believe it's possible to "indocrinate" EAs. To a first approximation, people either immediately get angry about it and bounce off, or they say "yes, this is the thing I've been groping toward my whole life". The best you can do is convince them to stay in a room long enough to notice whether it's the thing they've been groping towards their whole life or not.
Also, I don't think it's so much that we decided education was the best thing, so much as that Max did an exceptionally good job and sold his impact certificate for much cheaper than it should have been worth. Manifolio got almost as much money as Max, probably for about as much work, they just priced their certificate higher.
Yeah, I felt kind of bad pointing this out, since ultimately I don't think you really did anything *wrong*. At least, I don't have any obvious advice on what you should have done differently. But I think it needed to be said, since optics are a real thing that gets priced into many goods.
> thinking in these terms makes all education/movement-building grants vastly better than all other types of grants.
I have an (experimental! warning! careful in taking this at face value!) paper exploring this, which concludes that, in some cases, you should do an "asymptotic ponzi" and invest asymptotically everything into movement building:
I agree this makes intuitive sense (it's pretty cheap to talk to people, so it would be surprising if education didn't make your movement grow faster than your discount rate). I'm still viscerally uncomfortable with it. Partly this is just an emotional "ponzi schemes seem bad" thing, but if I had to put it on a firmer footing:
- I don't know of very many groups that have grown huge through a process of recruiting new people, then using their money and labor to recruit even more new people, and so on forever. Arguably early Christianity, Mormonism, and a few others sort of accomplished this, but there must be many more who would like to (political factions, religions, cults, etc) so it seems surprising that they failed. Maybe that means this doesn't work so well?
- We will do a better job recruiting if we have policy victories to point to, and useful positions to offer new recruits besides just becoming another recruiter.
- I worry that recruiting dilutes culture (in the sense of "good corporate culture", everyone knowing what they're doing and having good values), and engagement with the world replenishes culture, and that having too much of a recruit:engagement balance leaves you with a culture that isn't good enough to use the new recruits effectively.
I don't know if these are my real objections or just a patch over my philosophy to prevent what feels like an infinite-money loophole.
You should also not underestimate that there is a limited pool of potential EAers. Maximizing the pool is probably more important than fishing it dry. Being too focused on recruitment over action will greatly diminish your pool.
Being a member of the 'EA tribe' is inherently neutral. The value comes from average X% of EA tribe members doing things with average value Y. You could do a dirty estimate of value of the entire EA tribe of N people as N*X*Y. Increasing N (your recruitment projects) is positive EV if X and Y remain constant or decrease less than N increases. As any money spent on N could have been spent on X (internal culture alignment) or Y (actual charity work) instead, you need to have even higher confidence that increasing N doesn't decrease X or Y. I think this probably acts as a reasonable backstop to the infinite-money problem.
(it's fun to think about the second order effects - does increasing Y passively drive an increase in N through good publicity? Does increasing X make Y more resilient to increases in N, etc)
Quite a track record indeed, even if one considers Byzantium proper starting with Justinian. No offense intended, anyway. Just an association I should have kept unwritten here.
Having just finished Kaldellis' "The New Roman Empire" I apologize once more. Somehow Eastern Rome had mostly been presented as a failure to me before.
Personally: I'd be itchy calling these charitable.
Would like to see some that benefit people who aren't well-off. I guess I can understand if people want to invest in gigabrained crypto applications that could replace the dollar!!! and not apps that make third-worlders' lives easier, but you have to admit we're snorting clouds over here.
Did you read the post? There weren't any first-world crypto-related projects - the only one with any crypto connection was supposed to help poor villagers in India (and it lost money because we were skeptical of it).
More to the point, because this was a test round, I asked that we only consider forecasting-related projects. I think it's potentially charitable to donate to development of a science/methodology/community, but I'm not claiming this was the best possible use of money, just the easiest contained community to run a test in.
"This was a hard grant to value. One way to value it is something like: suppose that he keeps doing this x 3 years, and 5% of his students become long-term committed rationalists/EAs. That’s 9 new committed rationalist/EAs. Suppose half of those would have counterfactually found our community anyway; that’s about 4 new ones. Suppose of those four, one takes the GWWC pledge to donate 10% of lifetime income, and another goes into direct work and has a good career in some useful institution. Each of those people could plausibly generate $100K in charitable value. So maybe we should value this at $100K."
This is the whole problem with the approach - you're just making up numbers, or picking them out of a hat. Not only don't you have a clue if these numbers are right, they're not really testable either (unless you add a decades-long tracking of the participants, and even then their truthfulness might be in doubt). The value might as well be $0 - in fact, this might be the likelier number. The whole evaluation is empirically empty.
Sort of agreed, but this is the same problem any grantmaker faces - they have to determine whether to give a project money or not, even though vague benefits like education and happiness are impossible to value. We're not making this problem any worse, and we're sort of making it better, in the sense that we were able to value it after six months rather than immediately, and so we know that it at least succeeded in getting students (which a normal prospective grantmaker wouldn't have known).
We do then make the problem higher-stakes by making investor money payback contigent on our valuations. I'd like to solve that by having investors understand my value function - I have a history of giving grants, so would-be investors can know what I like and how much I tend to pay for different categories of things. In the future, I want big charities with long track records to join in, and then this will be on even firmer footing.
It's one thing to not be able to put numbers to things that are inherently extremely difficult to know, it's another to put purely made-up numbers on it and pretend you now know something. The latter seems like a recipe for false confidence.
Given that population numbers and the concomitant impact on the sustainability of the earth for numberless future human beings wouldn't it be a valid longtermist goal to bring human numbers in alignment with sustainability? They seem to be pinning their hopes on an assumption that we can just tech our way out of this. Which I suppose would have huge appeal to the "tech bros".
I think there's a division on whether we should be funding decreased population (to solve the environmental crisis) or increased population (to solve the fertility collapse / economic crisis).
I tend to think there is no environmental crisis related to overpopulation, in the sense that there are no current famines not related to political issues (ie we can grow the food and transport the food when warlords don't prevent us from distributing it), plus the history of things like the Simon-Ehrlich wager, plus the fact that non-immigrant population is set to decline on its own everywhere except Africa, and Africa is expected to stabilize soon. I'm also concerned that "fight overpopulation charities" have a really scary history (see https://www.astralcodexten.com/p/galton-ehrlich-buck ) that makes me want to run away screaming.
So I'm not currently funding any population-reduction charities in particular, although I am funding some more generic environmental/sustainability projects. I haven't funded any increase-population-charities yet either, mostly because I haven't found ones I like, although some of the IVG related charities I fund might do that as a side effect.
None of that matters here, because we only funded forecasting charities, to limit the complexity of this test round.
[I went to UMD for undergrad, as did a different Matt G; I wouldn't be surprised if there are more of us floating around the rat/EA sphere.]
UMD is a party school--but also it's the flagship university for the state of Maryland, and gets a lot of top students thru scholarships and being well-ranked for lots of prograsm, has a bunch of investment in undergraduate research, and so on; I am not surprised that one could put together a solid EA group there.
I feel like the median here is potentially awkward. Roughly speaking: six months ago, a bunch of people were asking "what will this be worth in six months?" and the project sold certificates to the person who gave the highest answer. And today, five people are asking "what's this worth today?" and the investors are selling certificates according to the median answer of that group. If the five people were buying individually (and could afford to buy everything they wanted), 9/18 of the projects would have made money. (Five had 1/5 valuing it higher than total equity, three had 2/5, one had 5/5, and nine had 0/5.)
I assume "a group of people decide among themselves what to invest in using some way to aggregate their opinions" also happens a lot in the stock market and VC, so I don't think this is a bad way to do things. But it stood out to me.
Noob questions: I couldn’t find a clear explanation. 1. What does having an “impact certificate” mean? Is it about the impact of the entire project for its entire duration, potentially indefinitely, or just for a specific part of the project? 2. Can the project owner issue new shares after certain milestones are met?
This is such a fascinating experiment in applying market dynamics to charitable impact! It’s impressive to see how innovative ideas like these are taking shape and pushing boundaries - who knew forecasting and charity could come together in such a "winner-takes-all" format? I especially loved seeing projects like Max Morawski’s workshops at UMD and OPTIC making waves—what a testament to the power of education and community building. Even though not every project paid off for investors, the learnings from this trial run feel invaluable for refining the model.
The comparison to venture capitalism here is spot on - sometimes it's about betting on people, not just projects. I'm excited to see where version 2.0 goes, especially with the idea of involving established charities to make the system more "real." It’s clear there’s a ton of potential here, and this first iteration shows just how creative the charitable ecosystem can get. Looking forward to more updates on this, and congrats to everyone involved!
This is amazing! Very cool that it worked to fund real projects that were interesting to read about.
Hopefully the first of many rounds of impact certs.
Congrats Austin, Rachel, and Scott!
Overall, I share Scott's view that this round went fine but was not a resounding success (I might have rated it a 6/10 fwiw), and have many thoughts about how to make our future rounds better. Chief among them: open up trading to nonaccredited investors via mana/charity dollars; do more frequent retro evaluations; provide more feedback and support for grantees.
Beyond the large ACX Grants round at the end of the year, we plan on a few other experiments with certs including one sponsored by Manifold specifically for Manifold community projects, and perhaps an essay contest with ChinaTalk backed by impact certs. If you'd like to sponsor an impact cert-based contest for your own org (for-profit or charity), reach out to austin@manifund.org!
Shoutout to the other 3 judges (Nathan Young, Marcus Abramovitch, and Drethelin) for taking the time to evaluate the retro grants. My own scores and retrospective notes are here (I'll likely clean them up and publish soon): https://manifoldmarkets.notion.site/Austin-s-Judging-Scores-54ff88af184441f9be684838c56112c5?pvs=4
And of course, a huge thanks to Scott for believing in impact markets, working with us to figure out how they could be set up, promoting this effort and funding the whole thing! None of this would have happened without Scott pushing for it~
A few other quick notes:
- I think the $8k invested in subsidizing Polymarket markets skews the metrics of this round a lot; of that, $7.4k or 93% was invested by a family member of the founder, which I'd argue is much closer to "founder keeping equity" than "external investor deciding the project was good". Removing that brings the total invested to $26k invested for ~$21-$31k of impact, which seems pretty reasonable.
- I'm not sure what Marcus is smoking but imo $50k for figma mocks and some backend glue code is crazy. I could see a slightly better case if they'd actually shipped a working prototype, and a much better case if people were actually using the thing.
- I am a bit disappointed by the investor's choice to pass on some projects, eg Daniel Reeves's small-group futarchy proposal: https://manifund.org/projects/implement-a-prognootling-app. Perhaps I should have funded this myself when it wasn't on track to hit it's funding bar...
- In general, the results do update me towards "for early stage projects, a YC-like entity paying a lot of attention to project selection may be better than a competitive marketplace system". But I think the accredited investor filter may have also cut out a lot of good capital allocation, and I'm hopeful that we can figure out a solution here.
Marcus here. I can explain my thinking here.
I try much more to evaluate the worth of the result as opposed to what I think it would cost to construct what was done. This is why I gave a lot of projects a score of $0. This is well recognized in the for-profit space where some companies are valued 100x or more of what VCs invested in them and often 0 but this doesn't seem to be the case in the non-profit world (yet).
For Devansh, I think he successfully accomplished step 1 of creating a viable and useful impact certificate market that solves some problems (only one impact cert per project, people actually coming in and retroactively funding the work, etc.). If successful, I think this is worth millions of dollars in improved outcomes in the non-profit space and thus getting past the first few obstacles makes it worth a lot more. Call this a 5-10% chance of the worth of the project being $>1M
Appreciate you sharing your thinking! My brain finds "100% chance of $50k" ludicrous but "5% chance of $1m" kinda reasonable, actually. Though, as Scott alluded to, it seems like Devansh's project is still mostly predicated on future impact, as opposed to current results -- a bit contrary to the goals of doing a retroactive impact analysis.
"If the project is partway finished, judge it based on its current state and what that suggests about its future results. For example, if a project is 95% done, that's less good than it being 100% done (because it still might fall apart later) but still good evidence that it will eventually be completed, so probably you should give it a score slightly lower than if it was completely done."
I took this to imply that I should judge it based on the current progress as well. If I were to be more clear, I think it is like 10% of the way to something worth several million but I discount that a lot due to risk, time discounting, etc.
I still struggle to understand how to balance the impact / earnings point here.
It seems very unlikely that these markets will return better (or even comparable with) simple capitalistic markets if I wanted to make money.
If I wanted to donate to charity / have a direct impact, my marginal tax rate is c.50% so I can essentially double my money by donating directly to a charity (this isn't quite doubling, as the value of the services funded by my taxes is a lot higher than zero, albeit a lot lower than e.g. malaria treatment).
This is, I acknowledge, a general critique of impact investing.
This seems exclusively useful for (and maybe this is in fact the entire point, albeit I see the potential scale here as small)
1) getting something off the ground that wouldn't reasonably qualify as a charity but which you think is socially useful; or
2) Harnessing investor time/expertise to assess projects and outsourcing that from grantmakers.
This tax rate issue is a good point. Is there anything that can be done here? Maybe not soon, but is there a path toward eventually getting impact markets categorized such that I could deduct such investments from my taxes?
Probably there' some tension here between the investment vehicle side and the charity side. But I dunno, it seems like it would be a pretty pro-social outcome if impact markets were a tax-advantaged way of making money over capital markets.
The issue with tax exemption is that it would require the government to opine on what constitutes 'impact', which is currently much broader than what constitutes charity; the net effect may actually be to sideline the novel, interesting ideas that impact markets are supposed to enable.
The other issue with the tax exemption is that it's too easy to turn this into a tax evasion scheme.
I think no. Suppose that you break even on an impact market, so you end up with exactly as much money as you started with. How should this affect your taxes?
If you profit on an impact market, I think this is normal capital gains. It would be neat if Congress changed that, but that sounds like the kind of thing that would take a twenty-year, multi-million dollar lobbying campaign.
The part I've been thinking about more is the final oracular funder's perspective. I think if these are charities, then they don't pay taxes anyway and it doesn't matter, but I'm not sure.
My first stab would be be, you would get a tax deduction equal to the amount you invested, and then capital gains tax on the money you get back. So it would effectively convert your tax on that money from your normal tax rate, to a capital gains tax rate. Yes, this is a hack to lower your tax rates, but it seems to be within the bounds of the sort of thing we already have? Like capital gains taxes existing at all, or charity deductions existing at all; both are ways of lowering your effective tax rate as long as you do something "virtuous" (investment/charity).
You can imagine exploits here, if someone spins up a fake charity or fake impact valuations. But I think societies already have some good experience prosecuting tax fraud, so like many legal areas, this probably doesn't need to be mathematically foolproof.
Note that my first-stab proposal above has the property that it encourages impact investing regardless of how good the investor is at predicting impact; the idea is that even if you gave the money to a zero-impact grant, you were doing it charitably, so the government should give you the deduction. Maybe we'd instead want a system that incentivizes *impactful* charitable giving? I'm not sure though; there's no impact-test for existing charitable donations, and you get tax benefits from investment capital *losses*.
Anyway, I agree this is probably mostly theorycrafting. Although it'd be nice to think our society could figure out improvements to the current tax code that encourage good things, the fact that prediction markets are still struggling with legality gives me little hope for impact markets getting any legal benefits.
I think the tax implications here are worse than you suggest. Suppose there's a charity which will do what it says with certainty: for example, some homeless shelter that always shelters the same number of homeless. You, a rich person, invest 100% of your money in impact certificates for that shelter; at the end of the year, funders pay you back exactly the same amount you put in, eg all your money. Now you don't have to pay any taxes. Why wouldn't everybody do this, so that the government couldn't collect any tax at all?
I assume by "all your money" you mean "all your income", not "all your preexisting wealth"? (Since wealth is not currently taxed anyway.)
In my model you would pay capital gains tax on the money you got back. So you've basically converted income tax into capital gains tax. Given that rich people's income is ~all capital gains already, this seems fine?
Sorry, I didn't understand that part. I still think this is problematic. Capital gains tax is ~20%, income tax is ~35% at high brackets. There are many people who make high incomes who would like to pay 20% instead of 35% on those high incomes - is there anything that would stop them?
It seems like the requirement for the founder to set the price is difficult here, and lead directly or indirectly to some of the failures here. Notably, Max ended up selling 100% of his project at what was (in hindsight) too low of a price, and the Base Rate Times didn't raise any money because the founder overestimated what investors/funders would be willing to pay.
In practice, the way VC markets work is closer to a fixed VC/founder equity ratio, with the amount of equity sold typically close to 10% per round across a wide variety of businesses. The _price_ of that equity is set by some combination of market demand for the equity and the expected costs of the startup over the next ~2 years until another fundraising is needed.
This is a result of some apparently-irrational behaviors, such as founders being reluctant to raise valuation even in the face of high investor demand, and investors being reluctant to buy "too much" equity for fear of reducing the incentives on the founder to succeed.
I wonder whether similar effects would be useful in an impact market. For example, if every participant was always required to distribute 75% of the equity for their project and keep 25% as "incentive", and the opening auction solely served to set the price for that 75%, which the founder has the binary choice to take or not.
I wouldn't describe the behaviors as irrational. There are a lot of risks to raising the valuation (down rounds and cap table resets). It's also harder to hire employees if you have an inflated valuation (since part of the promise is that they will see a growth in their equity). You already pointed to the problem from investor side about reducing the founder incentive (this is the biggest investor risk).
(I worked on this)
The last part of what you said is pretty close to what happened: the auction solely served to set the initial price, and the founder had the binary choice to take it or not—they just set their bar for taking it in advance, as a minimum valuation, and the Base Rate Times guy set it very high.
It's possible though that the auction didn't price set very efficiently because there weren't very many investors and they weren't that profit-maximizing, though as Austin said in his comment, we have ideas about how to do better next time on that.
I do think having a default (or forced) equity option is good though, and we're interested in doing that for future rounds! Insofar as it's in our control. Also thinking something like 75/25.
> the fact that we overall disagreed with funding decisions is pretty damning.
I feel there is an obvious solution here: have the funders write up a short initial impression on each project. This reduces risk for investors and lets them focus on the value they're actually bringing: predicting outcomes rather than predicting what judges will think about outcomes.
Like, much waste could have been avoided if something like this had been written *beforehand*:
> Polymarket - We have trouble figuring out why this would be better than the many other real money prediction markets that happen on Polymarket all the time, and we feel like we already have good data on how real-money markets compare to play-money ones. This is something we don't really want.
Presto, $8k not wasted.
[ Edit: you might respond "but this would discourage projects where we don't see the benefits" - and the answer is that this is only true if investors think you wouldn't see the benefits after the fact - in which case you, almost definitionally, don't want it funded anyway. ]
I don't think that's definitionally true—it's possible for an investor to worry they won't see the benefit after the fact, when actually they would see the benefit.
> I don't think that's definitionally true
I did say "almost".
Sure, its possible that these proposed paragraphs would, for a specific project, cause investors to more poorly guess the outcome, but it is (imo) completely unreasonable to expect these proposed paragraphs would make investors guess worse in expected value.
It's like if you're guessing the sum of 3 dice rolls. You pretty much always would prefer to know 2 dice rolls over just 1, but there are specific situations where knowing 1 would cause you to guess better (e.g. if the dice rolls are [6, 1, 6], knowing 1 dice would cause you to guess 13 while knowing the first two dice would cause you to guess 10.5).
Oh, yes, I think the paragraphs definitely help—I think they actively counteract the possible issue where investors are uncertain/wrong about how the funders will evaluate.
To me this makes the most sense as well.
What you're saying makes sense. Two reasons we're not doing that:
- First, one advantage of impact markets is supposed to be less use of grantmaker time, which is a big limiting resource in some charitable spaces. That's less of an issue here with eighteen projects. But ACX Grants got 500 projects last time. Writing something up on all of those would take more time than just judging them the traditional way!
- Second, even though this round was "predict Scott and Austin and a few other people's preferences", I'm hoping that a real impact market would have many potential buyers. Investors would use their pre-existing knowledge of what those buyers wanted rather than have each of the (potentially dozens) of end users weigh in on each project. At the limit, the set of potential buyers would just the population, and you could fund projects that you thought would be good according to anyone's values.
I think this point about multiple buyers is very important perhaps critical. On any time horizon buyouts are going to be subject to preferences of the grant maker. Investors can learn about those preferences but there is always going to be risk created from the mismatch. In markets, risk has to be rewarded so the grant maker has to pay investors extra for the risk of not knowing what the grant maker wants and that is dead weight loss. It doesnt achieve anything because its information the grant maker already had, but sacrificed through the process.
I dont think the solution is just knowing the grant makers preferences better. The solution is diversification. Diversifiable risk is not rewarded according to portfolio theory. So with enough buyers, the investor just asks is this good according to me, and gets enough buyers with preferences that approach or average out to their own, and the dead weight loss goes away.
Just let proposers or investors pay you (or other judges) a fee in exchange for your initial impression of the project.
I think that neatly solves both your objections.
Edit: To elaborate, the willingness to pay the fee should presumably be roughly equal to the expected value of the information to the investor. Hence, having a fee should roughly ensure you are only writing pre-paragraphs when doing so is most valuable.
I think the only way to make impact markets work would be to explicitly define what the criteria that will be used to evaluate the results will be in advance (e.g. prize money for inventing X or at minimum best at doing Y), otherwise your just doing normal charitable giving with a delay. Also I doubt a market will have significant impact (better than if you just gave the money on day 1) with less than a 5 year time horizon.
It seems undesirable for the final retroactive funding decision to have any sort of "future growth" component. The investors are already unicorn hunting, so the final decision should be based on concrete facts as much as possible. Otherwise, there'll be compounding forecasting error, plus the game becomes "predict which of these the judges will predict will grow the most", optimizing for hype over substance.
I agree with some version of this. Perhaps a longer time delay is necessary. That a panel of rationalists concluded that talking to people about rationalism was smashingly impactful without having to refer to final outcomes is parody.
This current round seems like a bet on what Scott and his friends will think is a good idea on an only slightly less apriori basis than the investors. On that basis it doesnt seem better than just letting Scott dole out the money. Well it saves Scott some due diligence I guess. He evaluates it after the class was conducted so filtered out potential hucksters. I can see why Scott would like that given his narrative of his grant program. As a long time consumer of great Scott output I can appreciate that, but it has less potential as a new paradigm.
I judged that getting a class of sixty people x 1 year was a very impactful result compared to what would have been my expectation when it was proposed (getting a few people to show up once, then drift off.)
As I said above, this is closer to selling during a Series B than selling during an IPO, but I wanted to run a test round in a short enough amount of time to learn something, and having to wait until all the educated people graduated and got careers and used their education wasn't compatible with the time schedule here.
Okay, I will re-evaluate under the possibility of multiple exit points.
A question under consideration is for project X, which takes a long time to succeed, but in the end everyone agrees is great, is there a reliable exit for its investors? I think maybe not. And that here only one project generated positive return (before adjustment) is a red flag.
If there is an early exit phase, we might imagine that for most projects we only learn if they got off the ground. Say the failure to launch is 50%. If the investors have perfect information about grant maker preference but no information about launch success, half the projects generate 2x, half generate 0% (+ returns for risk premia). If they have perfect information about launch success, then they all generate 1x. Here the donor is paying the investors to discover launch probability.
Now suppose one project also shows evidence not just of launch but a big success, then what happens? If the grant makers funds are limited, then the competition is relative and this project absorbs all the capital. In capital markets the best project might absorb funding from the "best" backers, but the investors in other projects retain their equity and the project can shop around for more backers of which there seem to be an indefinite supply. If the market functions, any project with positive EV can get funding. With charity it seems funds have to be finite, so the competition will always be relative.
What I think this means is that any tendency to have early exits biases projects towards those expected to show faster results. If half of projects are expected to show results before exit point, then investors only choose those, regardless of the EV in others because those are the ones likely to absorb capital in a relative competition. Investors would assess the grant size of each exit point and basket accordingly. If the launch is for covid solutions, that is all fine. If its for x-risk solutions, then you definitely want to weight the capital towards exit points much later. I think you saw a version of this above, where one panelist was super excited about the long term benefit of a project, but there was nothing yet to show for it. Investors would shy away from any project likely to resemble that.
Using capital markets as a comparator, I now agree that whenever your exit point is chosen you have to evaluate both on realized benefit (income) and expected value (growth) just as any equity is valued, though later exit points will have more income to assess.
This assumes we dont reach a point where enough charity donors just buy up old investor certificates and that this is anticipated, and that these types of projects rely on a pre commitment from donors.
(I didnt mean to be too snarky about the high assessment for that particular project, just it does look funny. But if a rationalist blogger didnt assign high value to spreading rationalist ideas what is he even doing)
Hypothesis of a bias toward proselytizing existing memes, potentially at the expense of other good work, seems disappointingly plausible with this data.
I sort of agree. This was a test run, part of the point of the test run was to iterate quickly, and part of "quickly" means that we couldn't wait for all the projects to bear final results. This was especially true for the "educate new EAs" project, where we know approximately how many new EAs were educated but not whether, say, some of them become professional biologists and invent a new vaccine twenty years later.
So I think I think of this as like an investor who buys in a seed round and sells in Series B (maybe this isn't a thing real investors do, I don't know). We bought completely new projects that were 100% potential. Now we're selling projects that are sort of developed, but which still have to be evaluated on some combination of results and potential. One day when there's a real impact market I hope it will last long enough to assess final results, and then people can trade beforehand in expectation of those results.
I was project creator for #15, the interpretable AI forecasting thing, AMA!
I think this part from Austin's notes applies:
> But in reality: A lot of the retro eval is just “did they actually deliver the thing they said they wanted to do”…
None of the deliverables that I had promised (code, models, data, paper) are out yet, mostly due to my personal failure to execute. Nuño is one of the most excellent people I have ever worked with, but I kept getting busy or distracted with other things instead of getting anything done.
I totally overestimated how much bandwidth $540 can buy, and underestimated how strongly I would lean towards perfectionist paralysis in the absence of short feedback loops, which lead to me holding on to work-in-progress far longer than reasonable because I wanted to find time to polish artifacts before sharing them with anyone (even close collaborators). Naturally I never found time to do this and momentum crawled to a halt before my unreasonable quality thresholds were met.
Overall my update is towards accepting that I have a tendency to become excited about something, sign up for it, do a burst of great work in the very beginning, proceed to become excited about something else, and run off towards that while leaving the previous thing unfinished.
In the moment this can feel like you're continuously getting great work done, but when you look at the bigger picture, it's spread out too thinly to have much of an impact unless you are lucky enough to come across a problem which can be solved in the short interval of time where you actually work on it.
Since I am an ambitious person, I lean towards selecting important problems, most (but not all) of which are hard. The strategy that has brought me this far in my career is gambling on some percentage of problems actually being easy. So the majority of projects do not survive but I make up for it with the sheer bulk of participating in everything. My capital is primarily concentrated in intellectual resources, not financial assets, and being prolific allows me to diversify the investment portfolio. The victory condition is outlier success in the long-tailed distribution of impact. In other words, I instinctively sacrifice consistent performance in exchange for seeking upside risk, exposing myself to a wider range of outcomes.
Although I cannot speak to the optimality of this greedy r-selected strategy with respect to my incentive structure given the chaotic environment of technical AI safety research, on a personal level, I hate being like this. I would happily take a pill that made me more k-selected with projects I choose to work on (note that this is not the same as being more "focused"), because I would strongly prefer to at least be able to make credible commitments to myself — let alone to my collaborators.
One fundamental property that I struggle with is this determination and optimistic delusion that prevents me from quitting things long past the point others would have. Perhaps this is my way of preserving optionality, or an unwillingness to confront the shortness of life, but I genuinely believe I'll get back to things later and no amount of evidence to the contrary ever changes my mind. Even faced with an unambiguous calibration curve showing me that I'm overconfident, I continue to make the same error.
I don't believe this necessarily paints me in a good light, but my intention here is to be as honest as possible, which means highlighting that lying to yourself is also a form of deceptive behavior. Generally speaking, deception is unethical.
I will use a computer analogy because computers are what I know. An operating system lies when presenting a contiguous address space to a running process. In reality, the physical memory is fragmented. The operating system's lies make a scarce resource appear more abundant. Almost all modern programs expect to be able to allocate or deallocate memory freely, and tend to fail in uncontrolled ways when that expectation is not met. This rate of failure rises sharply when a system's operating load is dangerously close to its peak capacity.
Clearly, since my intuitive sensors report misleading information, I'm closer to the process than I am to the operating system. But I don't see that as an acceptable explanation for the significant delay. In fact, this absence of reliability is not something that is isolated to this particular endeavor. To my chagrin, missed deadlines are almost a routine event. I am aware of the differences between the patterns I follow and the instructions provided to a safety critical real-time embedded system, but awareness does not equate to automacity. It will take further reinforcement before that gap closes.
Taking the outside view, this problem has a simple solution. Impose firmer constraints. Enforce rate limits. Set lower standards. Explicitly drop things to free up spare capacity. Prioritize. But a portion of me resists, arguing that overly rigid structure will rob me of a certain spark. Freedom, learning, exploration, creativity, spontaneity.
I think doing this was a neat experiment and I had good reasons to compromise on its success at the plot points where I did. It's on the backburner for now, but I will still finish it. Not out of duty or obligation or pressure or stress but because I still find it interesting to indulge in and this is a missing piece in a broader goal I'm working towards.
It's a delicate balance to strike. I don't want to get into the sleazy salesman's habit of exaggerating words without backing them up with action. That would place me past the point of distrust. I am quite limited in many aspects, and earnest attempts do not weigh the same as results. Part of the motivation behind wanting better predictive capabilities is to be able to better manage the division of my energy across opportunities competing for my attention. Thankfully, I can do that using the faculties I already have. Sleep, exercise, write faster, and test earlier.
Extremely relatable! I think your explanation is a little overwrought, but I really appreciated the frank update at the beginning. Good luck!
I have updated on this feedback :D
> Investors spent $33,220 to produce what we judged as $21,125 - $31,125 in charitable value, making the market inefficient. I don’t know exactly how to think about this: I wouldn’t want us to pay the investors much more than they invested, or I’d feel like we were being ripped off.
This is the mistake that leads people to support communism.
Your point of view is wrong and harmful. What you _should_ want is to end up paying the investors much, much more than they invested. You already decided what the project output is worth to you; you should see that it is a better world if $21,000 of charitable value costs $1,000 of monetary value to produce than if it costs $20,500. But your stated preference is for the second of those worlds.
(In slightly more detail, your position alllmost makes sense in the light of your stated commitment to spend no less than $20,000. In that scenario, an outcome you might want to ward off is that someone signs up, invests no money (good!), produces no results (bad!), and then you pay him $20,000, which is a lot more than he invested and is also a waste.
Except that you don't actually avoid that outcome with your rule. Where one person can sign up for free money for no effort, so can tens of thousands of others. When 40,000 people sign up for this deal, you end up paying each of them at most 50 cents, which is not much more than they invested and is therefore OK with you. The waste is still there, it's just as bad as ever, but your rule didn't see it.)
I think you make an interesting point, but I read the whole situation differently.
If the investors are grossly undervaluing the projects, then they're doing a bad job ~of predicting~. It's certainly good for the world—more value is being created than expected, which is great! But it's bad for the hypothesis that "investors can smartly allocate grant funds"—under-predicting is just as bad as over-predicting when the question is the efficiency of this prediction market, right?
And if grant investors are bad at predicting, then Scott is better off with old-fashioned grant-giving strategies. If Scott can decide as well as any "expert" which grants should be funded, there's no need for all this complexity.
In other words, Scott is getting "ripped off", not because extra value was created, but because if the experts aren't actually predicting anything he could've just been the original investor himself. (At least that's my interpretation. Just a random reader, definitely not an expert in this stuff.)
I think you're somewhat right, but that the more complicated factor is that it depends on the counterfactual.
I think most of these projects would have happened anyway even without the impact market. We just would have paid exactly what they cost, in the form of a grant. I agree that eventually if impact markets are to be useful, it will be because they encourage new projects, and that will be because we pay for them. But I think we're still at the "paying a little extra for projects that would have existed anyway" part of the growth cycle.
Reasonable.
I applaud the experiment.
Next time around I'd like to see a less self-licking selection of projects, though. While I can understand the logic behind the idea that the most effectively altruistic thing you can do with your money is to raise awareness of effective altruism so that more people donate more money which we can use to further raise awareness of effective altruism... it still makes me uneasy.
It would be nice to see another experiment done with a more limited scope to exclude these sorts of self-licking projects.
Yeah, I was the dissenting judge for the Maryland EA project. No knock on it, I think it's great that it exists but also that it falls outside the scope of forecasting grants, and vaguely feel like it won a bunch of funding because other judges pattern-matched it to "oh this is what OpenPhil would be willing to pay".
I'm not sure what a good remedy here is though -- in practice having good judges and aggregation mechanism is pretty tricky and will massively influence the final evals, and I think Scott's proposal of "pick 5 reasonably well informed people and take the median" was pretty good given the time constraints. And there's also an argument like "all of this is Scott's money anyways, so it's really up to him to allocate it however he likes". If you want to see different projects and are in a financial position to put up your own retroactive bounty pool, we'd be happy to set up an impact cert round for you!
What I'd like to do next time is get the EA Infrastructure Fund (a charity which exists to help promote EA) on board, so that all good help-EA-expand impact certs will be bought by a charity that's clearly earmarked for that purpose and nobody will be surprised.
I remember that when I was reading the descriptions of the projects, a lot of them seemed to (implicitly) go along the lines of "I was going to do this good thing anyway, but now that these markets are open, I might as well get paid by telling people it requires $5000."
Ah, I'm flattered by the shout-outs! In case you weren't aware, here's my impact markets villain origin story: https://www.astralcodexten.com/p/impact-markets-the-annoying-details/comment/7755694
Overall I feel this was a win. I'm glad the organizers feel like there was room for improvement, but as an investor, I was really glad I participated. I don't give almost anything to charity normally, but the format of this as a market, where I had to exercise my personal best judgement as to returns, act as an advisor to projects when desired, and potentially even make money, helped pull me in. I was very OK with not making any money---indeed, when I saw the unadjusted table first while scrolling through this article, my thought was "Oh well! At least I have some validation that I know how to pick more-impactful projects".
OPTIC seems like a huge success to me, and it was indeed great connecting with Saul and the team. The fact that they were proactive about reaching out to their investors for advice and feedback was a really good sign; I would encourage more projects to do that. (I think if I were a real VC-style impact investor, I would myself reach out and schedule meetings with my investees. But I'm a busy guy. So big kudos to them for taking up that part of the investor-investee relationship!) As part of that relationship, I was also happy that I was able to connect them with my coworker at Google who maintains our internal prediction market; he was able to take a train up to Boston and give a talk there: https://www.youtube.com/watch?v=pYYuMLQjpWY. This kind of incentives-aligned networking with your investor seems good!
My main criteria for picking were:
* Would Scott like this? In particular, this led me to the Kelly-optimal bets tool, and the superforecaster predictions. The latter, in particular, seemed to play well with a lot of the themes we see on ACX, comparing superforecasters with prediction markets and other prediction technologies. I was also interested in https://manifund.org/projects/predicting-replication-in-academic-economics from that aspect, but I wasn't fully convinced by the answers to my queries in the comments there.
* Would this have a big impact if successful? From this perspective OPTIC and Crystal Ballin' seemed high-leverage. The risk here is whether the work actually gets done; I tried to suss this out with Crystal Ballin' but it seems to have only partially panned out. Still, I remain hopeful for them, albeit not on the original timescale.
If people are interested in what kind of projects I investigated, even if I didn't invest, check out https://manifund.org/Domenic for my comment history.
The biggest areas for improvement, I feel, are in communication, both around the whole structure and for newbie investors/investees in particular.
Because you're dealing with something novel, it helps if the rules are really clear. Who exactly are the judges? It would be good to know their backgrounds and preferences ahead of time. Are they supposed to be judging actual realized impact at the 6-month mark, or future potential impact, or...? What exactly is the deadline? What exactly is the funding pot? ("Between $20,000 and $40,000" is a huge range!)
And for newbie investees/investors, I think there are just some basic market mechanics that could serve to be more signposted. E.g., the fact that we collectively bet $45k on a $20k-$40k pot should have generated big warning signs or something. The fact that a project asking for $20k (between 50% and 100% of the total pot) was unlikely to get funded should have been clearer. (Both of these could be addressed in the UI, ideally! Otherwise, putting it in some sort of pre-reading would work.) You could also try to help investors understand better what sort of things to watch out for, e.g. impact/tractability/neglectedness (seems like this would have caught the Polymarket case), investee track record (what many of my questions were focused on), over-ambitious vague projects, etc.
Finally, I told Austin this a while ago, but... where are my actual impact CERTIFICATES!? I want something pretty to print out and brag about to my friends! I want something where selling it back to Scott feels like parting with a small piece of my charitable soul! I want something where I stare at my 66.12% share for OPTIC, maybe in pie-chart form, and think... how could I get the other 33.88% percent?
Thanks for the thoughts, Domenic! I'm glad our round nudged you to think a bit about charity allocations, and congrats on the successful OPTIC bid (I recall putting in an investment offer myself but I think you edged me out :P)
Also really appreciate the feedback - will think about UI improvements to make these market mechanics clearer for next time. A big reason to do ACX Minigrants at all was to discover where these issues are, so we can fix them before trying more ambitious rounds. And fair point on the actual shiny CERTIFICATES as well!
Thanks for the helpful comments and for your participation.
I'm away from home now and don't have access to my graphics program, but maybe we can get fancy certificates working next time!
I know that this sort of post is less 'sexy' and seems to draw less appreciation than most others, so I just wanted to say that I think this is a fascinating and potentially very useful experiment, and I hope it keeps going and you keep posting about it here.
As one of the investors (invested $2k, now worth $1489), and someone who has raised a total of $85M from VCs in the real markets, I think you are way too pessimistic about this outcome. I feel like it was a huge success (9/10).
- you got $31k in value created for $31k without having to take any risk yourselves. Some projects did better than you expected! And you get to pay them more! Some did worse and you didn't have to pay them at all! Incredible! (Aside: this benefit to you is why it's ok for investors to make big returns; taking on risk and investing in a wide field is a highly valuable activity.)
- "we are skeptical of investors choices" is not a failure, it's a success! Think about how much investment goes into stupid startups that you are skeptical of; that's just how it goes. Sometimes though, you are wrong, and it's good that other people have different opinions, and we get Airbnb. As long as all the good stuff gets funded, it doesn't matter how much bad stuff other people waste money on. Relatedly...
- I would argue everything happened correctly with Base Rate Times -- the founder stated that they would not do the project unless they had a certain grant (valuation) that neither we nor you would have funded at that level (and in fact was not worth in the end). The failure here was the founder's pricing of their idea, stated preferences vs revealed preferences (i.e. do it anyways), or misunderstanding of how to correctly keep 100% equity but still participate in the round. In a real open impact market they would have gotten $6k at the end anyways, and it's only because of the constraints of this simulated market that they didn't (the requirement to raise any money). Plenty of startups are bootstrapped and then exit, and this is good.
- the real VC market has the exact same dynamic (actually spookily similar) of not only a power law for startup exits, but also a power law for VC returns -- most lose money, some roughly break even, but the top ones make insane returns. Everyone knows this and wants to participate anyways because _they_ think they can outperform the crowd.
- I don't think you should worry about overall investor returns at all in fact, because a) we were considering this charity in the first place and willingly overspent given that (I aimed to invest at EV rather than aiming to actually make a return, and didn't pay attention to total amount invested by others at all -- reasonable given the bidding structure), and b) the market will adjust naturally based on payouts, especially as the big losers one round forgo the next round. This is all natural and correct. If you keep paying out at the end, investors will find a balance where they do on average make money.
- Even overpaying for securities you and Austin especially like is reasonable! This happens all the time IRL for exits; it only takes one entity thinking it's worth it to buy a company at a valuation no one else believes in. IRL the top five buyers don't pool their money and take the median valuation -- the highest valuer just buys it. You two are not being irrational to shell out more for things you think are worth it.
- You're selling yourself short on the market structure. This was actually quite investor friendly. Usually only the highest-paying investor gets to participate in a startup, they have to wait 5 years for any payout, and they have no idea if in 5 years the startup will IPO into a bear market. It is weird for the total funding not to reflect the actual value creation, but actually you did adjust for this by having a floating total in the 20k-40k range, and I think you did a good job valuing stuff. In a real market the dynamic would be similar I'd think, where charities can buy impact certificates OR make grants, and do more of one vs the other depending on how good the certificates are (thereby adjusting the total end payout). (I agree that the round fails for long term projects like the golf ball one, though; that maybe needs 5 years.)
The one and only reason this wasn't a perfect 10 imo is that BRT/AI treaty didn't get an award at the end, because they were ineligible. In the real market, bootstrapped companies are just as eligible for exits.
Thanks, it's great to hear the perspective from real investors (which I was only LARPing as) and your reflections make me feel much better about some of our market design choices. I do wish we had made BRT eligible at the end, and I definitely plan on changing this for future rounds (eg the one that Manifold natively sponsors)
Thanks for the kind words.
As Austin mentioned, he wants to try something much more ambitious and real-stock-market-like. I'm much more cautious and he's going to humor me for the rest of the year, but next year I'll humor him and we'll try something bigger (hopefully not with my money or reputation on the line)
As one of the judges, I'll just say that it's really stressful to have to try and put financial values on all these things. I felt pretty dumb.
But somewhat gratifying that there are patterns across the funding. Phew.
Judges actually had pretty good inter-rater reliability, I think on average it was a correlation of about 0.4
The cynic in me can't help but feel conflicted about the only project that the judges valued as a standout success being "educate impressionable young people on our values". (A less charitable verb would be "indoctrinate"). Pretty much every talking point about wokeness in academia being a liberal brainwashing camp could be used here with very little modification.
Like, people already accuse EA of being a cult that spends more money on "paying middle class intellectuals to think about how to do charity" than actual charity, and having "gain new adherents to the philosophy" be your highest ROI in a fundraising contest is... not great optics? Especially when the goal is to sell the "impact" to corporations for pure-optics-reasons.
I don't even disagree with the ROI assessment ($300 is really low, and $10k isn't unreasonable), but if the goal is to sell EA benefits to non-EAs, again from a purely optics-position (which matter more to non-EAs than actual outcomes) that seems to miss the mark.
Yeah, and as a vaguely-rat-adjacent non-EA, when I met some of these kids I was not super impressed with how educated they had been. It felt like they'd learned to love the right sources/buzzwords/topics, but not particularly how to think any better.
(If anything, I would say they learned how to think *worse,* as the more they believe in any curriculum as gospel the less they are thinking for *themselves.*)
Of course from the EA perspective they're a win because some of them will become good EAs, I'm sure, (at least for some years if not for a lifetime), but from a 'raising the sanity waterline' perspective - which is what one of them said to me the education they were getting was all about - well, I don't think it turns out that's compatible with teaching people what the right answers to questions are.
You're urging me to distort a financial market for optics reasons! That would break the moral fabric that binds the universe together!
FWIW, I don't really believe it's possible to "indocrinate" EAs. To a first approximation, people either immediately get angry about it and bounce off, or they say "yes, this is the thing I've been groping toward my whole life". The best you can do is convince them to stay in a room long enough to notice whether it's the thing they've been groping towards their whole life or not.
Also, I don't think it's so much that we decided education was the best thing, so much as that Max did an exceptionally good job and sold his impact certificate for much cheaper than it should have been worth. Manifolio got almost as much money as Max, probably for about as much work, they just priced their certificate higher.
Yeah, I felt kind of bad pointing this out, since ultimately I don't think you really did anything *wrong*. At least, I don't have any obvious advice on what you should have done differently. But I think it needed to be said, since optics are a real thing that gets priced into many goods.
> thinking in these terms makes all education/movement-building grants vastly better than all other types of grants.
I have an (experimental! warning! careful in taking this at face value!) paper exploring this, which concludes that, in some cases, you should do an "asymptotic ponzi" and invest asymptotically everything into movement building:
- Paper: https://philiptrammell.com/static/Labor__Capital__and_Patience_in_the_Optimal_Growth_of_Social_Movements.pdf Could be food for thought.
- short explanation: https://forum.effectivealtruism.org/posts/FXPaccMDPaEZNyyre/a-model-of-patient-spending-and-movement-building
I agree this makes intuitive sense (it's pretty cheap to talk to people, so it would be surprising if education didn't make your movement grow faster than your discount rate). I'm still viscerally uncomfortable with it. Partly this is just an emotional "ponzi schemes seem bad" thing, but if I had to put it on a firmer footing:
- I don't know of very many groups that have grown huge through a process of recruiting new people, then using their money and labor to recruit even more new people, and so on forever. Arguably early Christianity, Mormonism, and a few others sort of accomplished this, but there must be many more who would like to (political factions, religions, cults, etc) so it seems surprising that they failed. Maybe that means this doesn't work so well?
- We will do a better job recruiting if we have policy victories to point to, and useful positions to offer new recruits besides just becoming another recruiter.
- I worry that recruiting dilutes culture (in the sense of "good corporate culture", everyone knowing what they're doing and having good values), and engagement with the world replenishes culture, and that having too much of a recruit:engagement balance leaves you with a culture that isn't good enough to use the new recruits effectively.
I don't know if these are my real objections or just a patch over my philosophy to prevent what feels like an infinite-money loophole.
You should also not underestimate that there is a limited pool of potential EAers. Maximizing the pool is probably more important than fishing it dry. Being too focused on recruitment over action will greatly diminish your pool.
Being a member of the 'EA tribe' is inherently neutral. The value comes from average X% of EA tribe members doing things with average value Y. You could do a dirty estimate of value of the entire EA tribe of N people as N*X*Y. Increasing N (your recruitment projects) is positive EV if X and Y remain constant or decrease less than N increases. As any money spent on N could have been spent on X (internal culture alignment) or Y (actual charity work) instead, you need to have even higher confidence that increasing N doesn't decrease X or Y. I think this probably acts as a reasonable backstop to the infinite-money problem.
(it's fun to think about the second order effects - does increasing Y passively drive an increase in N through good publicity? Does increasing X make Y more resilient to increases in N, etc)
I read barely half of it and it seemed pretty byzantine to me. Any barbarians at the gates yet?
Byzantium resisted the barbarians successfully for over a thousand years.
Quite a track record indeed, even if one considers Byzantium proper starting with Justinian. No offense intended, anyway. Just an association I should have kept unwritten here.
Having just finished Kaldellis' "The New Roman Empire" I apologize once more. Somehow Eastern Rome had mostly been presented as a failure to me before.
Personally: I'd be itchy calling these charitable.
Would like to see some that benefit people who aren't well-off. I guess I can understand if people want to invest in gigabrained crypto applications that could replace the dollar!!! and not apps that make third-worlders' lives easier, but you have to admit we're snorting clouds over here.
Did you read the post? There weren't any first-world crypto-related projects - the only one with any crypto connection was supposed to help poor villagers in India (and it lost money because we were skeptical of it).
More to the point, because this was a test round, I asked that we only consider forecasting-related projects. I think it's potentially charitable to donate to development of a science/methodology/community, but I'm not claiming this was the best possible use of money, just the easiest contained community to run a test in.
When I am actually trying to run a charitable grants round, it looks more like https://www.astralcodexten.com/p/acx-grants-results .
"This was a hard grant to value. One way to value it is something like: suppose that he keeps doing this x 3 years, and 5% of his students become long-term committed rationalists/EAs. That’s 9 new committed rationalist/EAs. Suppose half of those would have counterfactually found our community anyway; that’s about 4 new ones. Suppose of those four, one takes the GWWC pledge to donate 10% of lifetime income, and another goes into direct work and has a good career in some useful institution. Each of those people could plausibly generate $100K in charitable value. So maybe we should value this at $100K."
This is the whole problem with the approach - you're just making up numbers, or picking them out of a hat. Not only don't you have a clue if these numbers are right, they're not really testable either (unless you add a decades-long tracking of the participants, and even then their truthfulness might be in doubt). The value might as well be $0 - in fact, this might be the likelier number. The whole evaluation is empirically empty.
Sort of agreed, but this is the same problem any grantmaker faces - they have to determine whether to give a project money or not, even though vague benefits like education and happiness are impossible to value. We're not making this problem any worse, and we're sort of making it better, in the sense that we were able to value it after six months rather than immediately, and so we know that it at least succeeded in getting students (which a normal prospective grantmaker wouldn't have known).
We do then make the problem higher-stakes by making investor money payback contigent on our valuations. I'd like to solve that by having investors understand my value function - I have a history of giving grants, so would-be investors can know what I like and how much I tend to pay for different categories of things. In the future, I want big charities with long track records to join in, and then this will be on even firmer footing.
It's one thing to not be able to put numbers to things that are inherently extremely difficult to know, it's another to put purely made-up numbers on it and pretend you now know something. The latter seems like a recipe for false confidence.
Given that population numbers and the concomitant impact on the sustainability of the earth for numberless future human beings wouldn't it be a valid longtermist goal to bring human numbers in alignment with sustainability? They seem to be pinning their hopes on an assumption that we can just tech our way out of this. Which I suppose would have huge appeal to the "tech bros".
I think there's a division on whether we should be funding decreased population (to solve the environmental crisis) or increased population (to solve the fertility collapse / economic crisis).
I tend to think there is no environmental crisis related to overpopulation, in the sense that there are no current famines not related to political issues (ie we can grow the food and transport the food when warlords don't prevent us from distributing it), plus the history of things like the Simon-Ehrlich wager, plus the fact that non-immigrant population is set to decline on its own everywhere except Africa, and Africa is expected to stabilize soon. I'm also concerned that "fight overpopulation charities" have a really scary history (see https://www.astralcodexten.com/p/galton-ehrlich-buck ) that makes me want to run away screaming.
So I'm not currently funding any population-reduction charities in particular, although I am funding some more generic environmental/sustainability projects. I haven't funded any increase-population-charities yet either, mostly because I haven't found ones I like, although some of the IVG related charities I fund might do that as a side effect.
None of that matters here, because we only funded forecasting charities, to limit the complexity of this test round.
[I went to UMD for undergrad, as did a different Matt G; I wouldn't be surprised if there are more of us floating around the rat/EA sphere.]
UMD is a party school--but also it's the flagship university for the state of Maryland, and gets a lot of top students thru scholarships and being well-ranked for lots of prograsm, has a bunch of investment in undergraduate research, and so on; I am not surprised that one could put together a solid EA group there.
I feel like the median here is potentially awkward. Roughly speaking: six months ago, a bunch of people were asking "what will this be worth in six months?" and the project sold certificates to the person who gave the highest answer. And today, five people are asking "what's this worth today?" and the investors are selling certificates according to the median answer of that group. If the five people were buying individually (and could afford to buy everything they wanted), 9/18 of the projects would have made money. (Five had 1/5 valuing it higher than total equity, three had 2/5, one had 5/5, and nine had 0/5.)
I assume "a group of people decide among themselves what to invest in using some way to aggregate their opinions" also happens a lot in the stock market and VC, so I don't think this is a bad way to do things. But it stood out to me.
Noob questions: I couldn’t find a clear explanation. 1. What does having an “impact certificate” mean? Is it about the impact of the entire project for its entire duration, potentially indefinitely, or just for a specific part of the project? 2. Can the project owner issue new shares after certain milestones are met?
This is such a fascinating experiment in applying market dynamics to charitable impact! It’s impressive to see how innovative ideas like these are taking shape and pushing boundaries - who knew forecasting and charity could come together in such a "winner-takes-all" format? I especially loved seeing projects like Max Morawski’s workshops at UMD and OPTIC making waves—what a testament to the power of education and community building. Even though not every project paid off for investors, the learnings from this trial run feel invaluable for refining the model.
The comparison to venture capitalism here is spot on - sometimes it's about betting on people, not just projects. I'm excited to see where version 2.0 goes, especially with the idea of involving established charities to make the system more "real." It’s clear there’s a ton of potential here, and this first iteration shows just how creative the charitable ecosystem can get. Looking forward to more updates on this, and congrats to everyone involved!
https://retrobowl.pro/