> The person who spends one second picking a random number on a thousand questions will get more points than someone who spends an hour researching a really good answer to one question.
While it sounds intuitive, I recall reading some studies that the "wisdom of the crowds" works better when people don't talk to each other or research the questions [1], as these are subject to various cognitive biases that can easily lead you astray.
So if the goal of a prediction market is to be more accurate, maybe one with the kind of incentives you describe might work a little better.
[1] Don't have time to lookup references now, anyone else have them?
I have a much bigger issue with the Metaculus scoring system than what has been mentioned here:
In Metaculus, it is possible to predict the probability of the event *completely accurately* and *still lose points* if you are unwilling to come back to the market and update your prediction every time new data comes to light.
This is because the Metaculus scoring rule isn't applied instantaneously - you make a prediction and then the score of that prediction is averaged over the entire course of the market from opening to close (see the integral in the Metaculus FAQ scoring section). If you do a great deal of analysis to determine the probability that COVID deaths in a certain month are 50% likely to go over a certain value, you make a 50% prediction, and then halfway through the month it becomes clear that deaths will go above that value and the market starts predicting close to 100%, you are penalized as if you had made a 50% guess on a 100% probability outcome every day for the rest of that month.
Like most of the complaints about the Metaculus scoring system, this is a feature not a bug. Metaculus wants to incentivise activity and as you've explained, this does exactly that
This is my big problem with Good Judgement Open too. There's no reason why they couldn't let you withdraw a prediction and only get points for the time while it was up - I think they only do this to try to incentivise people to update their predictions frequently.
Hey Scott, this is totally unrelated to this post but not sure how else to contact you. Just saying I used to put all your posts on SSC onto the Pocket app where I could read in dark mode and also easily sync to my kobo ereader, but these posts on substack don't work (if I put a post onto pocket it refuses to recognise it as an article). I thought maybe this was a deliberate design choice on the part of substack to sabotage their articles going on pocket as they wanted readers to use their own app, but I've since discovered substack blogs for which it works fine so it's definitely a solvable problem. It's understandable if you don't have the time/inclination to investigate this, maybe this is a rare thing to be bothered about, but I guess all I can say is personally I'd be eternally grateful if you looked into this as I really miss reading your posts on my ereader :)
Really? I just tried again and definitely still broken for me. I think it's because I have a substack header at the top - it thinks every article is just the front page of ACT. Are you saving from a mobile device or desktop?
Did you contact Pocket? There might be some problem on their end too. And they could probably tell you what the problem is in more detail anyways. (They're friendly!)
Based on previous reports we've received on Substack, it seems that the new Substack blog AstralCodexTen uses Javascript in their items in a way our Parser is not able to handle. This is the reason these items are not properly displayed in Pocket's article view.
Moving forward, our team is aware of this area of improvement in our parser and will make sure to let you know if we update our parser to support these types of domains. In the meantime, opening the items on Web view would be the best approach to view their content.
What they wrote matches my own experiences with programming web scraping ("parser[s]") – it's tedious, error-prone, and that's even ignoring the arbitrary complexity of JavaScript.
A simple solution here is not to maximize for Metaculus points, but to maximize for average points per question. Some people have compiled the data necessary for this calculation.
This value needs some shrinkage to remove luck factor for small n (as done in baseball stats http://varianceexplained.org/r/empirical_bayes_baseball/), but otherwise, it is pretty nice. My average score is currently sitting at 31.5, putting me at the 36th spot according to that spreadsheet. One cannot just mindlessly spam median predictions to maximize this value, one has to predict on questions where one will beat the market.
This just returns to a zero-sum scoring, though. If I'm understanding the metaculus article Scott linked ( https://metaculus.medium.com/a-primer-on-the-metaculus-scoring-rule-eb9a974cd204 ) right, that could easily be achieved with some different scoring rule or parameter choice. Metaculus wants to incentivize you to bet on more questions than just those where you're reasonably certain you can beat the market.
The positive EV of the current system is a deliberate feature, not a bug.
There are ways to game this metric as well. You don't need to "beat the market", you could just choose to only predict on questions which are in the 0-10% and 90-100% range.
Cool. The one I linked to is a static copy, not one that I made. Your site does not have me anywhere as far as I can see. Maybe too many of my questions are private ones which have no cheating-prevention.
The ones in the extreme range would only produce high gains if one actually gets them right a lot of the time. Do you have data for average points per question, by median community prediction at closing? One could simply adjust for this statistically if there's some bonus from their algorithm.
> gold is currently worth $1700/ounce, and 20% chance for $2500 by 2025 seems plausible
In fact, that's very plausible. You can look up options prices for gold based on the end-of-2024 futures contract. At the moment of this writing, the end-of-2024 call option with a strike price of 2500 has a "delta" of about 0.25, giving a market expectation of about 25% that gold will be at or above $2500/oz in December 2024.
If the Foresight exchange operated on real money, there might be an arbitrage possibility there.
Notice however that in both cases this isn't a prediction that gold will be worth 20-25% more by 2025 because of the time value of money. Though the arbitrage opportunity would still be there.
“But if instead you put your energy into finding stocks that went up 6% per year on average, you would make 339x returns”
Your accounting assumes that someone has an edge in equity investing but fails to account for any edge they might have in the prediction market. Presumably if they’re investing in the prediction market, they think they have an edge there.
All you need to make the prediction market work is the existence of people who think they have an edge in prediction markets but not in equity markets.
I think the deeper problem is that the proposed index-fund scoring system is no longer proper if the equity markets might depend on the outcome of the bet itself. For example, a bet of "societal collapse by 2030" would probably also result in a collapse of equity markets, so the payout on a correct "yes" bet would be trivial.
Public support for both the ruling party and the Catholic Church took a nosedive, and the delicate Schelling fence of the 1993 "abortion compromise" (itself very close to the Catholic position) got completely wrecked. The law briefly became extremely restrictive, and it is expected that once the government changes it will become more liberal than ever, as most of the opposition aligned itself with the pro-choice protesters.
The major opposition party was previously playing a balancing act a la median voter theorem - not appearing too anti-church nor pro-church to avoid alienating anyone. Now they were forced to take a side.
Would index fund theory work on a true prediction market? Like naively I think you could make money on the bet that “prediction markets are good at predicting” by just betting with the house odds on every bet and periodically readjusting. Is this dumb for some reason I don’t realize?
To expand on what boog said, buying an index fund is buying shares of companies that have managed to get listed in an index, which tend to stick around and appreciate in value because they're producing something of value. A prediction market just reallocated money from bad or unlucky predictors to better or luckier predictors. It would need some way of not being zero-sum, the way a real economy is not zero-sum, for an index fund to produce a positive expected return.
In an ideally functioning prediction market, the expected value of buying YES and NO is equal. If one is less likely, it cheaper to buy and has better returns, with the other therefore being more likely but with worse returns. (If this isn’t the case, people are incentivized to trade in a way that corrects the price so that it is.) So, you would expect to overall get exactly the same about of money you put in (minus fees).
Index fund theory *works* here—the problem is just that the rate of return for the entire index is guaranteed to be 1. For a total US stock market index, we have a historical expectation that the entire index will have an annual return of ~1.07 over a long enough period of time.
In 1996, gold was $369 an ounce, so it must have seemed like a safe bet that it couldn't inflate so much that it would be worth $2500 in 30 years. (It would be an average of 7% inflation a year, which is bad but not exactly hyperinflation either.)
Gold prices skyrocketed around 2007 and never really came back down, even though inflation stayed level throughout the financial crisis. So I guess it's not as good a measure of inflation as they thought.
On the other hand, in 1966 gold was worth $35.50 an ounce, so it had already gone up 10x over the previous 30 years without anything approaching a US government collapse.
The idea that gold might go up another 6x over the next thirty years shouldn't have surprised anyone.
Or else measures of inflation are more heavily manipulated than the gold price is.
Numbers like CPI are Goodharted to hell, and so an actual inflationary trend may be masked by people's political incentive to make the widely-accepted, legible numbers look good. (Cf. legible expertise.) Meanwhile, there's definitely manipulation in the gold market, but there's a limit on how far it can push the price from that implied by actual underlying supply and demand.
Note also the Cantillon effect of monetary base inflation as separate from CPI inflation. The notional goal of programs like quantitative easing is to manipulate financial markets in ways the Fed considers desirable, while keeping the excess money from leaking out into the broader economy too much and causing CPI inflation. When it's successful, this kind of program means that financial assets all go up enormously while everyday goods stay relatively flat. (This also greatly exacerbates the wealth inequality between the asset-owning class and the wage-earning class.) In this context, gold is largely a financial asset and specifically one held as a hedge against inflation, so it gets the full benefit of the excess money in asset markets.
If you predict the maximum 99% and the event doesn't happen, you **lose** 351 points.
If you predict the minimum 1% and the event happens, you **lose** 353 points.
If you believe the event is 50% likely (not sure why would you believe that, but arguendo), you expect 50%*13+50%*18 = 15.5 points.
If you believe the community that the event is 45% likely, you expect 55%*29+45%*1 = 16.4 points.
If you believe the event is 75% likely, you expect 75%*77+25%*-84 = 36.8 points.
etc.
Importantly, the scoring rule means the point-maximising strategy _on any question_ is to predict your true credence.
This is separate from the point maximising strategy _on the whole site_. But as Scott points out, nobody does that, because that'd be unbelievably boring. Notice that the best users in absolute number of points have much more than 15 points per question (which is what you typically expect by predicting the community median). https://metaculusextras.com/points_per_question
Does Metaculus still need to exist in 100 years in order for the bet to pay out? Because in that case, you also need to multiply any win chances with the low percentage that this will in fact be the case.
Hmmm – how has this worked for past exchanges? Do they always 'blow up' when they 'die'? Or can they also be wound down, particularly in such a way that transactions can 'migrate' to other exchanges while that's happening?
I have a vague intuition that everything _could_ work fine, even if particular prediction markets come and go over time. It's the ongoing trading that's valuable, even apart from the 'final judgement'. (Regular financial equities don't seem that different. You can, of course, make a profit off of other activity related to an equity apart from making an accurate prediction about the entire path of any equity's 'true value' over its entire lifetime.)
I aggregate the top comments on Metaculus (by likes) and share them in the discord. I'm going to continue sharing them on Mantic Monday until someone asks me to stop.
Maybe instead of betting on outcome there should be traceable binary options? You can totally imagine options with 100y. Their current trading price will determine the odds and will also handle time value (I.e. price might change if less time remains for something to happen)
The question is: who would sell you such an option? “Market makers”?
The problem with this is that you have to put up capital. Say I want to bet $1 on event A to happen in 100 years which the market says has probability p. How much does it cost me to do so? There are several options:
- Put up p * $1 at the start. Never see the money again until you trade out. This has all the problems discussed. You're giving up lots of time value of money / alternative return sources etc
- Put up some smaller amount as margin. Pay margin whenever the price moves. (This is roughly how most futures and stock markets work). The issue with this is that for a *binary* event the probability can jump. So how much margin should I require from you so that when the probability moves I know you will pay your margin call or I can liquidate you? Ans: you have to be fully collateralised, since p can jump to 0 or 1 whenever the event occurs (or doesn't). There are some cases where this would work (things which resolve on a specific date AND can't be known before hand) but for lots of interesting questions this jumping problem is a death-knell for margined prediction markets
I still not sure where the problem is. Here’s how I imagine it could work:
Imagine an option that pays off $1 in 2121 on R winning the presidency.
Market Makers can hedge their risks by selling options against each other (R+D options portfolio constitutes a constant payout to the first approximation).
So, today, MM could sell options for the discounted value of payoff expectation. If, e.g. $1 in 100 years = $.01 now, then both R&D options will be half a penny.
50 years from now, both time value will be higher, and probabilities will change => you can sell your option for higher value and make some money before 100yr.
As for “fully collateralized” - I don’t think it is a requirement. Rather than being fully collateralized, MM keep low risk portfolio and move inventory to keep it that way. I.e. if more people will buy D options, they will lower price for R option to incentivize more people to buy it. Which is how market discovery works.
Of course this implies that MM can put your 1c to work over 100y horizon, but I feel that it is reasonable to assume stable investment vehicles to exist if you want long term prediction market to exist too.
I think you need to lay out more exchange mechanics than you have and then you will see the issue here.
Who can "create" these shares? If it is the exchange (creating a D+R worth $1 in 100 years) then they are implicitly issuing a 100y bond. The interest rate on that bond is going to be a combination of their credit worthiness and 100y rates. If it is the MM then that's the same issue (except now it really matters which MM issued the options since the credit worthiness will vary from trader to trader).
How are you forcing MM to keep "low risk portfolios"?
"Which is how market discovery works." Sort of? Maybe? I really don't think so.
"Of course this implies that MM can put your 1c to work over 100y horizon" well now you're exactly back to the index fund suggestion...
BTW, that and https://www.metaculus.com/questions/841/will-metaculus-exist-in-2030/ together imply 10% chance that Metaculus is operating on Jan 1st 2030 but there will be no question resolution after then, which sounds absurd to me, but I can't be bothered to arbitrage that away right now.
There are actual concerns with free and open money-based prediction markets. The market for "President Biden will be assassinated before Jan 1 2022" will create some perverse incentives. Or if you're using it for informing government, like Scott suggested in the recent GOP post, you have to ensure that, say, Exxon-Mobile doesn't pour billions into the "Global warming doesn't exist" market in order to ensure that the government won't regulate them. Even if they're just used for informing public opinion, you'd have similar incentives to dis-information (and if everyone believed Exxon-Mobile would manipulate the market, they might play along to make a buck, adding to the misinformation, depending on how things operate).
Is there literature on addressing concerns like these? Apparently the first problem is at least partially addressed by that crypto-currency market by including a "invalid/unethical" potential result.
I think the answer is probably something like 'this is a fad only among a small class of primarily American intellectuals, no other country has noticed/cared about the idea enough to put in the effort and capital to make a large one.'
> "which party will win the 2100 presidential election?" might change a little over the next 80 years as (for example) demographic change makes one or another party permanently stronger or weaker
Duverger's law makes demographic change, or even what issues the election is fought on, irrelevant. So long as the USA keeps the same electoral system, there will be 2 parties and each will have a roughly 50% chance of winning the election.
My forecast is a 90% chance of "no one." I doubt baseball will be around in 100 years, and if it is there is a very good chance that the name of the championship is changed. Replace "World Series" with "highest level professional baseball championship" and I put it at a 75% chance of no one.
The championship has been called that for the past 118 years, so a 60% chance that it changes in the next 82 seems pretty high - I don't think there's a particularly strong tendency for sports championships to change their names?
I remember late 2019, a lot of the SSC commenters had near certainty on who would win the 2020 election. Six months later, they all changed their mind, saying covid completely changed their predictions.
100 years is a long time, we are likely to experience at least one covid level event between now and then.
Metaculus spammers do hurt accuracy though! Suppose you put up a new question and the smart parts of the market think it's 35.8% yes. The spammers show up and reinforce that to an extremely confident 35.8%. Then some new information comes out that makes a yes look almost guaranteed. If the market was only smart participants it would shoot up to ninety-some percent, but the spammers (who don't bother to update their predictions because the whole point of spamming is that it's a low-effort strategy) drag the market's estimate way down.
What's the actual formula for the overweight-recent function? I'm curious how it acts if many automated predictors each make a small update after every human update.
If the bots predicted to median value whenever it changes and then waited for the median to change again, I believe it would make the median slightly harder to change, but you'd still need ~sqrt(predictors) new predictions to change it.
But isn't the whole point of spamming to increase your score by answering many questions without researching them? Wouldn't this strategy likely include frequently updating your predictions, since updating to the crowd score regularly would increase your score without increasing your need to do further research?
I can see how such a strategy starts to sound high-effort pretty fast unless one can auto-update to the crowd score without looking at it, which would resolve the problem you pose.
Does anyone know if these spammers really exist, or is this all theoretical? Because a spamming strategy sounds awfully boring. What is the incentive to do it? To win fake internet points in a game you have little interest in playing?
70% chance the officer who killed George Floyd will get acquitted *of the murder charges*. This is important - the difference between complete acquittal and only acquittal of the most grave charge is quite substantial.
I recently learned about Long Bets (https://longbets.org/), which tracks predictions and bets between individuals to be resolved many years in the future. Started in 2003, and originally funded by Jeff Bezos.
The winnings go to charity, and the money is invested in the meantime, with the gains being split between the foundation that runs it and the winner's charity.
Since you posted this, the Tether prediction has gone from 36% to 16%. While this may be in part due to the number of predictions doubling, it's interesting what it says about the attention of the market. On the other hand, no one here seems to have had an option on the Polish abortion rate. (The George Floyd question is already closed.)
Someone once said something like "Wikipedia ensures that the page on flatworms is written by people who really love writing about flatworms, and the page on the Roswell incident is written by people who really love writing about the Roswell incident".
The general rule here is that, for certain objectively-not-very-significant topics, there's a vocal crank position that says that the topic is actually extremely significant and the truth about it is <insert weird crank theory here>. If a system is set up so as to produce an aggregated opinion on a topic in such a manner that the people who spend a lot of time contributing to it have disproportionate control over the result, then you'll end up privileging such crank theories, because their proponents have more time to devote to promoting the theory than their (mainstream) detractors have to devote to debunking it. (This is adjacent to the general problem of obscure Wikipedia articles, where often the article about some company will be effectively written by PR agents to that company because no one else cares; or, conversely, some figure with a big Internet hatedom will have his page taken over by the hatedom and turned into an obsessively curated shrine to everything he ever did wrong.)
One advantage of an actual prediction market, rather than a nothing-at-stake scoring system like Metaculus, is that there's a built-in incentive to challenge cranks like this: you can profit off their being wrong. I don't have enough information to say how significant this really is to the operations of the market.
I think the crank theory has a lot of merit here. The people most interested in the question "is Tether going to collapse" are the people who have theories about the collapse of Tether. Everyone else probably wasn't paying much attention to the question, until Scott linked it with a surprisingly high number attached.
Disclosure: I'm a frequent predictor on Metaculus and it's hard in cases like this to not have my thoughts be dominated/emotionally compromised by naked tribalism.
That said, I thought, and continue to believe on reflection, that both Zvi's critique and Ross' writeup missed what I think are [fairly obvious points](https://twitter.com/LinchZhang/status/1360300621782392834). The primary point of Metaculus and platforms like it is to aggregate truthful information about the world and present it in a way that's collectively digestible, not sample-efficient ways for to get perfect individual rankings.
Obviously both desiderata are useful, and there's a case to be made that maybe the current Metaculus system rewards systematic accuracy and legibility too much over individual accuracy and legibility. But I think any reasonable critique should've been framed much more like "we think you're making the wrong tradeoff on this spectrum because of XYZ" rather than "my point on this spectrum is obviously correct and anybody who says otherwise is a moron." Worse, I think Zvi et.al implicitly aren't even acknowledging the tradeoff at all?
The point on "participation trophies" seems fairly off the mark. Whenever I talk to people who think about how to improve forecasting, one of the most common points is that "the problem isn't that people just or primarily make inaccurate predictions when they make precise numerical answers to well-defined questions, it's that they don't make much if any forecasts at all." To the degree that points incentivize behavior, having participation points is good because it incentivizes more forecasts, since if you want to get more of something, you should incentivize it.
In general, I feel like the critique seems more like "You claimed to be doing X. You did X, which is not like Y. I expected this to be Y, without any justification for why Y is better. Now I'm going to judge you on the metrics of Y, rather than X."
I was politer on Twitter when this kerfuffle first came out and I was deliberately trying to adjust for emotional compromise, but upon reflection I'm inclined to think that this type of motivated ignorance is somewhat suss.
I think it is difficult to express this in a way that is credibly sincere, but: Thanks, I appreciate this criticism. I think it helps me understand where our intuitions are going in different directions, and hopefully will help me to put my intended points more clearly in follow-up posts.
I also think I made mistakes in my original critique, because I do not believe (and did not intend to imply) that "anybody who says otherwise is a moron", and I'm disappointed that it reads as if framed that way.
On the substantive point: My core critique is *not* that the current scoring function does a worse job of accurately ranking individuals. (I understand that ranking individual predictors is not the primary desideratum.) Unfortunately, I don't expect that this comment thread is the best place to re-attempt an explanation of the concerns I actually have.
Thanks for interpreting my comment charitably. I now think I should have read your original post more carefully. I'll reread it again and I'll revise my comment if needed (or issue an appropriate retraction if I no longer stand by the sprit on a careful rereading).
I agree with you Linch. I read Zvi and Ross' posts and found both of them to uncharitable and frankly disrespectful. Metaculus isn't some part time senior project with 3 users. I don't use Metaculus but I think it already provides great value and I am incredibly excited for its future. The primary reason for that is Metaculus, unlike most every forecasting website, actually has an active community. Their scoring system plays a huge role in that.
I found Ross' critique's [1] major points interesting and worth contemplating. On the other hand, it started with this:
"In Zvi’s 2/11 Covid update, he turned to Metaculus for help. He looked at the numbers. Becase the man is an inveterate trader, he saw odds that were Wrong On The Internet and just couldn’t stop himself from creating an account to bet against it. And then he saw the absolutely bonkers payout structure and decided he was done after making a single prediction.
Seriously, I’m not even making this story up.
I spent ~20minutes with the Metaculus site and figured out how they borked this one up badly enough to drive away Zvi Mowshowitz, of all people. Anyway, because Zvi reportedly “can’t even”, and because I’ve already done the legwork, I’ll try to explain it here."
I was inclined to stop there. Lionizing certain people and their opinions with informed quality how they are obviously awesome in order to put other people down ... it is a bit uncouth style of argumentation.
My prediction: if Prediction Markets ever get to the point of relevance, Wall Street will end up being the only ones making significant money on them. It's the hedge funds, with their billions of dollars to devote to computing power and market research, that would end up being the "superpredictors".
I agree this is true, though more because the hedge funds can just hire the superforecasters. I'm not sure if it will be literal Wall Street or some other group of companies centered somewhere else, but it will probably happen. This is my best-case scenario - it'll mean we have really competitive profit-focused companies working on predicting important things for us with an incentive to get them right.
Probably random people who are good at predicting can still do it in their basement, the same way people can still trade stocks on Robin Hood.
Sports betting is the most liquid prediction market in existence (assuming you don't count the stock market, which I think is fair). Market makers make a lot of money, sophisticated syndicates make a lot of money. Some individuals also make money in various ways, including doing the same things the syndicates do. Part of the reason they can is that the market makers limit their exposure to the syndicates. Final betting lines are extraordinarily accurate.
The issue here is the Praeto Rule. In any competitive/creative arena the square root of the total participants, call them first derivative, will take half the earnings. The square root of the census in first derivative will take half of the earnings of that group.
What that means is that out of 10,000 people if everyone keeps playing 10 people will end up with 25% of the money. In any competitive endeavor in the history of the world this effect plays out.
Isn’t the stock market already the prediction market for the only questions worth betting large amounts of money on (namely, which companies will be making money and how much in the long run)?
Conservatism of Polish society is waning. However, politically it's on firmer ground than almost anywhere near the western world. That's because our ruling conservative party has an unusual pro-welfare bend, and the main opposition party are committed (European meaning, i.e. classical, free-market) liberals. Unless things change (and they could, the liberals are now losing support to a new amorphous, but potentially more socially sensitive centrist option, and we still have some nominally left-wing parties left), people will likely continue choosing social conservatism for the increased material stability it is packaged with.
It looks like New York allowed abortions generally before Roe, and three other states had some situations where they allowed them, so depending on how much people were able to cross state lines to do it, there might not have been as big a change in the legal picture as there would be in a place with uniform national laws.
It looks like statistics weren't counted before 1973, but abortions doubled within the next few years, and then have gradually fallen to below 1973 levels (presumably as a result of more birth control options, among other things).
Religious conservatism and secular conservatism are different movements, with the attitude towards gay marriage perhaps being a clear difference marker. So a strongly conservative society can still legalise gay marriage with an alliance between secular (and to be fair some religious) conservatives and the liberal minority.
The same alliance can support abortion as well, although as Deisarch says it's a long road. Conservatism need not be religious (witness Trump for a very clear example), and I doubt any western country does not have a potentially significant number of secular conservatives, even if it doesn't have an organised movement.
If Poland is like Ireland, there will have been activist campaigns nudging and nudging and nudging on this for years, plus some foreign foundations sticking their spoke in. We now have legal abortion (hurrah 😐) after a couple of referenda and decades working away at it. Constant dripping wears away the stone, and when it goes, it goes fast. https://en.wikipedia.org/wiki/Abortion_in_the_Republic_of_Ireland
The Irish situation is a bit disanalogous, since so many women went to England the notional ban didn't have the impact it might have otherwise had. (Especially on middle and upper classes who are most likely to be politically active) If there wasn't that option it would likely have been changed much sooner
I just looked up abortion statistics in Ireland. Quite wonderfully, there were 6,666 abortions in the year after legalization.
and I have a small sample size, but the Polish progressives I know seem unusually competent. I have much more hope for a liberal turn there than in other right-leaning Eastern European countries (think Hungary)
The current near-total ban on abortion in Poland is based on the constitutional court's ruling that Article 38 of the Polish constitution (which says simply "The Republic of Poland shall ensure the legal protection of the life of every human being.") includes fetuses, which means that re-legalizing abortion would require either a new ruling by the constitutional court overturning the recent precedent, or an amendment to the constitution. The court's justices serve for 9-year terms and are nominated by the lower house of the parliament, and almost all of the currently serving justices were nominated by PiS (the socially conservative party which has controlled the parliament since 2015), so it would be difficult and take a significant amount of time for the liberal parties, if they got back into power, to replace enough of the justices to have a chance at overturning the current precedent. An amendment to the relevant part of the constitution (as described in the constitution at http://www.sejm.gov.pl/prawo/konst/angielski/kon1.htm ) would have to be passed by a vote of 2/3 of the lower house of the parliament, then a majority of the upper house, then a majority of the voters in a referendum, which would also be difficult since (according to the polls summarized at https://en.wikipedia.org/wiki/Abortion_in_Poland#Public_opinion ) the majority of the people support abortion only in unusual circumstances (in cases of rape, incest, danger to the mother, or probable disability of the child) and only a rather smaller minority supports abortion on demand. So it seems unlikely that abortion in general will be legalized in Poland in the near future.
The constitutional court, with its current illegaly chosen set of judges, is not binding - at least according to one side of the controversy (to which I belong and so does half of the country). Once the government changes it will probably attempt to throw their rulings out of the window ASAP.
Constitutional articles are no guarantee. If you put them in, you can take them out. In 1983, after a referendum, we had the Eighth Amendment to the Constitution enacted:
"The Eighth Amendment of the Constitution Act 1983 was an amendment to the Constitution of Ireland which inserted a subsection recognising the equal right to life of the pregnant woman and the unborn. The Amendment inserted a new sub-section after section 3 of Article 40. The resulting Article 40.3.3º read:
The State acknowledges the right to life of the unborn and, with due regard to the equal right to life of the mother, guarantees in its laws to respect, and, as far as practicable, by its laws to defend and vindicate that right."
Well, then we had further referenda, some based on test cases:
"Three referendums were held in November 1992. The Twelfth Amendment of the Constitution Bill, 1992 sought to exclude "a risk of self-destruction" as grounds for abortion, to overturn the central element of the decision in the X Case. This was rejected by 65% to 35%. The Thirteenth Amendment, permitting travel to obtain abortion in another jurisdiction, was approved by 62% to 38%. The Fourteenth Amendment, permitting information about services in other countries, was approved by 60% to 40%.
After these amendments, which became law on 23 December 1992, Article 40.3.3º read,
The State acknowledges the right to life of the unborn and, with due regard to the equal right to life of the mother, guarantees in its laws to respect, and, as far as practicable, by its laws to defend and vindicate that right.
This subsection shall not limit freedom to travel between the State and another state.
This subsection shall not limit freedom to obtain or make available, in the State, subject to such conditions as may be laid down by law, information relating to services lawfully available in another state."
And in 2012, using the case of Savita Halappanavar (one which got global attention and which was more complicated than the soundbite "she was refused an abortion and died"), the activists started up again with the Repeal the Eighth Amendment campaign, which eventually came to flower in another referendum in 2018 and the replacement of the clauses above:
The Campaign to Repeal the Eighth Amendment has its roots in the unsuccessful Anti-Amendment Campaign in 1983. There was a campaign to Repeal the Eight Amendment after the X Case in 1992, and the three abortion referendums which followed it (the 12th, 13th and 14th). The campaign lay dormant for more than 20 years until the death of Savita Halappanavar in 2012.
On 14 June 2017, the Taoiseach Leo Varadkar announced his government's intention to bring forward legislation to facilitate the holding of a referendum on abortion in 2018.[50] On 9 March 2018, debate began in the Dáil on the Thirty-sixth Amendment of the Constitution Bill 2018, which would replace the current provisions on Article 40.3.3º with the following clause:
Provision may be made by law for the regulation of termination of pregnancy.
On 25 May 2018, the Irish people voted by 66.4% to remove the Eighth Amendment, choosing to replace it with the above text as a part of the Thirty-sixth Amendment to the Constitution, permitting the Oireachtas (parliament) to legislate for the regulation of termination of pregnancy. When it was signed in to law by the President of Ireland, it superseded the 8th, 13th and 14th amendments."
This has been the pattern of social liberalisation in Ireland; referenda after referenda on divorce etc. until the activists get their way, then suddenly it's "the people have spoken, this is the law, can't be changed now". I have no more hope for Poland regarding their constitution than for my own country, because the activists will push and push until they get their way. And they will use 'hard cases' (even though those make 'bad law'). The best present the pro-choice/pro-abortion movement in Ireland got was the death of that unfortunate woman because they were able to ride "see, we told you lack of access to abortion kills women!" all the way to victory.
Assuming no justices die/resign until the 2023 election, an opposition win in said election will allow them to replace at least 8 of them (a bare majority) before the end of the parliamentary term. It's also possible they would simply aim for a more complete overhaul of the whole institution, since due to the 2015 events ( https://en.wikipedia.org/wiki/Constitutional_Tribunal_(Poland)#2015–present:_Polish_Constitutional_Court_crisis ) they view the current Tribunal as illegitimate.
There also exist options to significantly ease abortion restrictions without messing with the constitution, e.g. decriminalization.
Either way, I feel the more relevant question in this case is not whether the opposition as it currently exists would be able to change the law in a significant manner, but whether it would be willing to. (Which, unlikely, though less unlikely than just a few years ago. The massive reaction against the ongoing pro-life offensive is pushing even the conservative parts of it into a pro-choice camp.)
So with regards to the search for Metaforecast.org, I am taking exact matches and ranking them by starts and then by number of forecasts, and then I'm using https://fusejs.io/ for the rest.
The code for this is open source, and can be found here: https://github.com/QURIresearch/metaforecast-website-nextjs/blob/master/pages/index.js, lines 120 to 189. The comment on line 130 refers to the fact that, because Guesstimate contains too many models (17k), I piggyback on its search rather than loading the models on initial page load, and I have to do this asynchronously.
Pull requests or suggestions are welcome!
As an addendum, Metaforecast also searches forecasting platforms (Metaculus, Good Judgment, etc.), not only prediction markets.
As for the issue that "I don’t think Metaculus considers accurately ranking people by predictive skill an especially high priority compared to actually important things like getting good predictions, and that’s fine, but it kind of sucks for people who were motivated by a competitive streak"... Metaculus really does care about assessing accuracy for use in weighting predictions in the aggregation method. We also want to give users what they want, and many of them want to compete with each other, so on the drawing boards are plans for an upgraded "leaderboards" page that allows more comparisons. The tricky thing is we don't want *too* many comparison methods, because then you'll always have ways to do particularly well at some of them. And if we make any of those methods based on non-proper scoring then we have to worry that people will have incorrect incentives. The basic problem is there's no such thing as a single quantifier of "predictive skill" so we have do think a bit about a few measures that seem especially valuable.
Using stock market returns to keep prediction returns competitive is inherently flawed, in that it distorts incentives on predictions where stock market returns are lower. If you have a prediction regarding future interest rates, predicting 'Yes' on higher rates becomes a problematic and distorted hedge on stock market returns - if rates increase, stock values drop and the 'Yes' bet becomes inherently less valuable. You might claim that the efficient market hypothesis will correct for this, but then you'd have to prove that the prediction market is inherently efficient.
High efficiency is a sort of already an assumption of any prediction market—if it wasn’t efficient, the predictions would be bad. So requiring that seems in a sense trivial.
For your example, doesn’t the ‘no’ bet drop proportionally as well, maintaining the symmetry? Not sure if that’s part of what you’re referring to by requiring efficiency.
Prediction markets can be semi-accurate without being efficient, as far as I understand it. We aren't just talking about using prediction markets to make money, but using them to accurately predict future events. If you index prediction returns to stock market returns, any prediction that, if true, could result in reduced stock market returns will be commensurately undervalued compared to its accuracy. Does that make sense, or am I confused on this?
My concern about making a prediction about something in 2121 is I have little confidence the prediction market will still be around and functioning then. A lot can change in 100 years. How is this risk dealt with?
I'm turned off by the long-term prospects of prediction markets simply because many of the questions I most want to see predictions on are "heads I win - tails we all lose", getting rid of any incentive to vote tails.
Let's say you want the market to estimate the probability of a world war, hyperinflation in the US, The Singularity, any political change causing major revisions of property rights (h/t David Friedman on another thread), and other similarly significant events. If any of the above actually were to happen, the chance that Metaculus is still around to give a meaningful payout would drop precipitously. And that means there's significantly less of an incentive to bet that those things will happen, even if you actually think one of them is likely to.
That severely damages Metaculus' ability to speculate on matters like X-risk, where I most wish we could benefit from pooling the collective brainpower and financial self-interest of thousands. And I don't see a fix, besides hoping people's honest (or vanity) will outweigh their self-interest.
Yeah, it seems to be a widespread misunderstanding that prediction markets will converge on a true probability. The theoretical fair value of a contract is not just the expected payout, it's the expected value of (payout) * (value of money at maturity). If there's a substantial correlation between the value of money and the payout--e.g. bets on the world ending or inflation, etc.--then the value of the contract could differ greatly from what its probability suggests.
On top of that, there's also the issue of risk premiums. Generally, people are risk-averse (e.g. concave utility functions) and they'll pay to hedge against bad outcomes. What this translates to is that prediction markets will tend to overstate the probability of bad things happening.
It's also very possible to go the other way, where it's like, "What you're predicting is the end of scarcity, where we all are octuple-billionaires. Even if I think it's relatively likely, I'm betting against it, because I don't care about having another $10,000 in a post-scarcity world."
One solution (originally proposed by Eliezer Yudkowsky) is to have the world destruction skeptic give the believer $100 to spend today. If the world doesn't end, the believer has to give the skeptic $200. That gives everyone the right incentives, but it might be hard to implement on a prediction market.
Now to find a way to actually secure the payment from the believer who spent all money on hand on booze and hookers in anticipation of the impending doom and is now penniless and sleeping under a bridge.
That's a cool and probably-somehow-implementable way to get a market to bet on "will this market exist". It doesn't give us any information about *how* people think the market is likely to cease to exist; we don't get any insight into the relative probabilities of nuclear war vs. singularity vs. financial collapse, which is what I really wish we could see. But it's better than nothing! Thanks.
I tried to make this kind of bet with a friend once, and he told me it was a good idea but he could get better odds by just taking out a loan. And if I lowered my rate to be competitive with a loan, I could just invest my money in index funds and get better returns.
In other words, the stock/bond market is sort of already a prediction market for "will the world end?" There's no reason for people to bet on the end of the world outside of it.
A possibly dumb question: if they want to make predicting positive-sum so that more people participate, why not just use the standard way to do it that academics have been working on for over a decade, namely adding a subsidized automated market maker (LMSR or something similar)? Seems like that would reduce the exploitability, as there would no longer be the positive-scoring no-op of making the same prediction as everyone else.
For these purposes there are two basic difference between a basic prediction market and a scoring system like Metaculus uses. The first is positive/zero/negative sum, which as you say could be matched by subsidizing the market. But there's also the issue that in a prediction market with real or fake money you have a limited amount of money to actually invest, which puts a fairly hard cap on how much predicting you can do (or at least how much influence you can have). We could adjust the scoring rule to make it more market-like in the sense of being more purely relative and slightly less positive sum, but the lack of a total amount of funding would still be a big difference. We've consciously taken a philosophy of removing every possible barrier to making predictions, because lack of liquidity has doomed (and is presently doing in) many prediction markets. As participation increases, we can turn the dial a bit in the opposite direction. A final issue (mentioned in my writeup) is that with a scoring system and direction predictions (rather than buying/selling) it is *much* simpler to extract the actual probabilities, especially for predicting a probability distribution over a number or date, which would be extremely awkward in a market system of buying and selling shares.
To give some context to this, Metaculus has around 1,100 active questions. Based on my experience with Polymarket, an automatic market maker would need to have $1000 to $10,000 liquidity in order for there not to be much slippage.
What if after 100 years, instead of getting 100 years of index fund returns plus the bet percentage, the winner gets the their index fund returns and the bet percentage those of the losers?
In that situation, if the total returns are 200x over time on a 50/50 bet, the winner gets 150x and the loser gets back only 50x. On an 80/20 bet, the winner gets 180x and the loser only 20x, etc...
Higher risk, but higher reward to the winner, but if you bet 50/50 one-hundred times and win half of them, you still break even. It also incentivizes taking sides on bets which are more controversial.
You could also have an annual mechanism which multiplies rewards for the winner by the number of years in the bet.
Lots of permutations, I don't think you have to be stuck with the current basic prediction market rules. More interesting to bet on if you don't.
Note that it's dropped to 10% just on the strength of being linked to by Scott, apparently. Not sure what lessons we should draw from that. (The other two questions he linked didn't experience a huge "ACX effect". They had also been open much longer, and had more predictions. The Tether question got around 45 new predictions via ACX, based on timing, which more than doubled its count. The abortion question got about 5, and the Chauvin question was already so popular that ACX's contribution is in the noise.)
The way the expert gets more clout is by accruing a track record of being correct. The "Metaculus prediction" aggregates predictions according to the predictors' track records. We feel this is better than sizing bets, because having more money to put into a market does not necessarily make you an expert (in fact it does not even guarantee that you care about being accurate or extracting money. My guess is that if/when large-scale prediction markets get going most of the money is actually going to come from financial firms hedging against events, where that hedge is valuable enough that they don't mind buying at inaccurate probabilities.)
To SBF's point though, the problem with that approach is that it does a good job of aggregating information from expert generalists, but a bad job of aggregating information from expert specialists. The existing research on forecasting suggests that generalists can often perform better than specialists IF those specialists don't have much at stake on the outcome and are influenced by other social or reputational factors. But I think specialists with very high stake are going to be the most accurate in the long run. Financial markets do a great job of encouraging high-stake participation by specialists, and it's worth thinking about how to structure forecasting platforms to accomplish the same thing, even if those platforms don't have an explicitly financial component.
Financial markets history shows that financial firms are terrible at hedging future fat tails events. Why would we assume they will start using prediction markets to hedge where they can already hedge using the financial market or the insurance market?
I think a prediction market would provide a much cleaner hedge against many types of particular events than a basket of stock that you have to work out that are correlated (or anti-correlated) with that event. And I don't think you can take out insurance, for example, against a particular supreme court decision or piece of legislation being passed.
"if you bet on a 50-50 outcome and get proven right in 2121, you've made 100% returns over 100 years. The stock market could easily make 10,000% returns over that time period, so why would you ever play the prediction markets instead?"
You don't need to do that. If someone else is holding your money for that century, the market equilibrium terms will be such that you get, on average, the market return. You pay $1 for what is currently viewed as a 50/50 bet, and if you win you get a sum in 2121 whose present value in 2021 is $2.
Consider the obvious analogy of buying a bond. Your analysis assumes that if it pays off in 10 years, the payoff will be the amount you paid for it scaled up to allow for the probability of default. But it will actually be that plus the accumulated interest.
OK, but isn't the mechanism that produces "market equilibrium terms" just people not doing things that are unfavorable? So if predictions markets don't have some sort of mechanism for scaling payouts with payout date (which they don't currently have, as far as know), then people just won't use them for questions more than a year or two in the future (or at least, the only people who use them will be the people who don't understand how interest works, which may not be the best place to look for Scott's desired superforecasters).
So I don't think that this is something to handwave away.
That also means that you need to tie up your money for that length of time. I agree that I'll "get a sum in 2121 whose present value in 2021 is $2," but doubling my real money value over a hundred years is still only a 1% real rate of return versus doing something active like (for example) owning real estate, where my rate of return is hopefully more than a 1% real rate of return.
The alternative that the post mentioned, where it's invested in index funds, means that you then get the 1% + stock market real rate of return. That is better, but it's still not great since you're limited by the risk and return of index funds. If my default hundred year investment is, for example, a leveraged index fund, then over a hundred year time scale that probably gets a better return than even a certain prediction + a normal index fund, since you're trading off risk.
"I think probably prediction markets will lose a lot of reliability and precision beyond a few decades. Here's a pessimistic way to think about this: even if we somehow found a way to make the incentives line up perfectly, could this perfect prediction market really predict things beyond a few decades anyway? "
You could potentially "fix" this issue by large institutional players with long term time horizons getting involved. If it seems like people are generally bad at predicting into the future, there could be another alternative investment class that arises for endowments, life insurers, etc. based on strong long-term predictions to take advantage of this potential inefficiency.
Maybe a more direct way to talk about the problem with long term prediction markets is to compare long term prediction to short term prediction rather than long term prediction to stocks.
If you can predict a 50-50 event 100 years from now correctly, you double your money (plus any increases in the index fund they invest in). If you predict a 50-50 event for next year correctly, you double your money (plus any increases in index fund) after only 1 year and can immediately reinvest in a new prediction market (or anything else).
I guess in equilibrium this means that the amount of work you should need to put in in order to do 1% better than average on a prediction market should be inversely proportional to the time until the payout. This means that markets with long time horizons will be relatively easy to beat the odds on (meaning that they are not particularly accurate).
I enjoyed this post, but I want to note that I feel a lot of skepticism about this line: "But if instead you put your energy into finding stocks that went up 6% per year on average, you would make 339x returns."
We should expect that consistently beating the stock market by 1% should be _really, really hard_ -- much harder than identifying a 5% mispricing in a very-long-term prediction market.
You can pretty easily beat the stock market as long as you are willing to take greater risks. For example, you can use leverage to invest in index funds and get better returns but at a much higher risk. For a retirement fund, you probably shouldn't, but for someone investing over a hundred year timespan, it might be the right move.
> We should expect that consistently beating the stock market by 1% should be _really, really hard_ -- much harder than identifying a 5% mispricing in a very-long-term prediction market.
How so ? If prediction markets are deregulated to the same status as stock market, (e.g. allowing large volumes of trades etc.), then how is betting on a stock different from betting on the probability that the stock will increase in value ?
If prediction markets are deregulated, what will prevent them from getting encrusted with all kinds of new and exciting financial instruments, high-frequency trading, and all the other accoutrements of the modern stock market ?
The idea of investing the collateral, so that it doesn't lose value to inflation over time, does seem like a good and obvious step. But there's another step you could take, which I think is almost as obvious: not requiring bets to be fully collateralized in the first place.
We do this with stock: you can buy stock "on margin", using the stock itself as collateral to buy more stock. This mostly works fine, because stock is decent collateral, which is probably not going to become suddenly worthless tomorrow (and if it declines gradually, there's plenty of time to ask the user for more money before it becomes an emergency.)
Obviously you want to do this in a way that doesn't allow people to mint free money at your expense. As a stockbroker, you know who your customers are, and at least in principle you can sue them if you have to. You almost never would, if someone gets wiped out in the market, even if they technically owe you money -- a bankrupt person has nothing worth suing for. But the possibility of lawsuits deters people from deliberately gaming the system with intentional defaults, for profit. (Plus that would be fraud, which is a crime.)
If you don't want to rely on the legal system, you can still extend some trust to users who have been around for a long time, and done a lot of betting. They can still build up lots of good reputation, only to use it to rip you off (in the cryptocurrency business, this is an "exit scam".) But you can bound the amount of risk you're taking on to acceptable levels.
And saving the best for last, here's one I _assume_ there must be papers about: you should always be able to use _independent_ bets as collateral for each other. Or, putting it another way, you should be able to share collateral between multiple independent bets. For example, if I place 100 bets, $1 each, on coinflips coming up heads, it should be obvious that $100 of collateral is too much to require. The probability that I will owe at least $60 is around 3%. The probability that I will owe $65 is almost negligible.
Lots of real-life predictions are only *somewhat* independent. A food shortage, a metal shortage, and an energy shortage are often caused by different things, but they can also all be caused by a single large event, such as a war. This makes the odds that you will lose on all of those bets at once considerably higher than if they were truly independent, in a difficult-to-quantify way.
Well, if you need a way to estimate the independence of various distributions, you have a whole site full of people to ask... it's prediction markets all the way down.
I'm mostly kidding. Those would be pretty hard to score. But you don't need the estimates to be perfect, they can be conservative. And some of them will be easier to make than others -- looking at Metaculus right now, I would say that the truth of the Collatz Conjecture is totally uncorrelated with ~all other questions on the site, for example.
And it seems like some questions may become less correlated (or more clearly uncorrelated) over time, which seems like it could be favorable if you otherwise want people to post more collateral as questions get closer to resolving, since it can partly balance that out. E.g. Tesla's 2025 vehicle production is not totally uncorrelated with various geopolitical questions not otherwise related to clean energy, insofar as there are major disruptive events that could impact both. But as time moves on and major disruptive events don't happen, that goes down.
If you’re the market maker you have equal exposure to YES and NO, so all you need to do is make sure the collateral you’re taking to place bets is matched by the other side. If 1 in 10 people in YES have TSLA as collateral but it plummets and they default, you’re fine because 1 in 10 people in NO also have TSLA as collateral, so you just wipe both of them off the book and continue.
Practically speaking, it’s probably simpler to just sell multiple tranches of each market for some popular investment vehicles (equities index, treasuries, cash, etc).
Sorry, of course that should be "the probability that I will _lose 60 flips_", and lose 65 flips, respectively. The amounts I owe would be $20 and $30 respectively, if I did the math correctly this time. The probability that I owe at least $60, which would require losing 80 flips, is absolutely negligible.
> The person who spends one second picking a random number on a thousand questions will get more points than someone who spends an hour researching a really good answer to one question.
While it sounds intuitive, I recall reading some studies that the "wisdom of the crowds" works better when people don't talk to each other or research the questions [1], as these are subject to various cognitive biases that can easily lead you astray.
So if the goal of a prediction market is to be more accurate, maybe one with the kind of incentives you describe might work a little better.
[1] Don't have time to lookup references now, anyone else have them?
Some references:
* https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.408.4760
* https://www.pnas.org/content/108/22/9020
I have a much bigger issue with the Metaculus scoring system than what has been mentioned here:
In Metaculus, it is possible to predict the probability of the event *completely accurately* and *still lose points* if you are unwilling to come back to the market and update your prediction every time new data comes to light.
This is because the Metaculus scoring rule isn't applied instantaneously - you make a prediction and then the score of that prediction is averaged over the entire course of the market from opening to close (see the integral in the Metaculus FAQ scoring section). If you do a great deal of analysis to determine the probability that COVID deaths in a certain month are 50% likely to go over a certain value, you make a 50% prediction, and then halfway through the month it becomes clear that deaths will go above that value and the market starts predicting close to 100%, you are penalized as if you had made a 50% guess on a 100% probability outcome every day for the rest of that month.
Like most of the complaints about the Metaculus scoring system, this is a feature not a bug. Metaculus wants to incentivise activity and as you've explained, this does exactly that
This is my big problem with Good Judgement Open too. There's no reason why they couldn't let you withdraw a prediction and only get points for the time while it was up - I think they only do this to try to incentivise people to update their predictions frequently.
Hey Scott, this is totally unrelated to this post but not sure how else to contact you. Just saying I used to put all your posts on SSC onto the Pocket app where I could read in dark mode and also easily sync to my kobo ereader, but these posts on substack don't work (if I put a post onto pocket it refuses to recognise it as an article). I thought maybe this was a deliberate design choice on the part of substack to sabotage their articles going on pocket as they wanted readers to use their own app, but I've since discovered substack blogs for which it works fine so it's definitely a solvable problem. It's understandable if you don't have the time/inclination to investigate this, maybe this is a rare thing to be bothered about, but I guess all I can say is personally I'd be eternally grateful if you looked into this as I really miss reading your posts on my ereader :)
Not sure if something changed since you posted this, but I just tried saving this article to Pocket and it got picked up correctly.
Really? I just tried again and definitely still broken for me. I think it's because I have a substack header at the top - it thinks every article is just the front page of ACT. Are you saving from a mobile device or desktop?
Did you contact Pocket? There might be some problem on their end too. And they could probably tell you what the problem is in more detail anyways. (They're friendly!)
I got a response now:
Based on previous reports we've received on Substack, it seems that the new Substack blog AstralCodexTen uses Javascript in their items in a way our Parser is not able to handle. This is the reason these items are not properly displayed in Pocket's article view.
Moving forward, our team is aware of this area of improvement in our parser and will make sure to let you know if we update our parser to support these types of domains. In the meantime, opening the items on Web view would be the best approach to view their content.
JavaScript is really a plague on the Web...
Nice follow-up!
What they wrote matches my own experiences with programming web scraping ("parser[s]") – it's tedious, error-prone, and that's even ignoring the arbitrary complexity of JavaScript.
A simple solution here is not to maximize for Metaculus points, but to maximize for average points per question. Some people have compiled the data necessary for this calculation.
https://www.metaculus.com/questions/935/platform-feature-suggestions/#comment-51338
https://docs.google.com/spreadsheets/d/1Fy7uqy_KgFAZhL_2br_O9_sV2vzki6Vzhzx-xvJ1_b4/edit#gid=0
This value needs some shrinkage to remove luck factor for small n (as done in baseball stats http://varianceexplained.org/r/empirical_bayes_baseball/), but otherwise, it is pretty nice. My average score is currently sitting at 31.5, putting me at the 36th spot according to that spreadsheet. One cannot just mindlessly spam median predictions to maximize this value, one has to predict on questions where one will beat the market.
This just returns to a zero-sum scoring, though. If I'm understanding the metaculus article Scott linked ( https://metaculus.medium.com/a-primer-on-the-metaculus-scoring-rule-eb9a974cd204 ) right, that could easily be achieved with some different scoring rule or parameter choice. Metaculus wants to incentivize you to bet on more questions than just those where you're reasonably certain you can beat the market.
The positive EV of the current system is a deliberate feature, not a bug.
https://metaculusextras.com/points_per_question I also maintain a ppq ranking. (Yours seems out of date).
There are ways to game this metric as well. You don't need to "beat the market", you could just choose to only predict on questions which are in the 0-10% and 90-100% range.
Cool. The one I linked to is a static copy, not one that I made. Your site does not have me anywhere as far as I can see. Maybe too many of my questions are private ones which have no cheating-prevention.
The ones in the extreme range would only produce high gains if one actually gets them right a lot of the time. Do you have data for average points per question, by median community prediction at closing? One could simply adjust for this statistically if there's some bonus from their algorithm.
https://metaculusextras.com/ppq_chart chart stolen
I limit it to people with >4000 points. (I'm considering stealing your chart though, and I'll include the full top 400 people on that)
I only have the publicly available data, so I can't do anything like that.
> gold is currently worth $1700/ounce, and 20% chance for $2500 by 2025 seems plausible
In fact, that's very plausible. You can look up options prices for gold based on the end-of-2024 futures contract. At the moment of this writing, the end-of-2024 call option with a strike price of 2500 has a "delta" of about 0.25, giving a market expectation of about 25% that gold will be at or above $2500/oz in December 2024.
If the Foresight exchange operated on real money, there might be an arbitrage possibility there.
and a different arbitrage possibility if the Foresight Exchange operated on real bullion
Notice however that in both cases this isn't a prediction that gold will be worth 20-25% more by 2025 because of the time value of money. Though the arbitrage opportunity would still be there.
“But if instead you put your energy into finding stocks that went up 6% per year on average, you would make 339x returns”
Your accounting assumes that someone has an edge in equity investing but fails to account for any edge they might have in the prediction market. Presumably if they’re investing in the prediction market, they think they have an edge there.
All you need to make the prediction market work is the existence of people who think they have an edge in prediction markets but not in equity markets.
I think the deeper problem is that the proposed index-fund scoring system is no longer proper if the equity markets might depend on the outcome of the bet itself. For example, a bet of "societal collapse by 2030" would probably also result in a collapse of equity markets, so the payout on a correct "yes" bet would be trivial.
Discussion of this issue below https://astralcodexten.substack.com/p/mantic-monday-scoring-rule-controversy#comment-1389738
Re the abortion situation:
The conservatives overplayed their hand really, really badly this fall - even people responsible admit they lost the cause.
https://www.theguardian.com/world/2020/oct/30/pro-choice-supporters-hold-biggest-ever-protest-against-polish-government
Public support for both the ruling party and the Catholic Church took a nosedive, and the delicate Schelling fence of the 1993 "abortion compromise" (itself very close to the Catholic position) got completely wrecked. The law briefly became extremely restrictive, and it is expected that once the government changes it will become more liberal than ever, as most of the opposition aligned itself with the pro-choice protesters.
The major opposition party was previously playing a balancing act a la median voter theorem - not appearing too anti-church nor pro-church to avoid alienating anyone. Now they were forced to take a side.
Would index fund theory work on a true prediction market? Like naively I think you could make money on the bet that “prediction markets are good at predicting” by just betting with the house odds on every bet and periodically readjusting. Is this dumb for some reason I don’t realize?
To expand on what boog said, buying an index fund is buying shares of companies that have managed to get listed in an index, which tend to stick around and appreciate in value because they're producing something of value. A prediction market just reallocated money from bad or unlucky predictors to better or luckier predictors. It would need some way of not being zero-sum, the way a real economy is not zero-sum, for an index fund to produce a positive expected return.
You could create an index of _predictors_ and that might consistently beat the overall prediction market.
In an ideally functioning prediction market, the expected value of buying YES and NO is equal. If one is less likely, it cheaper to buy and has better returns, with the other therefore being more likely but with worse returns. (If this isn’t the case, people are incentivized to trade in a way that corrects the price so that it is.) So, you would expect to overall get exactly the same about of money you put in (minus fees).
Index fund theory *works* here—the problem is just that the rate of return for the entire index is guaranteed to be 1. For a total US stock market index, we have a historical expectation that the entire index will have an annual return of ~1.07 over a long enough period of time.
Thank you for this!
That depends on whether prediction markets are, in fact, good at predicting.
In 1996, gold was $369 an ounce, so it must have seemed like a safe bet that it couldn't inflate so much that it would be worth $2500 in 30 years. (It would be an average of 7% inflation a year, which is bad but not exactly hyperinflation either.)
Gold prices skyrocketed around 2007 and never really came back down, even though inflation stayed level throughout the financial crisis. So I guess it's not as good a measure of inflation as they thought.
On the other hand, in 1966 gold was worth $35.50 an ounce, so it had already gone up 10x over the previous 30 years without anything approaching a US government collapse.
The idea that gold might go up another 6x over the next thirty years shouldn't have surprised anyone.
Or else measures of inflation are more heavily manipulated than the gold price is.
Numbers like CPI are Goodharted to hell, and so an actual inflationary trend may be masked by people's political incentive to make the widely-accepted, legible numbers look good. (Cf. legible expertise.) Meanwhile, there's definitely manipulation in the gold market, but there's a limit on how far it can push the price from that implied by actual underlying supply and demand.
Note also the Cantillon effect of monetary base inflation as separate from CPI inflation. The notional goal of programs like quantitative easing is to manipulate financial markets in ways the Fed considers desirable, while keeping the excess money from leaking out into the broader economy too much and causing CPI inflation. When it's successful, this kind of program means that financial assets all go up enormously while everyday goods stay relatively flat. (This also greatly exacerbates the wealth inequality between the asset-owning class and the wage-earning class.) In this context, gold is largely a financial asset and specifically one held as a hedge against inflation, so it gets the full benefit of the excess money in asset markets.
CPI overestimates inflation rather than underestimates it, to the tune of north of 1% per year cumulative.
This is why wages have been "stagnant" since the early 1970s while standard of living has gone up dramatically.
The idea that "standard of living" has "gone up dramatically" is certainly one that it is possible for a person to have.
For the record, "you’ll gain points on the prediction even if you were maximally wrong" is not true.
Taking the first open question on the main page: https://www.metaculus.com/questions/6330
If you predict the maximum 99% and the event doesn't happen, you **lose** 351 points.
If you predict the minimum 1% and the event happens, you **lose** 353 points.
If you believe the event is 50% likely (not sure why would you believe that, but arguendo), you expect 50%*13+50%*18 = 15.5 points.
If you believe the community that the event is 45% likely, you expect 55%*29+45%*1 = 16.4 points.
If you believe the event is 75% likely, you expect 75%*77+25%*-84 = 36.8 points.
etc.
Importantly, the scoring rule means the point-maximising strategy _on any question_ is to predict your true credence.
This is separate from the point maximising strategy _on the whole site_. But as Scott points out, nobody does that, because that'd be unbelievably boring. Notice that the best users in absolute number of points have much more than 15 points per question (which is what you typically expect by predicting the community median). https://metaculusextras.com/points_per_question
Does Metaculus still need to exist in 100 years in order for the bet to pay out? Because in that case, you also need to multiply any win chances with the low percentage that this will in fact be the case.
Hmmm – how has this worked for past exchanges? Do they always 'blow up' when they 'die'? Or can they also be wound down, particularly in such a way that transactions can 'migrate' to other exchanges while that's happening?
I have a vague intuition that everything _could_ work fine, even if particular prediction markets come and go over time. It's the ongoing trading that's valuable, even apart from the 'final judgement'. (Regular financial equities don't seem that different. You can, of course, make a profit off of other activity related to an equity apart from making an accurate prediction about the entire path of any equity's 'true value' over its entire lifetime.)
Or whatever company is used.
I aggregate the top comments on Metaculus (by likes) and share them in the discord. I'm going to continue sharing them on Mantic Monday until someone asks me to stop.
https://metaculusextras.com/top_comments?start_date=2021-02-22
Last week's top comments:
- Users are planning on living forever (https://www.metaculus.com/questions/6046/date-1-bitcoin-worth-1-million/?invite=tLxPdB#comment-56474, https://www.metaculus.com/questions/3399/when-will-the-last-metaculus-question-resolution-occur/?invite=tLxPdB#comment-56677, https://www.metaculus.com/questions/6046/date-1-bitcoin-worth-1-million/?invite=tLxPdB#comment-56420)
- Vaccine news: FDA gives EUA for J&J and Moderna manufacture a vaccine for a variant (https://www.metaculus.com/questions/6492/will-the-jj-vaccine-be-issued-an-eua-by-fda/?invite=tLxPdB#comment-56495, https://www.metaculus.com/questions/6007/vaccine-update-due-to-mutation/?invite=tLxPdB#comment-56587)
- Galen (the color whisperer) is back to let us know what's in style (https://www.metaculus.com/questions/5887/pantones-color-of-the-year-for-2022/?invite=joX2Fj#comment-56370)
- I explain the logic of why people are expecting a new SCOTUS vacancy soon (https://www.metaculus.com/questions/6634/matthew-yglesias-predicts-2021/?invite=tLxPdB#comment-56297)
- I make the bear case for Trumps on substack (https://www.metaculus.com/questions/6423/trump-substacker-before-2024-election/?invite=tLxPdB#comment-56424)
- aqsalose calculates the base rate of for new Russian PMs to calculate Navalny's chances (https://www.metaculus.com/questions/3416/alexei-navalny-to-become-president-or-prime-minister-of-russia-in-his-lifetime/?invite=tLxPdB#comment-56493)
- I lay out my estimate for a Scottish referendum being announced in 2021 (https://www.metaculus.com/questions/5922/scotland-independence-referendum-in-2021/?invite=Olx7Ci#comment-56410)
Maybe instead of betting on outcome there should be traceable binary options? You can totally imagine options with 100y. Their current trading price will determine the odds and will also handle time value (I.e. price might change if less time remains for something to happen)
The question is: who would sell you such an option? “Market makers”?
I assume by "traceable" you mean "tradable".
The problem with this is that you have to put up capital. Say I want to bet $1 on event A to happen in 100 years which the market says has probability p. How much does it cost me to do so? There are several options:
- Put up p * $1 at the start. Never see the money again until you trade out. This has all the problems discussed. You're giving up lots of time value of money / alternative return sources etc
- Put up some smaller amount as margin. Pay margin whenever the price moves. (This is roughly how most futures and stock markets work). The issue with this is that for a *binary* event the probability can jump. So how much margin should I require from you so that when the probability moves I know you will pay your margin call or I can liquidate you? Ans: you have to be fully collateralised, since p can jump to 0 or 1 whenever the event occurs (or doesn't). There are some cases where this would work (things which resolve on a specific date AND can't be known before hand) but for lots of interesting questions this jumping problem is a death-knell for margined prediction markets
I still not sure where the problem is. Here’s how I imagine it could work:
Imagine an option that pays off $1 in 2121 on R winning the presidency.
Market Makers can hedge their risks by selling options against each other (R+D options portfolio constitutes a constant payout to the first approximation).
So, today, MM could sell options for the discounted value of payoff expectation. If, e.g. $1 in 100 years = $.01 now, then both R&D options will be half a penny.
50 years from now, both time value will be higher, and probabilities will change => you can sell your option for higher value and make some money before 100yr.
As for “fully collateralized” - I don’t think it is a requirement. Rather than being fully collateralized, MM keep low risk portfolio and move inventory to keep it that way. I.e. if more people will buy D options, they will lower price for R option to incentivize more people to buy it. Which is how market discovery works.
Of course this implies that MM can put your 1c to work over 100y horizon, but I feel that it is reasonable to assume stable investment vehicles to exist if you want long term prediction market to exist too.
I think you need to lay out more exchange mechanics than you have and then you will see the issue here.
Who can "create" these shares? If it is the exchange (creating a D+R worth $1 in 100 years) then they are implicitly issuing a 100y bond. The interest rate on that bond is going to be a combination of their credit worthiness and 100y rates. If it is the MM then that's the same issue (except now it really matters which MM issued the options since the credit worthiness will vary from trader to trader).
How are you forcing MM to keep "low risk portfolios"?
"Which is how market discovery works." Sort of? Maybe? I really don't think so.
"Of course this implies that MM can put your 1c to work over 100y horizon" well now you're exactly back to the index fund suggestion...
What are the odds that Metaculus still exists in 100 years?
30%
https://www.metaculus.com/questions/3399/when-will-the-last-metaculus-question-resolution-occur/
Very obvious selection bias problem is that if the market no longer exists, a winning bet is worthless, so greater incentive to bet on it existing
...as opposed to the extremely valuable Metaculus points you earn if it keeps existing? ;-)
(Anyway, the Metaculus people know that -- see the last paragraph in the description of the question in my other comment)
BTW, that and https://www.metaculus.com/questions/841/will-metaculus-exist-in-2030/ together imply 10% chance that Metaculus is operating on Jan 1st 2030 but there will be no question resolution after then, which sounds absurd to me, but I can't be bothered to arbitrage that away right now.
If I understand correctly, the dream is free and open money based prediction markets, but we can't have that good thing yet for US legal reasons.
If that's the case, is there a reason we're not seeing money based prediction markets in other countries with different rules?
Surely there's incredible incentive to get this started in places with large populations and laxer laws.
https://smarkets.com/ is a UK based one. Betfair exchange is another one https://www.betfair.com/exchange/plus/
Both have a limited number of questions that can be bet on. And seem to legally operate like other bookies
There are actual concerns with free and open money-based prediction markets. The market for "President Biden will be assassinated before Jan 1 2022" will create some perverse incentives. Or if you're using it for informing government, like Scott suggested in the recent GOP post, you have to ensure that, say, Exxon-Mobile doesn't pour billions into the "Global warming doesn't exist" market in order to ensure that the government won't regulate them. Even if they're just used for informing public opinion, you'd have similar incentives to dis-information (and if everyone believed Exxon-Mobile would manipulate the market, they might play along to make a buck, adding to the misinformation, depending on how things operate).
Is there literature on addressing concerns like these? Apparently the first problem is at least partially addressed by that crypto-currency market by including a "invalid/unethical" potential result.
I think the answer is probably something like 'this is a fad only among a small class of primarily American intellectuals, no other country has noticed/cared about the idea enough to put in the effort and capital to make a large one.'
But I could be wrong.
> "which party will win the 2100 presidential election?" might change a little over the next 80 years as (for example) demographic change makes one or another party permanently stronger or weaker
Duverger's law makes demographic change, or even what issues the election is fought on, irrelevant. So long as the USA keeps the same electoral system, there will be 2 parties and each will have a roughly 50% chance of winning the election.
So, expert forecasters: What are your odds on who will win the 2120 baseball World Series?
My forecast is a 90% chance of "no one." I doubt baseball will be around in 100 years, and if it is there is a very good chance that the name of the championship is changed. Replace "World Series" with "highest level professional baseball championship" and I put it at a 75% chance of no one.
The championship has been called that for the past 118 years, so a 60% chance that it changes in the next 82 seems pretty high - I don't think there's a particularly strong tendency for sports championships to change their names?
You heard it here first: the Denver Dodgers.
odds?
1:212.
I remember late 2019, a lot of the SSC commenters had near certainty on who would win the 2020 election. Six months later, they all changed their mind, saying covid completely changed their predictions.
100 years is a long time, we are likely to experience at least one covid level event between now and then.
Anyone who was near-certain about the elections in 2019 had problems beyond just not predicting COVID.
It was more than a few regulars in the OTs. Am I remembering correctly?
Metaculus spammers do hurt accuracy though! Suppose you put up a new question and the smart parts of the market think it's 35.8% yes. The spammers show up and reinforce that to an extremely confident 35.8%. Then some new information comes out that makes a yes look almost guaranteed. If the market was only smart participants it would shoot up to ninety-some percent, but the spammers (who don't bother to update their predictions because the whole point of spamming is that it's a low-effort strategy) drag the market's estimate way down.
The community median massively overweighs recent predictions, so moving it isn't that hard.
Does it overweight recently-changed predictions in a similar way?
Yes!
What's the actual formula for the overweight-recent function? I'm curious how it acts if many automated predictors each make a small update after every human update.
The details are here: https://www.metaculus.com/help/faq/#community-prediction
If the bots predicted to median value whenever it changes and then waited for the median to change again, I believe it would make the median slightly harder to change, but you'd still need ~sqrt(predictors) new predictions to change it.
But isn't the whole point of spamming to increase your score by answering many questions without researching them? Wouldn't this strategy likely include frequently updating your predictions, since updating to the crowd score regularly would increase your score without increasing your need to do further research?
I can see how such a strategy starts to sound high-effort pretty fast unless one can auto-update to the crowd score without looking at it, which would resolve the problem you pose.
Does anyone know if these spammers really exist, or is this all theoretical? Because a spamming strategy sounds awfully boring. What is the incentive to do it? To win fake internet points in a game you have little interest in playing?
70% chance the officer who killed George Floyd will get acquitted *of the murder charges*. This is important - the difference between complete acquittal and only acquittal of the most grave charge is quite substantial.
Thanks, fixed.
I love Mantic Mondays
I recently learned about Long Bets (https://longbets.org/), which tracks predictions and bets between individuals to be resolved many years in the future. Started in 2003, and originally funded by Jeff Bezos.
The winnings go to charity, and the money is invested in the meantime, with the gains being split between the foundation that runs it and the winner's charity.
Yeah, though I think they only accept 50:50 bets, which is a pity
Since you posted this, the Tether prediction has gone from 36% to 16%. While this may be in part due to the number of predictions doubling, it's interesting what it says about the attention of the market. On the other hand, no one here seems to have had an option on the Polish abortion rate. (The George Floyd question is already closed.)
Someone once said something like "Wikipedia ensures that the page on flatworms is written by people who really love writing about flatworms, and the page on the Roswell incident is written by people who really love writing about the Roswell incident".
The general rule here is that, for certain objectively-not-very-significant topics, there's a vocal crank position that says that the topic is actually extremely significant and the truth about it is <insert weird crank theory here>. If a system is set up so as to produce an aggregated opinion on a topic in such a manner that the people who spend a lot of time contributing to it have disproportionate control over the result, then you'll end up privileging such crank theories, because their proponents have more time to devote to promoting the theory than their (mainstream) detractors have to devote to debunking it. (This is adjacent to the general problem of obscure Wikipedia articles, where often the article about some company will be effectively written by PR agents to that company because no one else cares; or, conversely, some figure with a big Internet hatedom will have his page taken over by the hatedom and turned into an obsessively curated shrine to everything he ever did wrong.)
One advantage of an actual prediction market, rather than a nothing-at-stake scoring system like Metaculus, is that there's a built-in incentive to challenge cranks like this: you can profit off their being wrong. I don't have enough information to say how significant this really is to the operations of the market.
I think the crank theory has a lot of merit here. The people most interested in the question "is Tether going to collapse" are the people who have theories about the collapse of Tether. Everyone else probably wasn't paying much attention to the question, until Scott linked it with a surprisingly high number attached.
Disclosure: I'm a frequent predictor on Metaculus and it's hard in cases like this to not have my thoughts be dominated/emotionally compromised by naked tribalism.
That said, I thought, and continue to believe on reflection, that both Zvi's critique and Ross' writeup missed what I think are [fairly obvious points](https://twitter.com/LinchZhang/status/1360300621782392834). The primary point of Metaculus and platforms like it is to aggregate truthful information about the world and present it in a way that's collectively digestible, not sample-efficient ways for to get perfect individual rankings.
Obviously both desiderata are useful, and there's a case to be made that maybe the current Metaculus system rewards systematic accuracy and legibility too much over individual accuracy and legibility. But I think any reasonable critique should've been framed much more like "we think you're making the wrong tradeoff on this spectrum because of XYZ" rather than "my point on this spectrum is obviously correct and anybody who says otherwise is a moron." Worse, I think Zvi et.al implicitly aren't even acknowledging the tradeoff at all?
The point on "participation trophies" seems fairly off the mark. Whenever I talk to people who think about how to improve forecasting, one of the most common points is that "the problem isn't that people just or primarily make inaccurate predictions when they make precise numerical answers to well-defined questions, it's that they don't make much if any forecasts at all." To the degree that points incentivize behavior, having participation points is good because it incentivizes more forecasts, since if you want to get more of something, you should incentivize it.
In general, I feel like the critique seems more like "You claimed to be doing X. You did X, which is not like Y. I expected this to be Y, without any justification for why Y is better. Now I'm going to judge you on the metrics of Y, rather than X."
I was politer on Twitter when this kerfuffle first came out and I was deliberately trying to adjust for emotional compromise, but upon reflection I'm inclined to think that this type of motivated ignorance is somewhat suss.
Metaculus's system just rewards the oldest and most active users of the site. There's little point in new users trying it.
I think it is difficult to express this in a way that is credibly sincere, but: Thanks, I appreciate this criticism. I think it helps me understand where our intuitions are going in different directions, and hopefully will help me to put my intended points more clearly in follow-up posts.
I also think I made mistakes in my original critique, because I do not believe (and did not intend to imply) that "anybody who says otherwise is a moron", and I'm disappointed that it reads as if framed that way.
On the substantive point: My core critique is *not* that the current scoring function does a worse job of accurately ranking individuals. (I understand that ranking individual predictors is not the primary desideratum.) Unfortunately, I don't expect that this comment thread is the best place to re-attempt an explanation of the concerns I actually have.
Thanks for interpreting my comment charitably. I now think I should have read your original post more carefully. I'll reread it again and I'll revise my comment if needed (or issue an appropriate retraction if I no longer stand by the sprit on a careful rereading).
I agree with you Linch. I read Zvi and Ross' posts and found both of them to uncharitable and frankly disrespectful. Metaculus isn't some part time senior project with 3 users. I don't use Metaculus but I think it already provides great value and I am incredibly excited for its future. The primary reason for that is Metaculus, unlike most every forecasting website, actually has an active community. Their scoring system plays a huge role in that.
I found Ross' critique's [1] major points interesting and worth contemplating. On the other hand, it started with this:
"In Zvi’s 2/11 Covid update, he turned to Metaculus for help. He looked at the numbers. Becase the man is an inveterate trader, he saw odds that were Wrong On The Internet and just couldn’t stop himself from creating an account to bet against it. And then he saw the absolutely bonkers payout structure and decided he was done after making a single prediction.
Seriously, I’m not even making this story up.
I spent ~20minutes with the Metaculus site and figured out how they borked this one up badly enough to drive away Zvi Mowshowitz, of all people. Anyway, because Zvi reportedly “can’t even”, and because I’ve already done the legwork, I’ll try to explain it here."
I was inclined to stop there. Lionizing certain people and their opinions with informed quality how they are obviously awesome in order to put other people down ... it is a bit uncouth style of argumentation.
[1] https://blog.rossry.net/metaculus/
My prediction: if Prediction Markets ever get to the point of relevance, Wall Street will end up being the only ones making significant money on them. It's the hedge funds, with their billions of dollars to devote to computing power and market research, that would end up being the "superpredictors".
I agree this is true, though more because the hedge funds can just hire the superforecasters. I'm not sure if it will be literal Wall Street or some other group of companies centered somewhere else, but it will probably happen. This is my best-case scenario - it'll mean we have really competitive profit-focused companies working on predicting important things for us with an incentive to get them right.
Probably random people who are good at predicting can still do it in their basement, the same way people can still trade stocks on Robin Hood.
Sports betting is the most liquid prediction market in existence (assuming you don't count the stock market, which I think is fair). Market makers make a lot of money, sophisticated syndicates make a lot of money. Some individuals also make money in various ways, including doing the same things the syndicates do. Part of the reason they can is that the market makers limit their exposure to the syndicates. Final betting lines are extraordinarily accurate.
The issue here is the Praeto Rule. In any competitive/creative arena the square root of the total participants, call them first derivative, will take half the earnings. The square root of the census in first derivative will take half of the earnings of that group.
What that means is that out of 10,000 people if everyone keeps playing 10 people will end up with 25% of the money. In any competitive endeavor in the history of the world this effect plays out.
Isn’t the stock market already the prediction market for the only questions worth betting large amounts of money on (namely, which companies will be making money and how much in the long run)?
Yes! But we _could_ expand that set of questions.
"Polish abortion rate will dectuple by 2030 (why? they seem pretty conservative; is their conservatism that precarious?)"
Yup:
https://www.unz.com/akarlin/poland-will-legalize-gay-marriage-within-10-years/
Conservatism of Polish society is waning. However, politically it's on firmer ground than almost anywhere near the western world. That's because our ruling conservative party has an unusual pro-welfare bend, and the main opposition party are committed (European meaning, i.e. classical, free-market) liberals. Unless things change (and they could, the liberals are now losing support to a new amorphous, but potentially more socially sensitive centrist option, and we still have some nominally left-wing parties left), people will likely continue choosing social conservatism for the increased material stability it is packaged with.
What happened to the US abortion rate (legal ones that show up in the statistics) after roe vs wade?
Also it's abortions in Poland, not by Polish nationals. I'd not be surprised if Poles were getting abortions in Germany or Czechia.
It looks like New York allowed abortions generally before Roe, and three other states had some situations where they allowed them, so depending on how much people were able to cross state lines to do it, there might not have been as big a change in the legal picture as there would be in a place with uniform national laws.
It looks like statistics weren't counted before 1973, but abortions doubled within the next few years, and then have gradually fallen to below 1973 levels (presumably as a result of more birth control options, among other things).
https://en.wikipedia.org/wiki/Abortion_in_the_United_States#Number_of_abortions
Religious conservatism and secular conservatism are different movements, with the attitude towards gay marriage perhaps being a clear difference marker. So a strongly conservative society can still legalise gay marriage with an alliance between secular (and to be fair some religious) conservatives and the liberal minority.
The same alliance can support abortion as well, although as Deisarch says it's a long road. Conservatism need not be religious (witness Trump for a very clear example), and I doubt any western country does not have a potentially significant number of secular conservatives, even if it doesn't have an organised movement.
If Poland is like Ireland, there will have been activist campaigns nudging and nudging and nudging on this for years, plus some foreign foundations sticking their spoke in. We now have legal abortion (hurrah 😐) after a couple of referenda and decades working away at it. Constant dripping wears away the stone, and when it goes, it goes fast. https://en.wikipedia.org/wiki/Abortion_in_the_Republic_of_Ireland
The Irish situation is a bit disanalogous, since so many women went to England the notional ban didn't have the impact it might have otherwise had. (Especially on middle and upper classes who are most likely to be politically active) If there wasn't that option it would likely have been changed much sooner
I just looked up abortion statistics in Ireland. Quite wonderfully, there were 6,666 abortions in the year after legalization.
and I have a small sample size, but the Polish progressives I know seem unusually competent. I have much more hope for a liberal turn there than in other right-leaning Eastern European countries (think Hungary)
The current near-total ban on abortion in Poland is based on the constitutional court's ruling that Article 38 of the Polish constitution (which says simply "The Republic of Poland shall ensure the legal protection of the life of every human being.") includes fetuses, which means that re-legalizing abortion would require either a new ruling by the constitutional court overturning the recent precedent, or an amendment to the constitution. The court's justices serve for 9-year terms and are nominated by the lower house of the parliament, and almost all of the currently serving justices were nominated by PiS (the socially conservative party which has controlled the parliament since 2015), so it would be difficult and take a significant amount of time for the liberal parties, if they got back into power, to replace enough of the justices to have a chance at overturning the current precedent. An amendment to the relevant part of the constitution (as described in the constitution at http://www.sejm.gov.pl/prawo/konst/angielski/kon1.htm ) would have to be passed by a vote of 2/3 of the lower house of the parliament, then a majority of the upper house, then a majority of the voters in a referendum, which would also be difficult since (according to the polls summarized at https://en.wikipedia.org/wiki/Abortion_in_Poland#Public_opinion ) the majority of the people support abortion only in unusual circumstances (in cases of rape, incest, danger to the mother, or probable disability of the child) and only a rather smaller minority supports abortion on demand. So it seems unlikely that abortion in general will be legalized in Poland in the near future.
The constitutional court, with its current illegaly chosen set of judges, is not binding - at least according to one side of the controversy (to which I belong and so does half of the country). Once the government changes it will probably attempt to throw their rulings out of the window ASAP.
Constitutional articles are no guarantee. If you put them in, you can take them out. In 1983, after a referendum, we had the Eighth Amendment to the Constitution enacted:
"The Eighth Amendment of the Constitution Act 1983 was an amendment to the Constitution of Ireland which inserted a subsection recognising the equal right to life of the pregnant woman and the unborn. The Amendment inserted a new sub-section after section 3 of Article 40. The resulting Article 40.3.3º read:
The State acknowledges the right to life of the unborn and, with due regard to the equal right to life of the mother, guarantees in its laws to respect, and, as far as practicable, by its laws to defend and vindicate that right."
Well, then we had further referenda, some based on test cases:
"Three referendums were held in November 1992. The Twelfth Amendment of the Constitution Bill, 1992 sought to exclude "a risk of self-destruction" as grounds for abortion, to overturn the central element of the decision in the X Case. This was rejected by 65% to 35%. The Thirteenth Amendment, permitting travel to obtain abortion in another jurisdiction, was approved by 62% to 38%. The Fourteenth Amendment, permitting information about services in other countries, was approved by 60% to 40%.
After these amendments, which became law on 23 December 1992, Article 40.3.3º read,
The State acknowledges the right to life of the unborn and, with due regard to the equal right to life of the mother, guarantees in its laws to respect, and, as far as practicable, by its laws to defend and vindicate that right.
This subsection shall not limit freedom to travel between the State and another state.
This subsection shall not limit freedom to obtain or make available, in the State, subject to such conditions as may be laid down by law, information relating to services lawfully available in another state."
And in 2012, using the case of Savita Halappanavar (one which got global attention and which was more complicated than the soundbite "she was refused an abortion and died"), the activists started up again with the Repeal the Eighth Amendment campaign, which eventually came to flower in another referendum in 2018 and the replacement of the clauses above:
The Campaign to Repeal the Eighth Amendment has its roots in the unsuccessful Anti-Amendment Campaign in 1983. There was a campaign to Repeal the Eight Amendment after the X Case in 1992, and the three abortion referendums which followed it (the 12th, 13th and 14th). The campaign lay dormant for more than 20 years until the death of Savita Halappanavar in 2012.
On 14 June 2017, the Taoiseach Leo Varadkar announced his government's intention to bring forward legislation to facilitate the holding of a referendum on abortion in 2018.[50] On 9 March 2018, debate began in the Dáil on the Thirty-sixth Amendment of the Constitution Bill 2018, which would replace the current provisions on Article 40.3.3º with the following clause:
Provision may be made by law for the regulation of termination of pregnancy.
On 25 May 2018, the Irish people voted by 66.4% to remove the Eighth Amendment, choosing to replace it with the above text as a part of the Thirty-sixth Amendment to the Constitution, permitting the Oireachtas (parliament) to legislate for the regulation of termination of pregnancy. When it was signed in to law by the President of Ireland, it superseded the 8th, 13th and 14th amendments."
This has been the pattern of social liberalisation in Ireland; referenda after referenda on divorce etc. until the activists get their way, then suddenly it's "the people have spoken, this is the law, can't be changed now". I have no more hope for Poland regarding their constitution than for my own country, because the activists will push and push until they get their way. And they will use 'hard cases' (even though those make 'bad law'). The best present the pro-choice/pro-abortion movement in Ireland got was the death of that unfortunate woman because they were able to ride "see, we told you lack of access to abortion kills women!" all the way to victory.
Assuming no justices die/resign until the 2023 election, an opposition win in said election will allow them to replace at least 8 of them (a bare majority) before the end of the parliamentary term. It's also possible they would simply aim for a more complete overhaul of the whole institution, since due to the 2015 events ( https://en.wikipedia.org/wiki/Constitutional_Tribunal_(Poland)#2015–present:_Polish_Constitutional_Court_crisis ) they view the current Tribunal as illegitimate.
There also exist options to significantly ease abortion restrictions without messing with the constitution, e.g. decriminalization.
Either way, I feel the more relevant question in this case is not whether the opposition as it currently exists would be able to change the law in a significant manner, but whether it would be willing to. (Which, unlikely, though less unlikely than just a few years ago. The massive reaction against the ongoing pro-life offensive is pushing even the conservative parts of it into a pro-choice camp.)
So with regards to the search for Metaforecast.org, I am taking exact matches and ranking them by starts and then by number of forecasts, and then I'm using https://fusejs.io/ for the rest.
The code for this is open source, and can be found here: https://github.com/QURIresearch/metaforecast-website-nextjs/blob/master/pages/index.js, lines 120 to 189. The comment on line 130 refers to the fact that, because Guesstimate contains too many models (17k), I piggyback on its search rather than loading the models on initial page load, and I have to do this asynchronously.
Pull requests or suggestions are welcome!
As an addendum, Metaforecast also searches forecasting platforms (Metaculus, Good Judgment, etc.), not only prediction markets.
As for the issue that "I don’t think Metaculus considers accurately ranking people by predictive skill an especially high priority compared to actually important things like getting good predictions, and that’s fine, but it kind of sucks for people who were motivated by a competitive streak"... Metaculus really does care about assessing accuracy for use in weighting predictions in the aggregation method. We also want to give users what they want, and many of them want to compete with each other, so on the drawing boards are plans for an upgraded "leaderboards" page that allows more comparisons. The tricky thing is we don't want *too* many comparison methods, because then you'll always have ways to do particularly well at some of them. And if we make any of those methods based on non-proper scoring then we have to worry that people will have incorrect incentives. The basic problem is there's no such thing as a single quantifier of "predictive skill" so we have do think a bit about a few measures that seem especially valuable.
Using stock market returns to keep prediction returns competitive is inherently flawed, in that it distorts incentives on predictions where stock market returns are lower. If you have a prediction regarding future interest rates, predicting 'Yes' on higher rates becomes a problematic and distorted hedge on stock market returns - if rates increase, stock values drop and the 'Yes' bet becomes inherently less valuable. You might claim that the efficient market hypothesis will correct for this, but then you'd have to prove that the prediction market is inherently efficient.
High efficiency is a sort of already an assumption of any prediction market—if it wasn’t efficient, the predictions would be bad. So requiring that seems in a sense trivial.
For your example, doesn’t the ‘no’ bet drop proportionally as well, maintaining the symmetry? Not sure if that’s part of what you’re referring to by requiring efficiency.
Prediction markets can be semi-accurate without being efficient, as far as I understand it. We aren't just talking about using prediction markets to make money, but using them to accurately predict future events. If you index prediction returns to stock market returns, any prediction that, if true, could result in reduced stock market returns will be commensurately undervalued compared to its accuracy. Does that make sense, or am I confused on this?
My concern about making a prediction about something in 2121 is I have little confidence the prediction market will still be around and functioning then. A lot can change in 100 years. How is this risk dealt with?
I'm turned off by the long-term prospects of prediction markets simply because many of the questions I most want to see predictions on are "heads I win - tails we all lose", getting rid of any incentive to vote tails.
Let's say you want the market to estimate the probability of a world war, hyperinflation in the US, The Singularity, any political change causing major revisions of property rights (h/t David Friedman on another thread), and other similarly significant events. If any of the above actually were to happen, the chance that Metaculus is still around to give a meaningful payout would drop precipitously. And that means there's significantly less of an incentive to bet that those things will happen, even if you actually think one of them is likely to.
That severely damages Metaculus' ability to speculate on matters like X-risk, where I most wish we could benefit from pooling the collective brainpower and financial self-interest of thousands. And I don't see a fix, besides hoping people's honest (or vanity) will outweigh their self-interest.
Yeah, it seems to be a widespread misunderstanding that prediction markets will converge on a true probability. The theoretical fair value of a contract is not just the expected payout, it's the expected value of (payout) * (value of money at maturity). If there's a substantial correlation between the value of money and the payout--e.g. bets on the world ending or inflation, etc.--then the value of the contract could differ greatly from what its probability suggests.
On top of that, there's also the issue of risk premiums. Generally, people are risk-averse (e.g. concave utility functions) and they'll pay to hedge against bad outcomes. What this translates to is that prediction markets will tend to overstate the probability of bad things happening.
It's also very possible to go the other way, where it's like, "What you're predicting is the end of scarcity, where we all are octuple-billionaires. Even if I think it's relatively likely, I'm betting against it, because I don't care about having another $10,000 in a post-scarcity world."
One solution (originally proposed by Eliezer Yudkowsky) is to have the world destruction skeptic give the believer $100 to spend today. If the world doesn't end, the believer has to give the skeptic $200. That gives everyone the right incentives, but it might be hard to implement on a prediction market.
Now to find a way to actually secure the payment from the believer who spent all money on hand on booze and hookers in anticipation of the impending doom and is now penniless and sleeping under a bridge.
That's a cool and probably-somehow-implementable way to get a market to bet on "will this market exist". It doesn't give us any information about *how* people think the market is likely to cease to exist; we don't get any insight into the relative probabilities of nuclear war vs. singularity vs. financial collapse, which is what I really wish we could see. But it's better than nothing! Thanks.
I tried to make this kind of bet with a friend once, and he told me it was a good idea but he could get better odds by just taking out a loan. And if I lowered my rate to be competitive with a loan, I could just invest my money in index funds and get better returns.
In other words, the stock/bond market is sort of already a prediction market for "will the world end?" There's no reason for people to bet on the end of the world outside of it.
Meh – _no_ reason? That doesn't seem to stop traders and what-not from inventing new financial instruments now or ever before!
But it's a good point.
A possibly dumb question: if they want to make predicting positive-sum so that more people participate, why not just use the standard way to do it that academics have been working on for over a decade, namely adding a subsidized automated market maker (LMSR or something similar)? Seems like that would reduce the exploitability, as there would no longer be the positive-scoring no-op of making the same prediction as everyone else.
For these purposes there are two basic difference between a basic prediction market and a scoring system like Metaculus uses. The first is positive/zero/negative sum, which as you say could be matched by subsidizing the market. But there's also the issue that in a prediction market with real or fake money you have a limited amount of money to actually invest, which puts a fairly hard cap on how much predicting you can do (or at least how much influence you can have). We could adjust the scoring rule to make it more market-like in the sense of being more purely relative and slightly less positive sum, but the lack of a total amount of funding would still be a big difference. We've consciously taken a philosophy of removing every possible barrier to making predictions, because lack of liquidity has doomed (and is presently doing in) many prediction markets. As participation increases, we can turn the dial a bit in the opposite direction. A final issue (mentioned in my writeup) is that with a scoring system and direction predictions (rather than buying/selling) it is *much* simpler to extract the actual probabilities, especially for predicting a probability distribution over a number or date, which would be extremely awkward in a market system of buying and selling shares.
To give some context to this, Metaculus has around 1,100 active questions. Based on my experience with Polymarket, an automatic market maker would need to have $1000 to $10,000 liquidity in order for there not to be much slippage.
What if after 100 years, instead of getting 100 years of index fund returns plus the bet percentage, the winner gets the their index fund returns and the bet percentage those of the losers?
In that situation, if the total returns are 200x over time on a 50/50 bet, the winner gets 150x and the loser gets back only 50x. On an 80/20 bet, the winner gets 180x and the loser only 20x, etc...
Higher risk, but higher reward to the winner, but if you bet 50/50 one-hundred times and win half of them, you still break even. It also incentivizes taking sides on bets which are more controversial.
You could also have an annual mechanism which multiplies rewards for the winner by the number of years in the bet.
Lots of permutations, I don't think you have to be stuck with the current basic prediction market rules. More interesting to bet on if you don't.
It's really hard to take metaculus seriously.
E.g. you can't size your bets. That's really bad!
It means that some random person bullshitting on a topic is weighted as much as an expert on it.
e.g. I happen to know that https://www.metaculus.com/questions/6656/tether-in-2021/ is very likely false and that fair is waaay less than the current market, but there's no way for me to say "I will sell huge size @ 10%".
Note that it's dropped to 10% just on the strength of being linked to by Scott, apparently. Not sure what lessons we should draw from that. (The other two questions he linked didn't experience a huge "ACX effect". They had also been open much longer, and had more predictions. The Tether question got around 45 new predictions via ACX, based on timing, which more than doubled its count. The abortion question got about 5, and the Chauvin question was already so popular that ACX's contribution is in the noise.)
The way the expert gets more clout is by accruing a track record of being correct. The "Metaculus prediction" aggregates predictions according to the predictors' track records. We feel this is better than sizing bets, because having more money to put into a market does not necessarily make you an expert (in fact it does not even guarantee that you care about being accurate or extracting money. My guess is that if/when large-scale prediction markets get going most of the money is actually going to come from financial firms hedging against events, where that hedge is valuable enough that they don't mind buying at inaccurate probabilities.)
To SBF's point though, the problem with that approach is that it does a good job of aggregating information from expert generalists, but a bad job of aggregating information from expert specialists. The existing research on forecasting suggests that generalists can often perform better than specialists IF those specialists don't have much at stake on the outcome and are influenced by other social or reputational factors. But I think specialists with very high stake are going to be the most accurate in the long run. Financial markets do a great job of encouraging high-stake participation by specialists, and it's worth thinking about how to structure forecasting platforms to accomplish the same thing, even if those platforms don't have an explicitly financial component.
Metaculus has ways to reward good forecasting performance in specific domains, such as contests and specialized sites:
https://www.metaculus.com/contests
https://pandemic.metaculus.com/
One could further incentivize specialist forecasters by e.g. having separate leaderboards for different tags/categories.
Financial markets history shows that financial firms are terrible at hedging future fat tails events. Why would we assume they will start using prediction markets to hedge where they can already hedge using the financial market or the insurance market?
I think a prediction market would provide a much cleaner hedge against many types of particular events than a basket of stock that you have to work out that are correlated (or anti-correlated) with that event. And I don't think you can take out insurance, for example, against a particular supreme court decision or piece of legislation being passed.
"if you bet on a 50-50 outcome and get proven right in 2121, you've made 100% returns over 100 years. The stock market could easily make 10,000% returns over that time period, so why would you ever play the prediction markets instead?"
You don't need to do that. If someone else is holding your money for that century, the market equilibrium terms will be such that you get, on average, the market return. You pay $1 for what is currently viewed as a 50/50 bet, and if you win you get a sum in 2121 whose present value in 2021 is $2.
Consider the obvious analogy of buying a bond. Your analysis assumes that if it pays off in 10 years, the payoff will be the amount you paid for it scaled up to allow for the probability of default. But it will actually be that plus the accumulated interest.
OK, but isn't the mechanism that produces "market equilibrium terms" just people not doing things that are unfavorable? So if predictions markets don't have some sort of mechanism for scaling payouts with payout date (which they don't currently have, as far as know), then people just won't use them for questions more than a year or two in the future (or at least, the only people who use them will be the people who don't understand how interest works, which may not be the best place to look for Scott's desired superforecasters).
So I don't think that this is something to handwave away.
That also means that you need to tie up your money for that length of time. I agree that I'll "get a sum in 2121 whose present value in 2021 is $2," but doubling my real money value over a hundred years is still only a 1% real rate of return versus doing something active like (for example) owning real estate, where my rate of return is hopefully more than a 1% real rate of return.
The alternative that the post mentioned, where it's invested in index funds, means that you then get the 1% + stock market real rate of return. That is better, but it's still not great since you're limited by the risk and return of index funds. If my default hundred year investment is, for example, a leveraged index fund, then over a hundred year time scale that probably gets a better return than even a certain prediction + a normal index fund, since you're trading off risk.
"I think probably prediction markets will lose a lot of reliability and precision beyond a few decades. Here's a pessimistic way to think about this: even if we somehow found a way to make the incentives line up perfectly, could this perfect prediction market really predict things beyond a few decades anyway? "
You could potentially "fix" this issue by large institutional players with long term time horizons getting involved. If it seems like people are generally bad at predicting into the future, there could be another alternative investment class that arises for endowments, life insurers, etc. based on strong long-term predictions to take advantage of this potential inefficiency.
The time horizons will be limited by insecurity of property rights over long periods.
Maybe a more direct way to talk about the problem with long term prediction markets is to compare long term prediction to short term prediction rather than long term prediction to stocks.
If you can predict a 50-50 event 100 years from now correctly, you double your money (plus any increases in the index fund they invest in). If you predict a 50-50 event for next year correctly, you double your money (plus any increases in index fund) after only 1 year and can immediately reinvest in a new prediction market (or anything else).
I guess in equilibrium this means that the amount of work you should need to put in in order to do 1% better than average on a prediction market should be inversely proportional to the time until the payout. This means that markets with long time horizons will be relatively easy to beat the odds on (meaning that they are not particularly accurate).
You should have a look at smarkets. UK betting exchange but has some markets on politics and covid.
I enjoyed this post, but I want to note that I feel a lot of skepticism about this line: "But if instead you put your energy into finding stocks that went up 6% per year on average, you would make 339x returns."
We should expect that consistently beating the stock market by 1% should be _really, really hard_ -- much harder than identifying a 5% mispricing in a very-long-term prediction market.
You can pretty easily beat the stock market as long as you are willing to take greater risks. For example, you can use leverage to invest in index funds and get better returns but at a much higher risk. For a retirement fund, you probably shouldn't, but for someone investing over a hundred year timespan, it might be the right move.
Investing with leverage depends on the level of interest rates at the time you are making the decision to invest with leverage.
> We should expect that consistently beating the stock market by 1% should be _really, really hard_ -- much harder than identifying a 5% mispricing in a very-long-term prediction market.
How so ? If prediction markets are deregulated to the same status as stock market, (e.g. allowing large volumes of trades etc.), then how is betting on a stock different from betting on the probability that the stock will increase in value ?
If prediction markets are deregulated, what will prevent them from getting encrusted with all kinds of new and exciting financial instruments, high-frequency trading, and all the other accoutrements of the modern stock market ?
Why would you want to prevent that? Those things will only help the prediction markets become more accurate and efficient.
Nothing, but it's not obvious to me that HFT and the "accoutrements" of the modern stock market are something to be bothered by.
The idea of investing the collateral, so that it doesn't lose value to inflation over time, does seem like a good and obvious step. But there's another step you could take, which I think is almost as obvious: not requiring bets to be fully collateralized in the first place.
We do this with stock: you can buy stock "on margin", using the stock itself as collateral to buy more stock. This mostly works fine, because stock is decent collateral, which is probably not going to become suddenly worthless tomorrow (and if it declines gradually, there's plenty of time to ask the user for more money before it becomes an emergency.)
Obviously you want to do this in a way that doesn't allow people to mint free money at your expense. As a stockbroker, you know who your customers are, and at least in principle you can sue them if you have to. You almost never would, if someone gets wiped out in the market, even if they technically owe you money -- a bankrupt person has nothing worth suing for. But the possibility of lawsuits deters people from deliberately gaming the system with intentional defaults, for profit. (Plus that would be fraud, which is a crime.)
If you don't want to rely on the legal system, you can still extend some trust to users who have been around for a long time, and done a lot of betting. They can still build up lots of good reputation, only to use it to rip you off (in the cryptocurrency business, this is an "exit scam".) But you can bound the amount of risk you're taking on to acceptable levels.
And saving the best for last, here's one I _assume_ there must be papers about: you should always be able to use _independent_ bets as collateral for each other. Or, putting it another way, you should be able to share collateral between multiple independent bets. For example, if I place 100 bets, $1 each, on coinflips coming up heads, it should be obvious that $100 of collateral is too much to require. The probability that I will owe at least $60 is around 3%. The probability that I will owe $65 is almost negligible.
Lots of real-life predictions are only *somewhat* independent. A food shortage, a metal shortage, and an energy shortage are often caused by different things, but they can also all be caused by a single large event, such as a war. This makes the odds that you will lose on all of those bets at once considerably higher than if they were truly independent, in a difficult-to-quantify way.
Well, if you need a way to estimate the independence of various distributions, you have a whole site full of people to ask... it's prediction markets all the way down.
I'm mostly kidding. Those would be pretty hard to score. But you don't need the estimates to be perfect, they can be conservative. And some of them will be easier to make than others -- looking at Metaculus right now, I would say that the truth of the Collatz Conjecture is totally uncorrelated with ~all other questions on the site, for example.
And it seems like some questions may become less correlated (or more clearly uncorrelated) over time, which seems like it could be favorable if you otherwise want people to post more collateral as questions get closer to resolving, since it can partly balance that out. E.g. Tesla's 2025 vehicle production is not totally uncorrelated with various geopolitical questions not otherwise related to clean energy, insofar as there are major disruptive events that could impact both. But as time moves on and major disruptive events don't happen, that goes down.
That would have been a convincing argument in 2006.
Hey – it works 100% about 97% of the time!
If you’re the market maker you have equal exposure to YES and NO, so all you need to do is make sure the collateral you’re taking to place bets is matched by the other side. If 1 in 10 people in YES have TSLA as collateral but it plummets and they default, you’re fine because 1 in 10 people in NO also have TSLA as collateral, so you just wipe both of them off the book and continue.
Practically speaking, it’s probably simpler to just sell multiple tranches of each market for some popular investment vehicles (equities index, treasuries, cash, etc).
Sorry, of course that should be "the probability that I will _lose 60 flips_", and lose 65 flips, respectively. The amounts I owe would be $20 and $30 respectively, if I did the math correctly this time. The probability that I owe at least $60, which would require losing 80 flips, is absolutely negligible.
The recursion problem reminds me of the unexpected hanging paradox https://en.m.wikipedia.org/wiki/Unexpected_hanging_paradox