> The person who spends one second picking a random number on a thousand questions will get more points than someone who spends an hour researching a really good answer to one question.
While it sounds intuitive, I recall reading some studies that the "wisdom of the crowds" works better when people don't talk to each other or research the questions [1], as these are subject to various cognitive biases that can easily lead you astray.
So if the goal of a prediction market is to be more accurate, maybe one with the kind of incentives you describe might work a little better.
[1] Don't have time to lookup references now, anyone else have them?
I have a much bigger issue with the Metaculus scoring system than what has been mentioned here:
In Metaculus, it is possible to predict the probability of the event *completely accurately* and *still lose points* if you are unwilling to come back to the market and update your prediction every time new data comes to light.
This is because the Metaculus scoring rule isn't applied instantaneously - you make a prediction and then the score of that prediction is averaged over the entire course of the market from opening to close (see the integral in the Metaculus FAQ scoring section). If you do a great deal of analysis to determine the probability that COVID deaths in a certain month are 50% likely to go over a certain value, you make a 50% prediction, and then halfway through the month it becomes clear that deaths will go above that value and the market starts predicting close to 100%, you are penalized as if you had made a 50% guess on a 100% probability outcome every day for the rest of that month.
Hey Scott, this is totally unrelated to this post but not sure how else to contact you. Just saying I used to put all your posts on SSC onto the Pocket app where I could read in dark mode and also easily sync to my kobo ereader, but these posts on substack don't work (if I put a post onto pocket it refuses to recognise it as an article). I thought maybe this was a deliberate design choice on the part of substack to sabotage their articles going on pocket as they wanted readers to use their own app, but I've since discovered substack blogs for which it works fine so it's definitely a solvable problem. It's understandable if you don't have the time/inclination to investigate this, maybe this is a rare thing to be bothered about, but I guess all I can say is personally I'd be eternally grateful if you looked into this as I really miss reading your posts on my ereader :)
A simple solution here is not to maximize for Metaculus points, but to maximize for average points per question. Some people have compiled the data necessary for this calculation.
This value needs some shrinkage to remove luck factor for small n (as done in baseball stats http://varianceexplained.org/r/empirical_bayes_baseball/), but otherwise, it is pretty nice. My average score is currently sitting at 31.5, putting me at the 36th spot according to that spreadsheet. One cannot just mindlessly spam median predictions to maximize this value, one has to predict on questions where one will beat the market.
> gold is currently worth $1700/ounce, and 20% chance for $2500 by 2025 seems plausible
In fact, that's very plausible. You can look up options prices for gold based on the end-of-2024 futures contract. At the moment of this writing, the end-of-2024 call option with a strike price of 2500 has a "delta" of about 0.25, giving a market expectation of about 25% that gold will be at or above $2500/oz in December 2024.
If the Foresight exchange operated on real money, there might be an arbitrage possibility there.
“But if instead you put your energy into finding stocks that went up 6% per year on average, you would make 339x returns”
Your accounting assumes that someone has an edge in equity investing but fails to account for any edge they might have in the prediction market. Presumably if they’re investing in the prediction market, they think they have an edge there.
All you need to make the prediction market work is the existence of people who think they have an edge in prediction markets but not in equity markets.
Public support for both the ruling party and the Catholic Church took a nosedive, and the delicate Schelling fence of the 1993 "abortion compromise" (itself very close to the Catholic position) got completely wrecked. The law briefly became extremely restrictive, and it is expected that once the government changes it will become more liberal than ever, as most of the opposition aligned itself with the pro-choice protesters.
The major opposition party was previously playing a balancing act a la median voter theorem - not appearing too anti-church nor pro-church to avoid alienating anyone. Now they were forced to take a side.
Would index fund theory work on a true prediction market? Like naively I think you could make money on the bet that “prediction markets are good at predicting” by just betting with the house odds on every bet and periodically readjusting. Is this dumb for some reason I don’t realize?
In 1996, gold was $369 an ounce, so it must have seemed like a safe bet that it couldn't inflate so much that it would be worth $2500 in 30 years. (It would be an average of 7% inflation a year, which is bad but not exactly hyperinflation either.)
Gold prices skyrocketed around 2007 and never really came back down, even though inflation stayed level throughout the financial crisis. So I guess it's not as good a measure of inflation as they thought.
If you predict the maximum 99% and the event doesn't happen, you **lose** 351 points.
If you predict the minimum 1% and the event happens, you **lose** 353 points.
If you believe the event is 50% likely (not sure why would you believe that, but arguendo), you expect 50%*13+50%*18 = 15.5 points.
If you believe the community that the event is 45% likely, you expect 55%*29+45%*1 = 16.4 points.
If you believe the event is 75% likely, you expect 75%*77+25%*-84 = 36.8 points.
etc.
Importantly, the scoring rule means the point-maximising strategy _on any question_ is to predict your true credence.
This is separate from the point maximising strategy _on the whole site_. But as Scott points out, nobody does that, because that'd be unbelievably boring. Notice that the best users in absolute number of points have much more than 15 points per question (which is what you typically expect by predicting the community median). https://metaculusextras.com/points_per_question
Does Metaculus still need to exist in 100 years in order for the bet to pay out? Because in that case, you also need to multiply any win chances with the low percentage that this will in fact be the case.
I aggregate the top comments on Metaculus (by likes) and share them in the discord. I'm going to continue sharing them on Mantic Monday until someone asks me to stop.
Maybe instead of betting on outcome there should be traceable binary options? You can totally imagine options with 100y. Their current trading price will determine the odds and will also handle time value (I.e. price might change if less time remains for something to happen)
The question is: who would sell you such an option? “Market makers”?
> "which party will win the 2100 presidential election?" might change a little over the next 80 years as (for example) demographic change makes one or another party permanently stronger or weaker
Duverger's law makes demographic change, or even what issues the election is fought on, irrelevant. So long as the USA keeps the same electoral system, there will be 2 parties and each will have a roughly 50% chance of winning the election.
I remember late 2019, a lot of the SSC commenters had near certainty on who would win the 2020 election. Six months later, they all changed their mind, saying covid completely changed their predictions.
100 years is a long time, we are likely to experience at least one covid level event between now and then.
Metaculus spammers do hurt accuracy though! Suppose you put up a new question and the smart parts of the market think it's 35.8% yes. The spammers show up and reinforce that to an extremely confident 35.8%. Then some new information comes out that makes a yes look almost guaranteed. If the market was only smart participants it would shoot up to ninety-some percent, but the spammers (who don't bother to update their predictions because the whole point of spamming is that it's a low-effort strategy) drag the market's estimate way down.
70% chance the officer who killed George Floyd will get acquitted *of the murder charges*. This is important - the difference between complete acquittal and only acquittal of the most grave charge is quite substantial.
I recently learned about Long Bets (https://longbets.org/), which tracks predictions and bets between individuals to be resolved many years in the future. Started in 2003, and originally funded by Jeff Bezos.
The winnings go to charity, and the money is invested in the meantime, with the gains being split between the foundation that runs it and the winner's charity.
Since you posted this, the Tether prediction has gone from 36% to 16%. While this may be in part due to the number of predictions doubling, it's interesting what it says about the attention of the market. On the other hand, no one here seems to have had an option on the Polish abortion rate. (The George Floyd question is already closed.)
Disclosure: I'm a frequent predictor on Metaculus and it's hard in cases like this to not have my thoughts be dominated/emotionally compromised by naked tribalism.
That said, I thought, and continue to believe on reflection, that both Zvi's critique and Ross' writeup missed what I think are [fairly obvious points](https://twitter.com/LinchZhang/status/1360300621782392834). The primary point of Metaculus and platforms like it is to aggregate truthful information about the world and present it in a way that's collectively digestible, not sample-efficient ways for to get perfect individual rankings.
Obviously both desiderata are useful, and there's a case to be made that maybe the current Metaculus system rewards systematic accuracy and legibility too much over individual accuracy and legibility. But I think any reasonable critique should've been framed much more like "we think you're making the wrong tradeoff on this spectrum because of XYZ" rather than "my point on this spectrum is obviously correct and anybody who says otherwise is a moron." Worse, I think Zvi et.al implicitly aren't even acknowledging the tradeoff at all?
The point on "participation trophies" seems fairly off the mark. Whenever I talk to people who think about how to improve forecasting, one of the most common points is that "the problem isn't that people just or primarily make inaccurate predictions when they make precise numerical answers to well-defined questions, it's that they don't make much if any forecasts at all." To the degree that points incentivize behavior, having participation points is good because it incentivizes more forecasts, since if you want to get more of something, you should incentivize it.
In general, I feel like the critique seems more like "You claimed to be doing X. You did X, which is not like Y. I expected this to be Y, without any justification for why Y is better. Now I'm going to judge you on the metrics of Y, rather than X."
I was politer on Twitter when this kerfuffle first came out and I was deliberately trying to adjust for emotional compromise, but upon reflection I'm inclined to think that this type of motivated ignorance is somewhat suss.
My prediction: if Prediction Markets ever get to the point of relevance, Wall Street will end up being the only ones making significant money on them. It's the hedge funds, with their billions of dollars to devote to computing power and market research, that would end up being the "superpredictors".
So with regards to the search for Metaforecast.org, I am taking exact matches and ranking them by starts and then by number of forecasts, and then I'm using https://fusejs.io/ for the rest.
The code for this is open source, and can be found here: https://github.com/QURIresearch/metaforecast-website-nextjs/blob/master/pages/index.js, lines 120 to 189. The comment on line 130 refers to the fact that, because Guesstimate contains too many models (17k), I piggyback on its search rather than loading the models on initial page load, and I have to do this asynchronously.
Pull requests or suggestions are welcome!
As an addendum, Metaforecast also searches forecasting platforms (Metaculus, Good Judgment, etc.), not only prediction markets.
As for the issue that "I don’t think Metaculus considers accurately ranking people by predictive skill an especially high priority compared to actually important things like getting good predictions, and that’s fine, but it kind of sucks for people who were motivated by a competitive streak"... Metaculus really does care about assessing accuracy for use in weighting predictions in the aggregation method. We also want to give users what they want, and many of them want to compete with each other, so on the drawing boards are plans for an upgraded "leaderboards" page that allows more comparisons. The tricky thing is we don't want *too* many comparison methods, because then you'll always have ways to do particularly well at some of them. And if we make any of those methods based on non-proper scoring then we have to worry that people will have incorrect incentives. The basic problem is there's no such thing as a single quantifier of "predictive skill" so we have do think a bit about a few measures that seem especially valuable.
Using stock market returns to keep prediction returns competitive is inherently flawed, in that it distorts incentives on predictions where stock market returns are lower. If you have a prediction regarding future interest rates, predicting 'Yes' on higher rates becomes a problematic and distorted hedge on stock market returns - if rates increase, stock values drop and the 'Yes' bet becomes inherently less valuable. You might claim that the efficient market hypothesis will correct for this, but then you'd have to prove that the prediction market is inherently efficient.
My concern about making a prediction about something in 2121 is I have little confidence the prediction market will still be around and functioning then. A lot can change in 100 years. How is this risk dealt with?
I'm turned off by the long-term prospects of prediction markets simply because many of the questions I most want to see predictions on are "heads I win - tails we all lose", getting rid of any incentive to vote tails.
Let's say you want the market to estimate the probability of a world war, hyperinflation in the US, The Singularity, any political change causing major revisions of property rights (h/t David Friedman on another thread), and other similarly significant events. If any of the above actually were to happen, the chance that Metaculus is still around to give a meaningful payout would drop precipitously. And that means there's significantly less of an incentive to bet that those things will happen, even if you actually think one of them is likely to.
That severely damages Metaculus' ability to speculate on matters like X-risk, where I most wish we could benefit from pooling the collective brainpower and financial self-interest of thousands. And I don't see a fix, besides hoping people's honest (or vanity) will outweigh their self-interest.
A possibly dumb question: if they want to make predicting positive-sum so that more people participate, why not just use the standard way to do it that academics have been working on for over a decade, namely adding a subsidized automated market maker (LMSR or something similar)? Seems like that would reduce the exploitability, as there would no longer be the positive-scoring no-op of making the same prediction as everyone else.
What if after 100 years, instead of getting 100 years of index fund returns plus the bet percentage, the winner gets the their index fund returns and the bet percentage those of the losers?
In that situation, if the total returns are 200x over time on a 50/50 bet, the winner gets 150x and the loser gets back only 50x. On an 80/20 bet, the winner gets 180x and the loser only 20x, etc...
Higher risk, but higher reward to the winner, but if you bet 50/50 one-hundred times and win half of them, you still break even. It also incentivizes taking sides on bets which are more controversial.
You could also have an annual mechanism which multiplies rewards for the winner by the number of years in the bet.
Lots of permutations, I don't think you have to be stuck with the current basic prediction market rules. More interesting to bet on if you don't.
"if you bet on a 50-50 outcome and get proven right in 2121, you've made 100% returns over 100 years. The stock market could easily make 10,000% returns over that time period, so why would you ever play the prediction markets instead?"
You don't need to do that. If someone else is holding your money for that century, the market equilibrium terms will be such that you get, on average, the market return. You pay $1 for what is currently viewed as a 50/50 bet, and if you win you get a sum in 2121 whose present value in 2021 is $2.
Consider the obvious analogy of buying a bond. Your analysis assumes that if it pays off in 10 years, the payoff will be the amount you paid for it scaled up to allow for the probability of default. But it will actually be that plus the accumulated interest.
"I think probably prediction markets will lose a lot of reliability and precision beyond a few decades. Here's a pessimistic way to think about this: even if we somehow found a way to make the incentives line up perfectly, could this perfect prediction market really predict things beyond a few decades anyway? "
You could potentially "fix" this issue by large institutional players with long term time horizons getting involved. If it seems like people are generally bad at predicting into the future, there could be another alternative investment class that arises for endowments, life insurers, etc. based on strong long-term predictions to take advantage of this potential inefficiency.
Maybe a more direct way to talk about the problem with long term prediction markets is to compare long term prediction to short term prediction rather than long term prediction to stocks.
If you can predict a 50-50 event 100 years from now correctly, you double your money (plus any increases in the index fund they invest in). If you predict a 50-50 event for next year correctly, you double your money (plus any increases in index fund) after only 1 year and can immediately reinvest in a new prediction market (or anything else).
I guess in equilibrium this means that the amount of work you should need to put in in order to do 1% better than average on a prediction market should be inversely proportional to the time until the payout. This means that markets with long time horizons will be relatively easy to beat the odds on (meaning that they are not particularly accurate).
I enjoyed this post, but I want to note that I feel a lot of skepticism about this line: "But if instead you put your energy into finding stocks that went up 6% per year on average, you would make 339x returns."
We should expect that consistently beating the stock market by 1% should be _really, really hard_ -- much harder than identifying a 5% mispricing in a very-long-term prediction market.
If prediction markets are deregulated, what will prevent them from getting encrusted with all kinds of new and exciting financial instruments, high-frequency trading, and all the other accoutrements of the modern stock market ?
The idea of investing the collateral, so that it doesn't lose value to inflation over time, does seem like a good and obvious step. But there's another step you could take, which I think is almost as obvious: not requiring bets to be fully collateralized in the first place.
We do this with stock: you can buy stock "on margin", using the stock itself as collateral to buy more stock. This mostly works fine, because stock is decent collateral, which is probably not going to become suddenly worthless tomorrow (and if it declines gradually, there's plenty of time to ask the user for more money before it becomes an emergency.)
Obviously you want to do this in a way that doesn't allow people to mint free money at your expense. As a stockbroker, you know who your customers are, and at least in principle you can sue them if you have to. You almost never would, if someone gets wiped out in the market, even if they technically owe you money -- a bankrupt person has nothing worth suing for. But the possibility of lawsuits deters people from deliberately gaming the system with intentional defaults, for profit. (Plus that would be fraud, which is a crime.)
If you don't want to rely on the legal system, you can still extend some trust to users who have been around for a long time, and done a lot of betting. They can still build up lots of good reputation, only to use it to rip you off (in the cryptocurrency business, this is an "exit scam".) But you can bound the amount of risk you're taking on to acceptable levels.
And saving the best for last, here's one I _assume_ there must be papers about: you should always be able to use _independent_ bets as collateral for each other. Or, putting it another way, you should be able to share collateral between multiple independent bets. For example, if I place 100 bets, $1 each, on coinflips coming up heads, it should be obvious that $100 of collateral is too much to require. The probability that I will owe at least $60 is around 3%. The probability that I will owe $65 is almost negligible.
> The person who spends one second picking a random number on a thousand questions will get more points than someone who spends an hour researching a really good answer to one question.
While it sounds intuitive, I recall reading some studies that the "wisdom of the crowds" works better when people don't talk to each other or research the questions [1], as these are subject to various cognitive biases that can easily lead you astray.
So if the goal of a prediction market is to be more accurate, maybe one with the kind of incentives you describe might work a little better.
[1] Don't have time to lookup references now, anyone else have them?
I have a much bigger issue with the Metaculus scoring system than what has been mentioned here:
In Metaculus, it is possible to predict the probability of the event *completely accurately* and *still lose points* if you are unwilling to come back to the market and update your prediction every time new data comes to light.
This is because the Metaculus scoring rule isn't applied instantaneously - you make a prediction and then the score of that prediction is averaged over the entire course of the market from opening to close (see the integral in the Metaculus FAQ scoring section). If you do a great deal of analysis to determine the probability that COVID deaths in a certain month are 50% likely to go over a certain value, you make a 50% prediction, and then halfway through the month it becomes clear that deaths will go above that value and the market starts predicting close to 100%, you are penalized as if you had made a 50% guess on a 100% probability outcome every day for the rest of that month.
Hey Scott, this is totally unrelated to this post but not sure how else to contact you. Just saying I used to put all your posts on SSC onto the Pocket app where I could read in dark mode and also easily sync to my kobo ereader, but these posts on substack don't work (if I put a post onto pocket it refuses to recognise it as an article). I thought maybe this was a deliberate design choice on the part of substack to sabotage their articles going on pocket as they wanted readers to use their own app, but I've since discovered substack blogs for which it works fine so it's definitely a solvable problem. It's understandable if you don't have the time/inclination to investigate this, maybe this is a rare thing to be bothered about, but I guess all I can say is personally I'd be eternally grateful if you looked into this as I really miss reading your posts on my ereader :)
A simple solution here is not to maximize for Metaculus points, but to maximize for average points per question. Some people have compiled the data necessary for this calculation.
https://www.metaculus.com/questions/935/platform-feature-suggestions/#comment-51338
https://docs.google.com/spreadsheets/d/1Fy7uqy_KgFAZhL_2br_O9_sV2vzki6Vzhzx-xvJ1_b4/edit#gid=0
This value needs some shrinkage to remove luck factor for small n (as done in baseball stats http://varianceexplained.org/r/empirical_bayes_baseball/), but otherwise, it is pretty nice. My average score is currently sitting at 31.5, putting me at the 36th spot according to that spreadsheet. One cannot just mindlessly spam median predictions to maximize this value, one has to predict on questions where one will beat the market.
> gold is currently worth $1700/ounce, and 20% chance for $2500 by 2025 seems plausible
In fact, that's very plausible. You can look up options prices for gold based on the end-of-2024 futures contract. At the moment of this writing, the end-of-2024 call option with a strike price of 2500 has a "delta" of about 0.25, giving a market expectation of about 25% that gold will be at or above $2500/oz in December 2024.
If the Foresight exchange operated on real money, there might be an arbitrage possibility there.
“But if instead you put your energy into finding stocks that went up 6% per year on average, you would make 339x returns”
Your accounting assumes that someone has an edge in equity investing but fails to account for any edge they might have in the prediction market. Presumably if they’re investing in the prediction market, they think they have an edge there.
All you need to make the prediction market work is the existence of people who think they have an edge in prediction markets but not in equity markets.
Re the abortion situation:
The conservatives overplayed their hand really, really badly this fall - even people responsible admit they lost the cause.
https://www.theguardian.com/world/2020/oct/30/pro-choice-supporters-hold-biggest-ever-protest-against-polish-government
Public support for both the ruling party and the Catholic Church took a nosedive, and the delicate Schelling fence of the 1993 "abortion compromise" (itself very close to the Catholic position) got completely wrecked. The law briefly became extremely restrictive, and it is expected that once the government changes it will become more liberal than ever, as most of the opposition aligned itself with the pro-choice protesters.
The major opposition party was previously playing a balancing act a la median voter theorem - not appearing too anti-church nor pro-church to avoid alienating anyone. Now they were forced to take a side.
Would index fund theory work on a true prediction market? Like naively I think you could make money on the bet that “prediction markets are good at predicting” by just betting with the house odds on every bet and periodically readjusting. Is this dumb for some reason I don’t realize?
In 1996, gold was $369 an ounce, so it must have seemed like a safe bet that it couldn't inflate so much that it would be worth $2500 in 30 years. (It would be an average of 7% inflation a year, which is bad but not exactly hyperinflation either.)
Gold prices skyrocketed around 2007 and never really came back down, even though inflation stayed level throughout the financial crisis. So I guess it's not as good a measure of inflation as they thought.
For the record, "you’ll gain points on the prediction even if you were maximally wrong" is not true.
Taking the first open question on the main page: https://www.metaculus.com/questions/6330
If you predict the maximum 99% and the event doesn't happen, you **lose** 351 points.
If you predict the minimum 1% and the event happens, you **lose** 353 points.
If you believe the event is 50% likely (not sure why would you believe that, but arguendo), you expect 50%*13+50%*18 = 15.5 points.
If you believe the community that the event is 45% likely, you expect 55%*29+45%*1 = 16.4 points.
If you believe the event is 75% likely, you expect 75%*77+25%*-84 = 36.8 points.
etc.
Importantly, the scoring rule means the point-maximising strategy _on any question_ is to predict your true credence.
This is separate from the point maximising strategy _on the whole site_. But as Scott points out, nobody does that, because that'd be unbelievably boring. Notice that the best users in absolute number of points have much more than 15 points per question (which is what you typically expect by predicting the community median). https://metaculusextras.com/points_per_question
Does Metaculus still need to exist in 100 years in order for the bet to pay out? Because in that case, you also need to multiply any win chances with the low percentage that this will in fact be the case.
I aggregate the top comments on Metaculus (by likes) and share them in the discord. I'm going to continue sharing them on Mantic Monday until someone asks me to stop.
https://metaculusextras.com/top_comments?start_date=2021-02-22
Last week's top comments:
- Users are planning on living forever (https://www.metaculus.com/questions/6046/date-1-bitcoin-worth-1-million/?invite=tLxPdB#comment-56474, https://www.metaculus.com/questions/3399/when-will-the-last-metaculus-question-resolution-occur/?invite=tLxPdB#comment-56677, https://www.metaculus.com/questions/6046/date-1-bitcoin-worth-1-million/?invite=tLxPdB#comment-56420)
- Vaccine news: FDA gives EUA for J&J and Moderna manufacture a vaccine for a variant (https://www.metaculus.com/questions/6492/will-the-jj-vaccine-be-issued-an-eua-by-fda/?invite=tLxPdB#comment-56495, https://www.metaculus.com/questions/6007/vaccine-update-due-to-mutation/?invite=tLxPdB#comment-56587)
- Galen (the color whisperer) is back to let us know what's in style (https://www.metaculus.com/questions/5887/pantones-color-of-the-year-for-2022/?invite=joX2Fj#comment-56370)
- I explain the logic of why people are expecting a new SCOTUS vacancy soon (https://www.metaculus.com/questions/6634/matthew-yglesias-predicts-2021/?invite=tLxPdB#comment-56297)
- I make the bear case for Trumps on substack (https://www.metaculus.com/questions/6423/trump-substacker-before-2024-election/?invite=tLxPdB#comment-56424)
- aqsalose calculates the base rate of for new Russian PMs to calculate Navalny's chances (https://www.metaculus.com/questions/3416/alexei-navalny-to-become-president-or-prime-minister-of-russia-in-his-lifetime/?invite=tLxPdB#comment-56493)
- I lay out my estimate for a Scottish referendum being announced in 2021 (https://www.metaculus.com/questions/5922/scotland-independence-referendum-in-2021/?invite=Olx7Ci#comment-56410)
Maybe instead of betting on outcome there should be traceable binary options? You can totally imagine options with 100y. Their current trading price will determine the odds and will also handle time value (I.e. price might change if less time remains for something to happen)
The question is: who would sell you such an option? “Market makers”?
What are the odds that Metaculus still exists in 100 years?
If I understand correctly, the dream is free and open money based prediction markets, but we can't have that good thing yet for US legal reasons.
If that's the case, is there a reason we're not seeing money based prediction markets in other countries with different rules?
Surely there's incredible incentive to get this started in places with large populations and laxer laws.
> "which party will win the 2100 presidential election?" might change a little over the next 80 years as (for example) demographic change makes one or another party permanently stronger or weaker
Duverger's law makes demographic change, or even what issues the election is fought on, irrelevant. So long as the USA keeps the same electoral system, there will be 2 parties and each will have a roughly 50% chance of winning the election.
So, expert forecasters: What are your odds on who will win the 2120 baseball World Series?
I remember late 2019, a lot of the SSC commenters had near certainty on who would win the 2020 election. Six months later, they all changed their mind, saying covid completely changed their predictions.
100 years is a long time, we are likely to experience at least one covid level event between now and then.
Metaculus spammers do hurt accuracy though! Suppose you put up a new question and the smart parts of the market think it's 35.8% yes. The spammers show up and reinforce that to an extremely confident 35.8%. Then some new information comes out that makes a yes look almost guaranteed. If the market was only smart participants it would shoot up to ninety-some percent, but the spammers (who don't bother to update their predictions because the whole point of spamming is that it's a low-effort strategy) drag the market's estimate way down.
70% chance the officer who killed George Floyd will get acquitted *of the murder charges*. This is important - the difference between complete acquittal and only acquittal of the most grave charge is quite substantial.
I love Mantic Mondays
I recently learned about Long Bets (https://longbets.org/), which tracks predictions and bets between individuals to be resolved many years in the future. Started in 2003, and originally funded by Jeff Bezos.
The winnings go to charity, and the money is invested in the meantime, with the gains being split between the foundation that runs it and the winner's charity.
Since you posted this, the Tether prediction has gone from 36% to 16%. While this may be in part due to the number of predictions doubling, it's interesting what it says about the attention of the market. On the other hand, no one here seems to have had an option on the Polish abortion rate. (The George Floyd question is already closed.)
Disclosure: I'm a frequent predictor on Metaculus and it's hard in cases like this to not have my thoughts be dominated/emotionally compromised by naked tribalism.
That said, I thought, and continue to believe on reflection, that both Zvi's critique and Ross' writeup missed what I think are [fairly obvious points](https://twitter.com/LinchZhang/status/1360300621782392834). The primary point of Metaculus and platforms like it is to aggregate truthful information about the world and present it in a way that's collectively digestible, not sample-efficient ways for to get perfect individual rankings.
Obviously both desiderata are useful, and there's a case to be made that maybe the current Metaculus system rewards systematic accuracy and legibility too much over individual accuracy and legibility. But I think any reasonable critique should've been framed much more like "we think you're making the wrong tradeoff on this spectrum because of XYZ" rather than "my point on this spectrum is obviously correct and anybody who says otherwise is a moron." Worse, I think Zvi et.al implicitly aren't even acknowledging the tradeoff at all?
The point on "participation trophies" seems fairly off the mark. Whenever I talk to people who think about how to improve forecasting, one of the most common points is that "the problem isn't that people just or primarily make inaccurate predictions when they make precise numerical answers to well-defined questions, it's that they don't make much if any forecasts at all." To the degree that points incentivize behavior, having participation points is good because it incentivizes more forecasts, since if you want to get more of something, you should incentivize it.
In general, I feel like the critique seems more like "You claimed to be doing X. You did X, which is not like Y. I expected this to be Y, without any justification for why Y is better. Now I'm going to judge you on the metrics of Y, rather than X."
I was politer on Twitter when this kerfuffle first came out and I was deliberately trying to adjust for emotional compromise, but upon reflection I'm inclined to think that this type of motivated ignorance is somewhat suss.
My prediction: if Prediction Markets ever get to the point of relevance, Wall Street will end up being the only ones making significant money on them. It's the hedge funds, with their billions of dollars to devote to computing power and market research, that would end up being the "superpredictors".
"Polish abortion rate will dectuple by 2030 (why? they seem pretty conservative; is their conservatism that precarious?)"
Yup:
https://www.unz.com/akarlin/poland-will-legalize-gay-marriage-within-10-years/
So with regards to the search for Metaforecast.org, I am taking exact matches and ranking them by starts and then by number of forecasts, and then I'm using https://fusejs.io/ for the rest.
The code for this is open source, and can be found here: https://github.com/QURIresearch/metaforecast-website-nextjs/blob/master/pages/index.js, lines 120 to 189. The comment on line 130 refers to the fact that, because Guesstimate contains too many models (17k), I piggyback on its search rather than loading the models on initial page load, and I have to do this asynchronously.
Pull requests or suggestions are welcome!
As an addendum, Metaforecast also searches forecasting platforms (Metaculus, Good Judgment, etc.), not only prediction markets.
As for the issue that "I don’t think Metaculus considers accurately ranking people by predictive skill an especially high priority compared to actually important things like getting good predictions, and that’s fine, but it kind of sucks for people who were motivated by a competitive streak"... Metaculus really does care about assessing accuracy for use in weighting predictions in the aggregation method. We also want to give users what they want, and many of them want to compete with each other, so on the drawing boards are plans for an upgraded "leaderboards" page that allows more comparisons. The tricky thing is we don't want *too* many comparison methods, because then you'll always have ways to do particularly well at some of them. And if we make any of those methods based on non-proper scoring then we have to worry that people will have incorrect incentives. The basic problem is there's no such thing as a single quantifier of "predictive skill" so we have do think a bit about a few measures that seem especially valuable.
Using stock market returns to keep prediction returns competitive is inherently flawed, in that it distorts incentives on predictions where stock market returns are lower. If you have a prediction regarding future interest rates, predicting 'Yes' on higher rates becomes a problematic and distorted hedge on stock market returns - if rates increase, stock values drop and the 'Yes' bet becomes inherently less valuable. You might claim that the efficient market hypothesis will correct for this, but then you'd have to prove that the prediction market is inherently efficient.
My concern about making a prediction about something in 2121 is I have little confidence the prediction market will still be around and functioning then. A lot can change in 100 years. How is this risk dealt with?
I'm turned off by the long-term prospects of prediction markets simply because many of the questions I most want to see predictions on are "heads I win - tails we all lose", getting rid of any incentive to vote tails.
Let's say you want the market to estimate the probability of a world war, hyperinflation in the US, The Singularity, any political change causing major revisions of property rights (h/t David Friedman on another thread), and other similarly significant events. If any of the above actually were to happen, the chance that Metaculus is still around to give a meaningful payout would drop precipitously. And that means there's significantly less of an incentive to bet that those things will happen, even if you actually think one of them is likely to.
That severely damages Metaculus' ability to speculate on matters like X-risk, where I most wish we could benefit from pooling the collective brainpower and financial self-interest of thousands. And I don't see a fix, besides hoping people's honest (or vanity) will outweigh their self-interest.
A possibly dumb question: if they want to make predicting positive-sum so that more people participate, why not just use the standard way to do it that academics have been working on for over a decade, namely adding a subsidized automated market maker (LMSR or something similar)? Seems like that would reduce the exploitability, as there would no longer be the positive-scoring no-op of making the same prediction as everyone else.
What if after 100 years, instead of getting 100 years of index fund returns plus the bet percentage, the winner gets the their index fund returns and the bet percentage those of the losers?
In that situation, if the total returns are 200x over time on a 50/50 bet, the winner gets 150x and the loser gets back only 50x. On an 80/20 bet, the winner gets 180x and the loser only 20x, etc...
Higher risk, but higher reward to the winner, but if you bet 50/50 one-hundred times and win half of them, you still break even. It also incentivizes taking sides on bets which are more controversial.
You could also have an annual mechanism which multiplies rewards for the winner by the number of years in the bet.
Lots of permutations, I don't think you have to be stuck with the current basic prediction market rules. More interesting to bet on if you don't.
It's really hard to take metaculus seriously.
E.g. you can't size your bets. That's really bad!
It means that some random person bullshitting on a topic is weighted as much as an expert on it.
e.g. I happen to know that https://www.metaculus.com/questions/6656/tether-in-2021/ is very likely false and that fair is waaay less than the current market, but there's no way for me to say "I will sell huge size @ 10%".
"if you bet on a 50-50 outcome and get proven right in 2121, you've made 100% returns over 100 years. The stock market could easily make 10,000% returns over that time period, so why would you ever play the prediction markets instead?"
You don't need to do that. If someone else is holding your money for that century, the market equilibrium terms will be such that you get, on average, the market return. You pay $1 for what is currently viewed as a 50/50 bet, and if you win you get a sum in 2121 whose present value in 2021 is $2.
Consider the obvious analogy of buying a bond. Your analysis assumes that if it pays off in 10 years, the payoff will be the amount you paid for it scaled up to allow for the probability of default. But it will actually be that plus the accumulated interest.
"I think probably prediction markets will lose a lot of reliability and precision beyond a few decades. Here's a pessimistic way to think about this: even if we somehow found a way to make the incentives line up perfectly, could this perfect prediction market really predict things beyond a few decades anyway? "
You could potentially "fix" this issue by large institutional players with long term time horizons getting involved. If it seems like people are generally bad at predicting into the future, there could be another alternative investment class that arises for endowments, life insurers, etc. based on strong long-term predictions to take advantage of this potential inefficiency.
Maybe a more direct way to talk about the problem with long term prediction markets is to compare long term prediction to short term prediction rather than long term prediction to stocks.
If you can predict a 50-50 event 100 years from now correctly, you double your money (plus any increases in the index fund they invest in). If you predict a 50-50 event for next year correctly, you double your money (plus any increases in index fund) after only 1 year and can immediately reinvest in a new prediction market (or anything else).
I guess in equilibrium this means that the amount of work you should need to put in in order to do 1% better than average on a prediction market should be inversely proportional to the time until the payout. This means that markets with long time horizons will be relatively easy to beat the odds on (meaning that they are not particularly accurate).
You should have a look at smarkets. UK betting exchange but has some markets on politics and covid.
I enjoyed this post, but I want to note that I feel a lot of skepticism about this line: "But if instead you put your energy into finding stocks that went up 6% per year on average, you would make 339x returns."
We should expect that consistently beating the stock market by 1% should be _really, really hard_ -- much harder than identifying a 5% mispricing in a very-long-term prediction market.
If prediction markets are deregulated, what will prevent them from getting encrusted with all kinds of new and exciting financial instruments, high-frequency trading, and all the other accoutrements of the modern stock market ?
The idea of investing the collateral, so that it doesn't lose value to inflation over time, does seem like a good and obvious step. But there's another step you could take, which I think is almost as obvious: not requiring bets to be fully collateralized in the first place.
We do this with stock: you can buy stock "on margin", using the stock itself as collateral to buy more stock. This mostly works fine, because stock is decent collateral, which is probably not going to become suddenly worthless tomorrow (and if it declines gradually, there's plenty of time to ask the user for more money before it becomes an emergency.)
Obviously you want to do this in a way that doesn't allow people to mint free money at your expense. As a stockbroker, you know who your customers are, and at least in principle you can sue them if you have to. You almost never would, if someone gets wiped out in the market, even if they technically owe you money -- a bankrupt person has nothing worth suing for. But the possibility of lawsuits deters people from deliberately gaming the system with intentional defaults, for profit. (Plus that would be fraud, which is a crime.)
If you don't want to rely on the legal system, you can still extend some trust to users who have been around for a long time, and done a lot of betting. They can still build up lots of good reputation, only to use it to rip you off (in the cryptocurrency business, this is an "exit scam".) But you can bound the amount of risk you're taking on to acceptable levels.
And saving the best for last, here's one I _assume_ there must be papers about: you should always be able to use _independent_ bets as collateral for each other. Or, putting it another way, you should be able to share collateral between multiple independent bets. For example, if I place 100 bets, $1 each, on coinflips coming up heads, it should be obvious that $100 of collateral is too much to require. The probability that I will owe at least $60 is around 3%. The probability that I will owe $65 is almost negligible.