Has there been a work-up done yet about the regulatory hurdles to robust prediction markets and a road map to their navigation? I am feeling increasingly optimistic about the potential social benefits and would be curious to learn more about how to make the dream a reality
Also curious whether there are identified biases with the internet fans. They seem more likely than the average expert to pick up trends like Gamestonk (or even the early rise of Pres. Trump's trajectory), but I would bet there would be at least commensurate shortfalls in an internet-heavy-predictor market
I wonder how much of the lack of performance of prediction markets is due to the fact that a) it still feels a bit like gambling which isn't an easy mental model to get over, b) there just isn't enough liquidity as it doesn't seem embedded in regular analysis life and c) there still isn't a robin hood equivalent that makes this just ... easy. The first two require behaviour changes, where we all act like Tetlocks forecasters with the third enabling the first two.
At least for me the Schelling point I'm stuck at is to just not use it at all.
All that said it seems like the adoption would be like a step function. Once it's visible enough and liquid enough then the answers that come from the markets become useful, which drives a positive feedback loop of even more adoption and its integration into 538 and Vox.
If one could make a lot of money wagering on Covid deaths or on Covid mutations, that could either be an invitation to a bad (evil) actor or maybe a Netflix series.
Another thing that's great about metaculus is that it's very open and easy to get into. There is a tutorial and quiz where you can check how well-calibrated you are, and you can start answering questions and commenting, or even open your own questions, quite fast!
I wonder what kind of analysis has been done regarding which markets are better predictors. Metaculus has a track record page:
I'm both interested in and deeply confused by prediction markets as a concept. I'd love to see a tutorial about how these markets work mechanically, so I can feel confident placing money in them.
Examples of questions I have include: What am I actually buying when I buy a share? I believe Scott at one point said that a share pays out at some point if the prediction it corresponds to comes true, but where does that money come from? When a market first opens, who are you buying "fresh" stock from and how do they decide what to charge? Also, how can they run markets for complex questions, as opposed to just "yes/no" stuff?
These are the sorts of questions I've spent the last few years understanding the answers to for the "real" stock market, and there's both much more information published about those and much more incentive to invest time, and still very hard to understand. All the "you should care about prediction markets" stuff that has been shared with me so far is either very abstract ("they're just great, for reasons, trust me") or else incredibly in the weeds.
One problem that I see with the reasoning here is that there is still no evidence that the vaccines interrupt transmission of this virus. I was hoping that such evidence would have been published by now, and I would have thought that a vaccine that is 95% effective at preventing symptomatic disease would do so, so I am becoming worried. If this holds up, it would mean that some significant percentage of vaccinated patients can carry and transmit the virus. Your description seems to indicate that vaccinated patients cannot be infected and transmit the virus. Is there an indication of what directions these models would go if vaccinated patients can transmit the virus?
"Before 2023, will the United States CDC recommend that those who have already been vaccinated for SARS-CoV-2 (COVID-19) be vaccinated again due to a mutation in the virus?"
I would put the likely answer to the first part of that question at nearly 100%, although not for the specific reason in the latter clause of the question.
Infection or vaccination do not seem to confer permanent immunity, and the SARS-CoV-2 is almost certain to become endemic like its close relative "the common cold" and like influenza. If we're LUCKY, vaccination for "COVID-19" will become an annual event as with the flu vaccine. If we're less lucky, six months may be the norm.
Hypermind is pretty good. Like Metaculus, you trade in fake internet money. But Hypermind has some institutions/people who pay to put markets up, and that money is split between the people who made the most fake money on that particular market. Hypermind then will send you an Amazon gift card once you get to 15 Euros. So there's an extra incentive to spend your fake money wisely.
Philip Tetlock who wrote the Superforecasting book is also behind https://www.gjopen.com which also uses fake internet points. The questions are more varied and there's more user interaction.
He also got a group of people that were the best "superforecasters" and their predictions are now at https://goodjudgment.io/superforecasts/. I've been following it pretty closely and they've been tracking the question about vaccines better than Metaculus.
My intuitive concern about this concept is that A) useful insights will tend to get Dunning-Krugered into oblivion and B) people's betting choices are as much a matter of personal risk taking tendencies and disposable income as any confidence about what they're betting on.
Insofar as large parts of the financial industry are essentially prediction markets, I feel like their behavior generally vindicates these concerns.
I wonder if a site could strike a deal with regulators allowing a higher volume of trades if they only paid out based on users' average performance over a lot of predictions, rather than on individual trades.
That would make it much harder to win or lose money on luck alone, and the site would be more clearly a skill-based competition.
Re vaccine hesitancy, I haven't seen anyone talk about this but if I had been infected with COVID, I could imagine being hesitant to take a vaccine for some combination of reasons. (Do I need it? Do I need it before others? Is it worth any even slim risk, given that I already have antibodies?)
If 25% of the US has already had COVID, would we expect a lot of the hesitancy to come from that camp? Or maybe not? I haven't seen this discussed or studied but I'm curious about it.
I think this post leans a little too hard on efficient markets. By the end of the post at times you seemed to be treating the Metaculus probabilities as though they are the actual probabilities.
Metaculus obviously has less "dumb money" than PredictIt. But it still has weird biases or cases where the market is obviously wrong, due to Silicon Valley biases, long-running questions where people drop out and don't update, overoptimism about certain technologies (like reinforcement learning), etc.
We're probably still in the world where more hedging language is merited when reporting the probability coming out of a prediction market. They don't really give "the probability" yet, I don't think, although a thick and liquid enough market might, for some things.
Financial markets work well (to the extent they work well) because liquidity and speed of information transfer as well as transaction clearance effectively prevents arbitrage. You need to have no arbitrage to even have a theory of pricing that remotely corresponds to fundamental value, and prediction markets are seemingly useless if the bets don't correspond to the bettors' actual beliefs regarding fundamental probability and are instead attempts to benefit from arbitrage.
In financial markets, this happens because of market makers, institutions with enormous cash reserves of lines of credit ready to take the opposite side of any bet in order to profit purely off of juice. This is the same purpose a book serves in normal betting, except books also set the prices. This means the prices of bets are not even intended to reflect anything about the probability of an event happening, but rather the book's assessment of how much money is going to take each side of a bet at a particular price.
It's a subtle difference, but it can result in systematic deviation of betting lines from reality where some team can very consistently over or underperform their spreads for a long time because the betting public includes so much stupid money placing bets for bad reasons.
How do prediction markets avoid this? Seemingly, the point is to make optimal predictions, but every incentive from the perspective of the book is to make money off of systematic and predictable human irrationality instead.
This post succeeded in making me more interested in actively participating in Metaculus. I've been winding down my participation in PredictIt (mediocre place to put capital even if you can beat the average), but previously had no interest in Internet points. If you're planning on keeping up a recurring review of interesting developments in the space, I think it could actually drive a significant amount of traffic in their direction.
Wouldn't any full fledged prediction market have serious "insider trading" problems, at least for many predictions of interest? E.g. AOC, or anyone close to her, massively shorting herself just before dropping out? Typically, actors don't have their hand forced (like officials trying to avoid as many covid deaths as possible) but can choose from a number of options, right? Seems like a lot of infrastructure and regulation would be necessary to avoid such problems, likely more than for stock markets nsider trading.
For Europeans, I think a notable market is BetFair. Much lower prices than PredictIT (generally 5% on profits, but can be less with discounts) and very good liqudity on popular markets and when the topic is emotionally charged, you can still make easy money https://www.lesswrong.com/posts/y8RWtNBiksbSzm9j4/bet-on-biden
I understand the value of prediction markets, for where people are predicting their own behavior (who will I vote for, when will I get vaccinated), but I see a lot less value in predicting events that people have no real control over, like COVID deaths.
Sure, we can make a group guess, but why is that at all accurate? What info can I get from that prediction that I could not get from just reading the newspaper (the same info source that all the other punters have)?
Let's say we had a liquid, stable market for the Presidential election on election night. Do we really think it would have been stable, in favor of Biden, around 10pm EST? My guess is, everyone would have been watching the same numbers come in, and the bets would have swung wildly from Biden, to Trump, and back again by early Wednesday morning. All of that to say, if no one really knows anything (and certainly not anything different), why do these markets have any value?
I strongly suspect Scott has a bigger readership than the entire Metaculus userbase, and that approximately 100% of existing Metaculus users are subscribed here. If this becomes a regular feature, I expect a large bump in predictions on the questions featured in ASX every Monday, and for them to move in the direction hinted at by Scott's commentary. I guess the screenshots in the post are enough information to track that.
For the reinfection question (currently 277 predictions), the fine print says:
"If coronavirus infection confers partial immunity to the new strain, such that getting the disease is less likely but still possible, this may still count so long as scientific evidence exists (for example in a published paper) that the protection is significantly less for the new strain than the old."
And the discussion in the comments suggests that this is to be interpreted as 'statistically significant', in which case I think it's basically certain that there will be at least one variant with a statistically significantly different reinfection rate that infects that many people globally. In fact, I wouldn't be surprised if it's already happened. The main reason I'm not more confidently yes, is that the wording isn't completely clear on what it means by 'significant'.
I also think that the modal scenario is that we start needing annual COVID vaccines, and everyone I know who works on COVID modelling seems to agree with this. So far, we've only seen a small number of notable new variants, but that's without any selection pressure on the virus from vaccines. I guess the main way that this doesn't become a thing is if somehow the dominant strain is mild enough that vaccination isn't worthwhile (or politics happens, I suppose).
I tried out Metaculus myself after the last time it was mentioned. While I really like it. There were two things about it that bothered me.
First, when making a prediction using a probability density, I would expect that setting the variance to be very high would minimize the impact that that prediction would have to my score, regardless of the outcome. Instead it seems to have the opposite effect. Can anyone explain why that would be?
Second, it seems like most of the questions are fairly long term. Long term questions seem less useful for calibrating. Is there a way to find shorter term questions so I can make more progress?
It seems like the basic principle behind prediction markets is that most people want to make money and that smart money will overwhelm the noise from dumb money. This premise seems a bit shaky as of late? For a thinly-traded market, it seems like semi-organized dumb money from the 4chan/lizardman/McBoatface contingent is quite capable of launching a denial-of-service attack if they feel like it, basically by throwing money. Piling on in this way is apparently as much fun as gambling and as feasible as any Kickstarter. It seems like a good profit opportunity for people who can manage to catch the money they throw, but this doesn't prevent them from throwing, and so it's not so good for rational price discovery.
This is on top of skepticism about anything based on asking people questions without cross-examination. Scott has written about problems with surveys before, but for more skepticism, see @literalbanana's Survey Chicken essay: https://carcinisation.com/2020/12/11/survey-chicken/
If there is a denial-of-service attack on a prediction market then it will usually be obvious and we can just ignore the result. But perhaps there will be cases where the result is plausible?
Regulated as gambling, eh? What's the nearest Indian Reservation to Wall Street?
Also UK bookmakers are allowed to take bets on political questions. So you can check the odds for things like next POTUS (Kamala leading at 4) or PM (Sunak at 29/10) there.
(Bonus: senate almost certainly won't convict, Joe is more likely than not to make it to '24 and Puerto Rico probably won't be a state by '22) https://www.oddschecker.com/politics
ok nice intro to the prediction markets. more if this please. a little less arrogance @ "I don't think the average Metaculan truly understands the difficulty...", as well as contradiction "...this is a pretty impressive example of decentralized expertise weighing in..."
The FDA is already going to speed up clinical trials for COVID booster shots against new variants:
"Peter Marks, the head of the Food and Drug Administration division that oversees vaccines, said Friday that the agency will do what it can to speed the process. It won’t require big clinical trials, for instance. Rather than studies of tens of thousands of people, the agency will mandate much smaller studies of a few hundred. The goal would be to ensure that the vaccines produce the desired immune response and to see whether the products cover just the new variants or the original virus as well as the new variation, he said.
“We would intend to be pretty nimble with this . . . so that we can get these variants covered as quickly as possible,” Marks said on an American Medical Association webinar."
i'm also aware of prediqt : https://prediqt.com/ , which uses the EOS blockchain, and foretell : https://www.cset-foretell.com/ , run by georgetown university. i don't have much personal experience with either, but my dream for metaculus is for it to eventually aggregate other prediction markets and forecasts (like those from fivethirtyeight)
Is it possible to place error bars around the Metaculus prediction for deaths w/ and w/out challenge trials, based on the actual track record of similar questions? Evaluating counterfactuals such as how many people would have died if challenge trials were allowed to proceed is really only useful insofar as the Metaculus predictions in the past are somewhat accurate and the difference in predictions is outside the error range of past predictions.
I see that Metaculus has a track record page, but I'm not statistically advanced enough to convert their metrics to standard deviations around individual question predictions.
Most interesting near term prediction to me: the current petition for a recall election for CA Gov. Newsom at 75% chance of succeeding on Metaculus (about 65% on PredictIt). Something I've of course read about in general news but had no clue what the probability was.
How do prediction markets handle conditional hypotheticals such as "how many people would die if we did challenge trials"? My understanding is that prediction markets work by eventually paying out when the real world provides a result. But in the case of these conditionals there's no way to ever know the actual outcome. Do those "shares" all expire worthless? Are they refunded for purchase price? This seems pretty important for any proposed policy question (if we implement policy x, result will be y). I'd like to understand this mechanism better.
I wonder if it's legal to use virtual currencies in a prediction market? Between Apple/Google app stores, Steam and other game platforms, in-app and in-game purchases, etc., I'm pretty sure the amount spent in these online marketplaces is in the hundreds of billions globally. If these currencies were made sufficiently fungible with real money, that should be enough to attract investors.
Or, hell, could Bezos just run a prediction market that pays out in Amazon gift certificates? How do the gambling laws actually deal with such alternate currencies?
As a general practical question, even if prediction markets weren't classified as gambling and you could attract real investors dumping billions into predictions, I wonder how you protect them from manipulation?
Like, in a fully open market, how do ask a question like 'On what day will Company X announce their new vaccine' without Company X just dumping a billion dollars into that market on a much later day they're sure they'll be ready by, and waiting until then to announce?
I guess in that case the market did tell you when they would announce with great accuracy, but only by fucking up the thing being measured.
You can’t really say, “if only we had done challenge trials we would have saved 50,000 lives,” since the world where we did challenge trials looks different in other ways from the world where we didn’t e.g. additional bureaucracy lifted, or even greater urgency due to larger death rate.
> we don't have herd immunity for the danger period in winter 2021
Why are you assuming Covid is seasonal? As far as I can tell, Covid likes hot weather just fine. Summer was less deadly in North America because the big cities locked down and flyover country didn't have it yet, not because Covid doesn't like the heat. Brazil is getting hit badly in the summer heat now.
AFAIK, Governor Newsom of California is targeting June for general vaccine rollout, which makes the May 19th nationwide prediction somewhat optimistic.
I am a user of CSET Foretell, and it is interesting drawing comparisons between Foretell, a subsidized opinion pooling platform, and a traditional prediction market. The first difference is that it doesn't give a strong appearance of gambling, which is a plus in the policy making context. To get people to participate, there are some financial prizes such as lotteries for people who place in the top of the leaderboard or forecast during a time period. There are also social incentives though such as the ability to join teams, which have a separate leaderboard, and the feeling of contributing to pressing national security problems. This is separate from UI engineering that I think also adds great value like automatic reminders to update forecasts and historical data visualization for those of us who don't like having to track down and plot everything.
I think a likely future incentive would be CSET recruiting top forecasters into an elite opinion pool that consults with policymakers. It's not a coincidence that Cultivate Labs built both CSET's and the Good Judgment Project's platforms, with superforecaster cachet being a major reason to join Good Judgment Open.
I am convinced by arguments I've seen elsewhere that for real-money prediction markets to attract serious researchers placing smart bets, the markets need to be subsidized so that they are positive-sum rather than zero-sum for the participants. Commodities futures markets are zero-sum in terms of money, but positive-sum when you account for the benefit to the people who are able to use them to hedge their business. Those people effectively subsidize the market makers and speculators in exchange for the service of hedging.
If we are serious about getting a prediction market to answer a question, we need to subsidize the market with a bunch of money for the smart bettors to win.
I don't know much about prediction markets, so probably a dumb question: What about derivatives on predictions? Wouldn't trading in, say, options on a prediction provide incentives to game the system and destroy the predictive power of the original market?
So here are some other links and platforms you might be interested in:
- Good Judgment Open: https://www.gjopen.com/. Top 2% go on be selected as superforecasters, so amateur participants have some motivation. I have the impression that this platform is comparable to Metaculus in terms of accuracy, but I don't have actual data. They're particularly centered on geopolitical questions.
- CSET-foretell: cset-foretell.com/. Similar to Metaculus, but smaller and attempts to influence public policy in the US, particularly around transformative technology. Also sponsored by OpenPhilanthropy.
- Polymarket has already been mentioned by other people. Also by you in the past: "I got emails from two different prediction aggregators saying they would show they cared by opening markets into whether the Times would end up doxxing me or not. One of them ended up with a total trade volume in the four digits. For a brief moment, I probably had more advanced decision-making technology advising me in my stupid conflict with a newspaper than the CIA uses for some wars. I am humbled by their support."; Polymarket was one of them, though the market was from their old site and thus only remains in the archives. https://web.archive.org/web/20200717060418/https://poly.market/market/will-the-new-york-times-publish-an-article-revealing-scott-alexanders-full-name In particular, PolyMarket might have boring markets, but they have recently been great for making money from unsophisticated traders.
- Omen. https://omen.eth.link/ This seems to be truly decentralized. They sometimes use Kleros (kleros.io), a decentralized judging system which I find really elegant. They haven't seen much volume recently, probably because of usability problems / high ETH fees. They powered https://coronainformationmarkets.com/ for a while, but they didn't see much volume either.
- https://metaforecast.org/: A search engine, but for probabilities. In very early beta. But the idea is that you can just search for something, like "covid", and it will get you the forecasts from the platforms I mentioned above (and PredictIt, which you also mention)
-I also worked with foretold.io at some point in the past. They're good if you want to create your own private communities and forecasts with friends, and it has a powerful distribution editor based on the one from Guesstimate, but which takes some getting used to. It also hasn't seen much activity recently, but I've been working on using it as a backend for making large numbers of predictions at once. As an example cool thing one can do, a while back I entered things Dominic Cummings said: https://www.foretold.io/c/952d822e-b979-4c42-8560-7f878ad23c9e?state=pending, to see how accurate he was (I haven't seen how these resolve yet, though)
At least to start with, I'd be much more interested in a backwards-looking analysis of how well these markets have done at predicting events in the past.
My prediction for US coronavirus deaths as of late last year was 665,000 (median). Posted and defended on DSL, where I hope it earned me a bit of respect from my SSC/DSL/ACX peers, which is a sort of "fake internet point" that I understand and place some value on. Enough to do about an hour of dedicated research and math in that particular case, on top of the general Covid situational awareness stuff that I'd have been doing anyway.
If there'd been a way to make serious money off that sort of prediction, I'd have put in a lot more than an hour's work. And judging by the Metaculus results, probably come to about the same conclusion and made no serious money, but worth a try at least. I don't understand the value of Metaculus fake internet points well enough to devote any great length of time pursuing them. If that changes, and if my understanding of MFIPs is that they are really good things to have, then I'll probably engage with Metaculus.
If it's just that there's a small nerdy community that will respect me for my Big Number, I've already got a big IQ I can brag about to people who care about that sort of thing, and I've got a community where I can earn a more nuanced and informed sort of respect than "He has a Big Number".
FYI Scott, the US intelligence community had an internal prediction market only accessible from TS networks until maybe mid last year. I enrolled in it a few years ago, but never personally made any predictions due to just not having any basis for making a good guess about anything without investing a ton of time.
Theoretically, this probably should have performed better than a public prediction market since the participants have access to classified information. I didn't pay enough attention to know to what extent it was ever evaluated for predictive accuracy compared to individual experts and public prediction markets. Of course, you couldn't earn actual money, so I think lack of activity is the biggest reason it shut down and may not have outperformed public markets even with the additional information.
It was voluntary and not used to inform real policy decisions. I can't remember now which agency set this up, but probably NSA. They tend to be behind most of the classified but otherwise public-facing applications anyone with IC PKI identities can get to.
On a semi-related note, something I miss about no longer working in a SCIF is, even with a clearance, I can't access these sites any more since I don't have physical access to a connected workstation. One of the more interesting things you learn when you first get access to the same briefings given to the Joint Chiefs of Staff every day is just how wrong a lot of public news actually is, along with how much isn't really unknown but only not publicly revealed to avoid revealing that we have the capability to know it.
Is there any kind of research on the differences in cognitive bias between risk-averse and risk-prone individuals? Because it feels like prediction markets definitely attract more risk-prone people.
Like some other commenters, I'm puzzled by Scott's implied assumption that the consensus that forms around any prediction in a prediction market will be inherently valuable (not to say accurate). Whay should this be? How can we know that predictors in such markets aren't simply pooling their ignorance?
Is there an assumption that such markets will tend to be populated by people who are more likely to be accurate predictors? If so, what's the evidence for that assumption?
Excellent explication of conditional prediction markets! (For those who don't know, those are the basis of Robin Hanson's Futarchy.)
Sadly, I think the truest true answer to many forecasting questions is "no one has any clue, it's an utter coin flip" and no amount of market liquidity will change that. It's possible that's what's happening with the two conditional markets for COVID deaths with and without challenge trials. They're saying "339k deaths with" and "426k deaths without" but maybe what that really means in both cases is "gosh, no clue, a few hundred thousand ish I guess?" The Metaculus distributions for the with-vs-without predictions do mostly overlap each other. Still, lived saved in expectation is a big deal!
So I'm not actually disagreeing with anything here.
> of course, you can still buy all the Gamestop stock you want
this gave me an idea: can't we hijack stocks to turn them into predictions ?
for each bet create two corporations, for example the Trump Will Get Impeached Corporation and the Trump Will Not Get Impeached Corporation. they'd both be contractually obligated to, if they lose give all of their wealth to the other one, and if they win, buy the shares back from all shareholders at the new, increased value.
Ugh. What are covid "deaths"? The PCR test isn't standardized, and by varying the number of cycles, you can vary the covid death rate. What if someone who is in control of the number of cycles also bets on the number of deaths?
BTW, PCR is why the death prediction keeps rising -- because PCR counts people who were merely exposed but destroyed the virus, people who were exposed and had no or very mild symptoms, people who were sick but aren't anymore, and finally the ones who are showing symptoms. Eventually everyone will fit into one of those categories and so every death will be a covid death.
1. The persistent popularity of casinos, lotteries, and other forms of gambling indicate that yes, people are more than happy to throw their money away on bad odds. No matter how bad someone is at predicting, they probably still aren't bad enough to drop their odds lower than that of buying a lottery ticket (i.e., they aren't being any less rational). And yet lottery ticket buyers still persist. This call into question the ability of prediction markets to "select" for high-knowledge participants over the long run
2. The reason this works, is because prediction markets (and all other forms of gambling) aren't closed systems. Sure, someone who makes all of their money Blackjack, and funnels 100% of their earning back into more Blackjack, will eventually sink-or-swim based on their Blackjack playing skills. But in practice, most people spend a small portion of their earnings on Blackjack, and have outside sources of income, such that losing in Blackjack is only a minor deterrence to continuing play. So long as people have outside jobs, "dumb-money" will continually pour into prediction markets with little consequence.
3. More abstractly: Due to the diminishing marginal utility of money, "dollars do not equal knowledge units". A wealthy person would gladly gamble away $10,000 on a whim, while a poor person would only bet this if he know the outcome with near-certainty. This means that the degree of knowledge a person has in regards to an event occurring, *cannot be directly inferred by the amount they are willing to bet*. The implications of this are significant. Because for prediction markets, pooling dollars bet ≠ the aggregated knowledge of an event, so you cannot even say with certainty in a market with (for example) 60% yes and 40% no, "the bettors in this market think the event is more likely to occur than not". It's *evidence* for it, sure, but other forms of polling may find different outcomes in the same group of people, if they adjust for levels of certainty using different measures.
4. The common sentiment in the rationalist community seems to be something like "betting markets are beautiful in theory, but flawed in practice". I take the inverse position: Betting markets are flawed in theory, but have practical application so long as you understand their theoretical limitations (e.g., they are poll-like things that basically conduct themselves, for free). It's good that we have these prediction markets, because having information is better than not having information. But that's as far as my hope goes: If there was some way to bet against prediction markets eclipsing the usefulness of traditional polling in the coming years, I would.
I have the impression that he would give quite different predictions than the average / median on Metaculus. If only there were more than fake internet points up for grabs there, he might have an incentive to participate in those predictions.
I go through every question in Metaculus and if I don't have any prediction of my own, I jsut pick what the crowd picks. I want to see how well I perform over time and get those super nifty Tachyons. I don't actually care about the future...just winning.
I did PredictIt this election cycle and made around $200 on a $500 investment. Never again. What a horrible experience. First its pure tribalism distilled and slammed one shot after the next. Second, the rules are super specific (which they should be and sometimes aren't in Metaculus--though discussion usually helps) and people make money off of gaming the rules of a question (This is the best example: https://www.predictit.org/markets/detail/4353). third, it's far more important to understand trading (hedging, arbitrage, etc.) than making correct guesses. In the market I listed above, somehow, they pumped the price of Republicans winning the senate after the Georgia election because of the rules and a ton of people made money by choosing the wrong answer. I think Metaculus avoids this problem beautifully.
I support Metaculus Mondays. There's even some questions I'm interested in, like, "When will Epistemic Humility become a common phrase?" Support them on Patreon: https://www.patreon.com/metaculus/posts
I started to type out a snarky comment about how 6% of PredictIt users don't even realize AOC won't be old enough in 2024, and then thought I'd better google it. Turns out she'll turn 35 three weeks before the election, which makes her several years older than I thought she was. Interesting.
Using prediction markets to inform governmental policy seems like it could create some really perverse incentives. For instance, if the question is "What will be the employment rate in we (rasie/don't raise) the minimum wage to $15?"
A government will raise the minimum wage if the predicted decrease in employment isn't too great. Doesn't this create strong incentives for business owners to enter the market and enter overly dramatic predictions of employment decreases in order to manipulate policy makers into not implementing a policy that goes against their interests?
(2)
Curious about how these markets work with actual money. For binary predictions it seems simple enough. I'm less sure for continous predictions. Is payout proportional to how close the predictor's value was to the actual value? How does the money placed get distributed?
(3)
What is the consensus view on cryptocurrency-based / decentralized prediction markets? I see a few built ontop of etherium (https://defiprime.com/prediction-markets) but I haven't heard nearly as much about them as I've heard about Metaculus. Do they lack the technical expertise and polish that Metaculus offers? Or is it just that the inherent exoticness of cryptocurrencies makes them less accessible than mainstream alternatives?
This seems like the perfect use case for crypto: A clear need to introduce actual money into prediction markets + the technology to skirt government regulation
I have a question that I think is really obvious, but I don't see it being addressed. I wonder if I'm missing something.
If one were to use prediction markets in the "creepy deep magic way", i.e. asking people what the consequences of political decisions would be - wouldn't people want to influence the outcome to support their political leanings?
I understand that this would come at some cost for them, and this surely is a correcting factor, but I would guess that political motivations might still sometimes drastically alter prediction poll results.
Sorry for any grammatical mistakes, this is not my native tongue.
Has there been a work-up done yet about the regulatory hurdles to robust prediction markets and a road map to their navigation? I am feeling increasingly optimistic about the potential social benefits and would be curious to learn more about how to make the dream a reality
Also curious whether there are identified biases with the internet fans. They seem more likely than the average expert to pick up trends like Gamestonk (or even the early rise of Pres. Trump's trajectory), but I would bet there would be at least commensurate shortfalls in an internet-heavy-predictor market
Facebook just released a prediction market app called Forecast. Uses fake money, but has pretty decent execution on the UI/UX.
https://forecastapp.net/
I wonder how much of the lack of performance of prediction markets is due to the fact that a) it still feels a bit like gambling which isn't an easy mental model to get over, b) there just isn't enough liquidity as it doesn't seem embedded in regular analysis life and c) there still isn't a robin hood equivalent that makes this just ... easy. The first two require behaviour changes, where we all act like Tetlocks forecasters with the third enabling the first two.
At least for me the Schelling point I'm stuck at is to just not use it at all.
All that said it seems like the adoption would be like a step function. Once it's visible enough and liquid enough then the answers that come from the markets become useful, which drives a positive feedback loop of even more adoption and its integration into 538 and Vox.
If one could make a lot of money wagering on Covid deaths or on Covid mutations, that could either be an invitation to a bad (evil) actor or maybe a Netflix series.
Another thing that's great about metaculus is that it's very open and easy to get into. There is a tutorial and quiz where you can check how well-calibrated you are, and you can start answering questions and commenting, or even open your own questions, quite fast!
I wonder what kind of analysis has been done regarding which markets are better predictors. Metaculus has a track record page:
https://www.metaculus.com/questions/track-record/ (unsure if links are allowed, but let me try)
Do the other contenders have something similar, and have they been compared?
Typo thread:
There seems to be a spurious link pasted in the last sentence of the Augur section.
I'm both interested in and deeply confused by prediction markets as a concept. I'd love to see a tutorial about how these markets work mechanically, so I can feel confident placing money in them.
Examples of questions I have include: What am I actually buying when I buy a share? I believe Scott at one point said that a share pays out at some point if the prediction it corresponds to comes true, but where does that money come from? When a market first opens, who are you buying "fresh" stock from and how do they decide what to charge? Also, how can they run markets for complex questions, as opposed to just "yes/no" stuff?
These are the sorts of questions I've spent the last few years understanding the answers to for the "real" stock market, and there's both much more information published about those and much more incentive to invest time, and still very hard to understand. All the "you should care about prediction markets" stuff that has been shared with me so far is either very abstract ("they're just great, for reasons, trust me") or else incredibly in the weeds.
One problem that I see with the reasoning here is that there is still no evidence that the vaccines interrupt transmission of this virus. I was hoping that such evidence would have been published by now, and I would have thought that a vaccine that is 95% effective at preventing symptomatic disease would do so, so I am becoming worried. If this holds up, it would mean that some significant percentage of vaccinated patients can carry and transmit the virus. Your description seems to indicate that vaccinated patients cannot be infected and transmit the virus. Is there an indication of what directions these models would go if vaccinated patients can transmit the virus?
"Before 2023, will the United States CDC recommend that those who have already been vaccinated for SARS-CoV-2 (COVID-19) be vaccinated again due to a mutation in the virus?"
I would put the likely answer to the first part of that question at nearly 100%, although not for the specific reason in the latter clause of the question.
Infection or vaccination do not seem to confer permanent immunity, and the SARS-CoV-2 is almost certain to become endemic like its close relative "the common cold" and like influenza. If we're LUCKY, vaccination for "COVID-19" will become an annual event as with the flu vaccine. If we're less lucky, six months may be the norm.
"But,https://www.metaculus.com/questions/5908/confirmed-us-covid-deaths-by-2022/ like the real Messiah, it’s taking its sweet time."
Hypermind is pretty good. Like Metaculus, you trade in fake internet money. But Hypermind has some institutions/people who pay to put markets up, and that money is split between the people who made the most fake money on that particular market. Hypermind then will send you an Amazon gift card once you get to 15 Euros. So there's an extra incentive to spend your fake money wisely.
Philip Tetlock who wrote the Superforecasting book is also behind https://www.gjopen.com which also uses fake internet points. The questions are more varied and there's more user interaction.
He also got a group of people that were the best "superforecasters" and their predictions are now at https://goodjudgment.io/superforecasts/. I've been following it pretty closely and they've been tracking the question about vaccines better than Metaculus.
FYI the word for a Metaculus user is a 'Metaculite'. See e.g. https://www.metaculus.com/questions/4786/what-will-be-the-minimum-credence-metaculites-will-give-trumps-re-election-chances-in-2020/
My intuitive concern about this concept is that A) useful insights will tend to get Dunning-Krugered into oblivion and B) people's betting choices are as much a matter of personal risk taking tendencies and disposable income as any confidence about what they're betting on.
Insofar as large parts of the financial industry are essentially prediction markets, I feel like their behavior generally vindicates these concerns.
Am I missing something?
I wonder if a site could strike a deal with regulators allowing a higher volume of trades if they only paid out based on users' average performance over a lot of predictions, rather than on individual trades.
That would make it much harder to win or lose money on luck alone, and the site would be more clearly a skill-based competition.
Re vaccine hesitancy, I haven't seen anyone talk about this but if I had been infected with COVID, I could imagine being hesitant to take a vaccine for some combination of reasons. (Do I need it? Do I need it before others? Is it worth any even slim risk, given that I already have antibodies?)
If 25% of the US has already had COVID, would we expect a lot of the hesitancy to come from that camp? Or maybe not? I haven't seen this discussed or studied but I'm curious about it.
Does anyone know whether there's a UK equivalent of PredictIt?
I think this post leans a little too hard on efficient markets. By the end of the post at times you seemed to be treating the Metaculus probabilities as though they are the actual probabilities.
Metaculus obviously has less "dumb money" than PredictIt. But it still has weird biases or cases where the market is obviously wrong, due to Silicon Valley biases, long-running questions where people drop out and don't update, overoptimism about certain technologies (like reinforcement learning), etc.
We're probably still in the world where more hedging language is merited when reporting the probability coming out of a prediction market. They don't really give "the probability" yet, I don't think, although a thick and liquid enough market might, for some things.
The prediction markets I looked at did terrible for the presidential primaries.
Financial markets work well (to the extent they work well) because liquidity and speed of information transfer as well as transaction clearance effectively prevents arbitrage. You need to have no arbitrage to even have a theory of pricing that remotely corresponds to fundamental value, and prediction markets are seemingly useless if the bets don't correspond to the bettors' actual beliefs regarding fundamental probability and are instead attempts to benefit from arbitrage.
In financial markets, this happens because of market makers, institutions with enormous cash reserves of lines of credit ready to take the opposite side of any bet in order to profit purely off of juice. This is the same purpose a book serves in normal betting, except books also set the prices. This means the prices of bets are not even intended to reflect anything about the probability of an event happening, but rather the book's assessment of how much money is going to take each side of a bet at a particular price.
It's a subtle difference, but it can result in systematic deviation of betting lines from reality where some team can very consistently over or underperform their spreads for a long time because the betting public includes so much stupid money placing bets for bad reasons.
How do prediction markets avoid this? Seemingly, the point is to make optimal predictions, but every incentive from the perspective of the book is to make money off of systematic and predictable human irrationality instead.
This post succeeded in making me more interested in actively participating in Metaculus. I've been winding down my participation in PredictIt (mediocre place to put capital even if you can beat the average), but previously had no interest in Internet points. If you're planning on keeping up a recurring review of interesting developments in the space, I think it could actually drive a significant amount of traffic in their direction.
Wouldn't any full fledged prediction market have serious "insider trading" problems, at least for many predictions of interest? E.g. AOC, or anyone close to her, massively shorting herself just before dropping out? Typically, actors don't have their hand forced (like officials trying to avoid as many covid deaths as possible) but can choose from a number of options, right? Seems like a lot of infrastructure and regulation would be necessary to avoid such problems, likely more than for stock markets nsider trading.
For Europeans, I think a notable market is BetFair. Much lower prices than PredictIT (generally 5% on profits, but can be less with discounts) and very good liqudity on popular markets and when the topic is emotionally charged, you can still make easy money https://www.lesswrong.com/posts/y8RWtNBiksbSzm9j4/bet-on-biden
I understand the value of prediction markets, for where people are predicting their own behavior (who will I vote for, when will I get vaccinated), but I see a lot less value in predicting events that people have no real control over, like COVID deaths.
Sure, we can make a group guess, but why is that at all accurate? What info can I get from that prediction that I could not get from just reading the newspaper (the same info source that all the other punters have)?
Let's say we had a liquid, stable market for the Presidential election on election night. Do we really think it would have been stable, in favor of Biden, around 10pm EST? My guess is, everyone would have been watching the same numbers come in, and the bets would have swung wildly from Biden, to Trump, and back again by early Wednesday morning. All of that to say, if no one really knows anything (and certainly not anything different), why do these markets have any value?
I strongly suspect Scott has a bigger readership than the entire Metaculus userbase, and that approximately 100% of existing Metaculus users are subscribed here. If this becomes a regular feature, I expect a large bump in predictions on the questions featured in ASX every Monday, and for them to move in the direction hinted at by Scott's commentary. I guess the screenshots in the post are enough information to track that.
For the reinfection question (currently 277 predictions), the fine print says:
"If coronavirus infection confers partial immunity to the new strain, such that getting the disease is less likely but still possible, this may still count so long as scientific evidence exists (for example in a published paper) that the protection is significantly less for the new strain than the old."
And the discussion in the comments suggests that this is to be interpreted as 'statistically significant', in which case I think it's basically certain that there will be at least one variant with a statistically significantly different reinfection rate that infects that many people globally. In fact, I wouldn't be surprised if it's already happened. The main reason I'm not more confidently yes, is that the wording isn't completely clear on what it means by 'significant'.
I also think that the modal scenario is that we start needing annual COVID vaccines, and everyone I know who works on COVID modelling seems to agree with this. So far, we've only seen a small number of notable new variants, but that's without any selection pressure on the virus from vaccines. I guess the main way that this doesn't become a thing is if somehow the dominant strain is mild enough that vaccination isn't worthwhile (or politics happens, I suppose).
I tried out Metaculus myself after the last time it was mentioned. While I really like it. There were two things about it that bothered me.
First, when making a prediction using a probability density, I would expect that setting the variance to be very high would minimize the impact that that prediction would have to my score, regardless of the outcome. Instead it seems to have the opposite effect. Can anyone explain why that would be?
Second, it seems like most of the questions are fairly long term. Long term questions seem less useful for calibrating. Is there a way to find shorter term questions so I can make more progress?
It seems like the basic principle behind prediction markets is that most people want to make money and that smart money will overwhelm the noise from dumb money. This premise seems a bit shaky as of late? For a thinly-traded market, it seems like semi-organized dumb money from the 4chan/lizardman/McBoatface contingent is quite capable of launching a denial-of-service attack if they feel like it, basically by throwing money. Piling on in this way is apparently as much fun as gambling and as feasible as any Kickstarter. It seems like a good profit opportunity for people who can manage to catch the money they throw, but this doesn't prevent them from throwing, and so it's not so good for rational price discovery.
This is on top of skepticism about anything based on asking people questions without cross-examination. Scott has written about problems with surveys before, but for more skepticism, see @literalbanana's Survey Chicken essay: https://carcinisation.com/2020/12/11/survey-chicken/
If there is a denial-of-service attack on a prediction market then it will usually be obvious and we can just ignore the result. But perhaps there will be cases where the result is plausible?
Regulated as gambling, eh? What's the nearest Indian Reservation to Wall Street?
Also UK bookmakers are allowed to take bets on political questions. So you can check the odds for things like next POTUS (Kamala leading at 4) or PM (Sunak at 29/10) there.
(Bonus: senate almost certainly won't convict, Joe is more likely than not to make it to '24 and Puerto Rico probably won't be a state by '22) https://www.oddschecker.com/politics
ok nice intro to the prediction markets. more if this please. a little less arrogance @ "I don't think the average Metaculan truly understands the difficulty...", as well as contradiction "...this is a pretty impressive example of decentralized expertise weighing in..."
The FDA is already going to speed up clinical trials for COVID booster shots against new variants:
"Peter Marks, the head of the Food and Drug Administration division that oversees vaccines, said Friday that the agency will do what it can to speed the process. It won’t require big clinical trials, for instance. Rather than studies of tens of thousands of people, the agency will mandate much smaller studies of a few hundred. The goal would be to ensure that the vaccines produce the desired immune response and to see whether the products cover just the new variants or the original virus as well as the new variation, he said.
“We would intend to be pretty nimble with this . . . so that we can get these variants covered as quickly as possible,” Marks said on an American Medical Association webinar."
https://www.washingtonpost.com/health/covid-mutations-herd-immunity/2021/01/30/0741722e-627c-11eb-9430-e7c77b5b0297_story.html
i'm also aware of prediqt : https://prediqt.com/ , which uses the EOS blockchain, and foretell : https://www.cset-foretell.com/ , run by georgetown university. i don't have much personal experience with either, but my dream for metaculus is for it to eventually aggregate other prediction markets and forecasts (like those from fivethirtyeight)
Is it possible to place error bars around the Metaculus prediction for deaths w/ and w/out challenge trials, based on the actual track record of similar questions? Evaluating counterfactuals such as how many people would have died if challenge trials were allowed to proceed is really only useful insofar as the Metaculus predictions in the past are somewhat accurate and the difference in predictions is outside the error range of past predictions.
I see that Metaculus has a track record page, but I'm not statistically advanced enough to convert their metrics to standard deviations around individual question predictions.
Most interesting near term prediction to me: the current petition for a recall election for CA Gov. Newsom at 75% chance of succeeding on Metaculus (about 65% on PredictIt). Something I've of course read about in general news but had no clue what the probability was.
How do prediction markets handle conditional hypotheticals such as "how many people would die if we did challenge trials"? My understanding is that prediction markets work by eventually paying out when the real world provides a result. But in the case of these conditionals there's no way to ever know the actual outcome. Do those "shares" all expire worthless? Are they refunded for purchase price? This seems pretty important for any proposed policy question (if we implement policy x, result will be y). I'd like to understand this mechanism better.
I wonder if it's legal to use virtual currencies in a prediction market? Between Apple/Google app stores, Steam and other game platforms, in-app and in-game purchases, etc., I'm pretty sure the amount spent in these online marketplaces is in the hundreds of billions globally. If these currencies were made sufficiently fungible with real money, that should be enough to attract investors.
Or, hell, could Bezos just run a prediction market that pays out in Amazon gift certificates? How do the gambling laws actually deal with such alternate currencies?
As a general practical question, even if prediction markets weren't classified as gambling and you could attract real investors dumping billions into predictions, I wonder how you protect them from manipulation?
Like, in a fully open market, how do ask a question like 'On what day will Company X announce their new vaccine' without Company X just dumping a billion dollars into that market on a much later day they're sure they'll be ready by, and waiting until then to announce?
I guess in that case the market did tell you when they would announce with great accuracy, but only by fucking up the thing being measured.
Seems like a hard problem.
You can’t really say, “if only we had done challenge trials we would have saved 50,000 lives,” since the world where we did challenge trials looks different in other ways from the world where we didn’t e.g. additional bureaucracy lifted, or even greater urgency due to larger death rate.
> we don't have herd immunity for the danger period in winter 2021
Why are you assuming Covid is seasonal? As far as I can tell, Covid likes hot weather just fine. Summer was less deadly in North America because the big cities locked down and flyover country didn't have it yet, not because Covid doesn't like the heat. Brazil is getting hit badly in the summer heat now.
https://www.cnn.com/2021/01/27/americas/manaus-brazil-covid-19-new-variant-intl/index.html
AFAIK, Governor Newsom of California is targeting June for general vaccine rollout, which makes the May 19th nationwide prediction somewhat optimistic.
I am a user of CSET Foretell, and it is interesting drawing comparisons between Foretell, a subsidized opinion pooling platform, and a traditional prediction market. The first difference is that it doesn't give a strong appearance of gambling, which is a plus in the policy making context. To get people to participate, there are some financial prizes such as lotteries for people who place in the top of the leaderboard or forecast during a time period. There are also social incentives though such as the ability to join teams, which have a separate leaderboard, and the feeling of contributing to pressing national security problems. This is separate from UI engineering that I think also adds great value like automatic reminders to update forecasts and historical data visualization for those of us who don't like having to track down and plot everything.
I think a likely future incentive would be CSET recruiting top forecasters into an elite opinion pool that consults with policymakers. It's not a coincidence that Cultivate Labs built both CSET's and the Good Judgment Project's platforms, with superforecaster cachet being a major reason to join Good Judgment Open.
There are no coronavirus predictions due to the tight question moderation, but if you want a worrying prediction, the crowd forecast of a military conflict in the South China Sea before July is 11%. https://www.cset-foretell.com/questions/85-will-the-chinese-military-or-other-maritime-security-forces-fire-upon-another-country-s-civil-or-military-vessel-in-the-south-china-sea-between-january-1-and-june-30-2021-inclusive
I am convinced by arguments I've seen elsewhere that for real-money prediction markets to attract serious researchers placing smart bets, the markets need to be subsidized so that they are positive-sum rather than zero-sum for the participants. Commodities futures markets are zero-sum in terms of money, but positive-sum when you account for the benefit to the people who are able to use them to hedge their business. Those people effectively subsidize the market makers and speculators in exchange for the service of hedging.
If we are serious about getting a prediction market to answer a question, we need to subsidize the market with a bunch of money for the smart bettors to win.
I don't know much about prediction markets, so probably a dumb question: What about derivatives on predictions? Wouldn't trading in, say, options on a prediction provide incentives to game the system and destroy the predictive power of the original market?
I am not an American and thus I wish metaculus was less focused on US-centric questions which I usually don't care about :(
So here are some other links and platforms you might be interested in:
- Good Judgment Open: https://www.gjopen.com/. Top 2% go on be selected as superforecasters, so amateur participants have some motivation. I have the impression that this platform is comparable to Metaculus in terms of accuracy, but I don't have actual data. They're particularly centered on geopolitical questions.
- CSET-foretell: cset-foretell.com/. Similar to Metaculus, but smaller and attempts to influence public policy in the US, particularly around transformative technology. Also sponsored by OpenPhilanthropy.
- Hypermind: http://hypermind.com/. Similar to PredictIt, but Frenchy. They recently got >$7k from OpenPhilanthropy to have some COVID-19 predictions, which can be seen here: https://prod.hypermind.com/ngdp/en/showcase2/showcase.html?sc=Covid19
- Polymarket has already been mentioned by other people. Also by you in the past: "I got emails from two different prediction aggregators saying they would show they cared by opening markets into whether the Times would end up doxxing me or not. One of them ended up with a total trade volume in the four digits. For a brief moment, I probably had more advanced decision-making technology advising me in my stupid conflict with a newspaper than the CIA uses for some wars. I am humbled by their support."; Polymarket was one of them, though the market was from their old site and thus only remains in the archives. https://web.archive.org/web/20200717060418/https://poly.market/market/will-the-new-york-times-publish-an-article-revealing-scott-alexanders-full-name In particular, PolyMarket might have boring markets, but they have recently been great for making money from unsophisticated traders.
- PredictionBook. https://predictionbook.com/. This site is *old* Gwern was a power user for a while. You can see other people's predictions here: https://predictionbook.com/predictions
- Omen. https://omen.eth.link/ This seems to be truly decentralized. They sometimes use Kleros (kleros.io), a decentralized judging system which I find really elegant. They haven't seen much volume recently, probably because of usability problems / high ETH fees. They powered https://coronainformationmarkets.com/ for a while, but they didn't see much volume either.
- Elicit is another forecast aggregator, which I associate with casually eliciting probabilities from many people at once. It can be embedded on LessWrong, and substack could maybe get it to work as well (so e.g., people to make forecasts while they're reading SSC). See here: https://www.lesswrong.com/posts/JLrnbThMyCYDBa6Gu/embedded-interactive-predictions-on-lesswrong-2 (embedded), or here: https://www.lesswrong.com/posts/hQysqfSEzciRazx8k/forecasting-thread-ai-timelines (not embedded, but still cool). Also, you might be able to embed Metaculus predictions as well?
- Man, I can't believe you missed https://www.replicationmarkets.com/ and https://socialscienceprediction.org/. They try to predict whether papers will replicate. The first one used to pay really well ($100k in prizes), but they are in between projects right now.
I also have several things of my own:
- https://metaforecast.org/: A search engine, but for probabilities. In very early beta. But the idea is that you can just search for something, like "covid", and it will get you the forecasts from the platforms I mentioned above (and PredictIt, which you also mention)
- I have a monthly forecasting newsletter: https://forecasting.substack.com, which covers forecasting related stuff. You might be interested in the edition for January 2021, which I just posted: https://forecasting.substack.com/p/forecasting-newsletter-january-2021, or in an overview of 2020: https://forecasting.substack.com/p/2020-forecasting-in-review
- This list: https://docs.google.com/spreadsheets/d/1XB1GHfizNtVYTOAD_uOyBLEyl_EV7hVtDYDXLQwgT7k/ of prediction markets, created by Jacob Lagerros which I also try to mantain.
-I also worked with foretold.io at some point in the past. They're good if you want to create your own private communities and forecasts with friends, and it has a powerful distribution editor based on the one from Guesstimate, but which takes some getting used to. It also hasn't seen much activity recently, but I've been working on using it as a backend for making large numbers of predictions at once. As an example cool thing one can do, a while back I entered things Dominic Cummings said: https://www.foretold.io/c/952d822e-b979-4c42-8560-7f878ad23c9e?state=pending, to see how accurate he was (I haven't seen how these resolve yet, though)
At least to start with, I'd be much more interested in a backwards-looking analysis of how well these markets have done at predicting events in the past.
My prediction for US coronavirus deaths as of late last year was 665,000 (median). Posted and defended on DSL, where I hope it earned me a bit of respect from my SSC/DSL/ACX peers, which is a sort of "fake internet point" that I understand and place some value on. Enough to do about an hour of dedicated research and math in that particular case, on top of the general Covid situational awareness stuff that I'd have been doing anyway.
If there'd been a way to make serious money off that sort of prediction, I'd have put in a lot more than an hour's work. And judging by the Metaculus results, probably come to about the same conclusion and made no serious money, but worth a try at least. I don't understand the value of Metaculus fake internet points well enough to devote any great length of time pursuing them. If that changes, and if my understanding of MFIPs is that they are really good things to have, then I'll probably engage with Metaculus.
If it's just that there's a small nerdy community that will respect me for my Big Number, I've already got a big IQ I can brag about to people who care about that sort of thing, and I've got a community where I can earn a more nuanced and informed sort of respect than "He has a Big Number".
FYI Scott, the US intelligence community had an internal prediction market only accessible from TS networks until maybe mid last year. I enrolled in it a few years ago, but never personally made any predictions due to just not having any basis for making a good guess about anything without investing a ton of time.
Theoretically, this probably should have performed better than a public prediction market since the participants have access to classified information. I didn't pay enough attention to know to what extent it was ever evaluated for predictive accuracy compared to individual experts and public prediction markets. Of course, you couldn't earn actual money, so I think lack of activity is the biggest reason it shut down and may not have outperformed public markets even with the additional information.
It was voluntary and not used to inform real policy decisions. I can't remember now which agency set this up, but probably NSA. They tend to be behind most of the classified but otherwise public-facing applications anyone with IC PKI identities can get to.
On a semi-related note, something I miss about no longer working in a SCIF is, even with a clearance, I can't access these sites any more since I don't have physical access to a connected workstation. One of the more interesting things you learn when you first get access to the same briefings given to the Joint Chiefs of Staff every day is just how wrong a lot of public news actually is, along with how much isn't really unknown but only not publicly revealed to avoid revealing that we have the capability to know it.
Is there any kind of research on the differences in cognitive bias between risk-averse and risk-prone individuals? Because it feels like prediction markets definitely attract more risk-prone people.
Like some other commenters, I'm puzzled by Scott's implied assumption that the consensus that forms around any prediction in a prediction market will be inherently valuable (not to say accurate). Whay should this be? How can we know that predictors in such markets aren't simply pooling their ignorance?
Is there an assumption that such markets will tend to be populated by people who are more likely to be accurate predictors? If so, what's the evidence for that assumption?
Mike Tenpence
Hillary Clinton was 5% to be the nominee in March 2020. QAnon has given lots of rational PredictIt bettors free money.
Excellent explication of conditional prediction markets! (For those who don't know, those are the basis of Robin Hanson's Futarchy.)
Sadly, I think the truest true answer to many forecasting questions is "no one has any clue, it's an utter coin flip" and no amount of market liquidity will change that. It's possible that's what's happening with the two conditional markets for COVID deaths with and without challenge trials. They're saying "339k deaths with" and "426k deaths without" but maybe what that really means in both cases is "gosh, no clue, a few hundred thousand ish I guess?" The Metaculus distributions for the with-vs-without predictions do mostly overlap each other. Still, lived saved in expectation is a big deal!
So I'm not actually disagreeing with anything here.
> of course, you can still buy all the Gamestop stock you want
this gave me an idea: can't we hijack stocks to turn them into predictions ?
for each bet create two corporations, for example the Trump Will Get Impeached Corporation and the Trump Will Not Get Impeached Corporation. they'd both be contractually obligated to, if they lose give all of their wealth to the other one, and if they win, buy the shares back from all shareholders at the new, increased value.
This term "lighter side" I do not think it means what you think it means...
Ugh. What are covid "deaths"? The PCR test isn't standardized, and by varying the number of cycles, you can vary the covid death rate. What if someone who is in control of the number of cycles also bets on the number of deaths?
BTW, PCR is why the death prediction keeps rising -- because PCR counts people who were merely exposed but destroyed the virus, people who were exposed and had no or very mild symptoms, people who were sick but aren't anymore, and finally the ones who are showing symptoms. Eventually everyone will fit into one of those categories and so every death will be a covid death.
I'd like to offer a few brief counter-arguments:
1. The persistent popularity of casinos, lotteries, and other forms of gambling indicate that yes, people are more than happy to throw their money away on bad odds. No matter how bad someone is at predicting, they probably still aren't bad enough to drop their odds lower than that of buying a lottery ticket (i.e., they aren't being any less rational). And yet lottery ticket buyers still persist. This call into question the ability of prediction markets to "select" for high-knowledge participants over the long run
2. The reason this works, is because prediction markets (and all other forms of gambling) aren't closed systems. Sure, someone who makes all of their money Blackjack, and funnels 100% of their earning back into more Blackjack, will eventually sink-or-swim based on their Blackjack playing skills. But in practice, most people spend a small portion of their earnings on Blackjack, and have outside sources of income, such that losing in Blackjack is only a minor deterrence to continuing play. So long as people have outside jobs, "dumb-money" will continually pour into prediction markets with little consequence.
3. More abstractly: Due to the diminishing marginal utility of money, "dollars do not equal knowledge units". A wealthy person would gladly gamble away $10,000 on a whim, while a poor person would only bet this if he know the outcome with near-certainty. This means that the degree of knowledge a person has in regards to an event occurring, *cannot be directly inferred by the amount they are willing to bet*. The implications of this are significant. Because for prediction markets, pooling dollars bet ≠ the aggregated knowledge of an event, so you cannot even say with certainty in a market with (for example) 60% yes and 40% no, "the bettors in this market think the event is more likely to occur than not". It's *evidence* for it, sure, but other forms of polling may find different outcomes in the same group of people, if they adjust for levels of certainty using different measures.
4. The common sentiment in the rationalist community seems to be something like "betting markets are beautiful in theory, but flawed in practice". I take the inverse position: Betting markets are flawed in theory, but have practical application so long as you understand their theoretical limitations (e.g., they are poll-like things that basically conduct themselves, for free). It's good that we have these prediction markets, because having information is better than not having information. But that's as far as my hope goes: If there was some way to bet against prediction markets eclipsing the usefulness of traditional polling in the coming years, I would.
Have you seen Zvi's weekly series of Covid posts? (Most recent post: https://www.lesswrong.com/posts/Rhy5g75NdRKdw9ibB/covid-1-28-muddling-through )
I have the impression that he would give quite different predictions than the average / median on Metaculus. If only there were more than fake internet points up for grabs there, he might have an incentive to participate in those predictions.
I go through every question in Metaculus and if I don't have any prediction of my own, I jsut pick what the crowd picks. I want to see how well I perform over time and get those super nifty Tachyons. I don't actually care about the future...just winning.
I did PredictIt this election cycle and made around $200 on a $500 investment. Never again. What a horrible experience. First its pure tribalism distilled and slammed one shot after the next. Second, the rules are super specific (which they should be and sometimes aren't in Metaculus--though discussion usually helps) and people make money off of gaming the rules of a question (This is the best example: https://www.predictit.org/markets/detail/4353). third, it's far more important to understand trading (hedging, arbitrage, etc.) than making correct guesses. In the market I listed above, somehow, they pumped the price of Republicans winning the senate after the Georgia election because of the rules and a ton of people made money by choosing the wrong answer. I think Metaculus avoids this problem beautifully.
I support Metaculus Mondays. There's even some questions I'm interested in, like, "When will Epistemic Humility become a common phrase?" Support them on Patreon: https://www.patreon.com/metaculus/posts
I started to type out a snarky comment about how 6% of PredictIt users don't even realize AOC won't be old enough in 2024, and then thought I'd better google it. Turns out she'll turn 35 three weeks before the election, which makes her several years older than I thought she was. Interesting.
A couple takeaways:
(1)
Using prediction markets to inform governmental policy seems like it could create some really perverse incentives. For instance, if the question is "What will be the employment rate in we (rasie/don't raise) the minimum wage to $15?"
A government will raise the minimum wage if the predicted decrease in employment isn't too great. Doesn't this create strong incentives for business owners to enter the market and enter overly dramatic predictions of employment decreases in order to manipulate policy makers into not implementing a policy that goes against their interests?
(2)
Curious about how these markets work with actual money. For binary predictions it seems simple enough. I'm less sure for continous predictions. Is payout proportional to how close the predictor's value was to the actual value? How does the money placed get distributed?
(3)
What is the consensus view on cryptocurrency-based / decentralized prediction markets? I see a few built ontop of etherium (https://defiprime.com/prediction-markets) but I haven't heard nearly as much about them as I've heard about Metaculus. Do they lack the technical expertise and polish that Metaculus offers? Or is it just that the inherent exoticness of cryptocurrencies makes them less accessible than mainstream alternatives?
This seems like the perfect use case for crypto: A clear need to introduce actual money into prediction markets + the technology to skirt government regulation
I have a question that I think is really obvious, but I don't see it being addressed. I wonder if I'm missing something.
If one were to use prediction markets in the "creepy deep magic way", i.e. asking people what the consequences of political decisions would be - wouldn't people want to influence the outcome to support their political leanings?
I understand that this would come at some cost for them, and this surely is a correcting factor, but I would guess that political motivations might still sometimes drastically alter prediction poll results.
Sorry for any grammatical mistakes, this is not my native tongue.