144 Comments

Typo thread

"If Russia invades Ukraine, this person will win +58 points; if it doesn’t, they will win +32 points. Why does Manifold allow this?" - This should say "Metaculus".

Expand full comment

You're right, thank you.

Expand full comment

Recent prediction market convert and they do things I really, really like. They teach people to forecast, they use the wisdom of crowds super efficiently, they tie consequences to being correct or incorrect, and they incentivize participation. It’s like turning in a search light and aiming it into the future.

Question: Does anyone have a long term strategy on how to make this stick to decision makers? That’s my main point of curiosity. Is the hope that they just get so efficient they can’t be ignored? I’m sure this can make money but I had hoped something like this (I have my own weird scheme I’m super into just like I’m sure everyone here does) could be a civilization’s sense organ.

Right now it seems like the plan is to make really good eyeballs and figure out how to hook them up to the brain later? Is that right? Genuine curiosity.

Expand full comment

Mostly agree with this; right now the problem is scaling the technology and making sure it works well, after that decision-makers will either notice on their own or we can pressure them into it. Though if you're not familiar with Philip Tetlock's work with IARPA that would be a good place to start as an example of government/forecasting interfaces.

Expand full comment

Thanks Scott. Buying one of his books now. Been reading the Hobbit to my son but at two months, I bet he doesn’t notice the substitution.

Expand full comment

We have talked to a few managers who want a private version of Manifold for internal company use. (And of course, we use it to guide our own decisions eg https://manifold.markets/Austin/what-new-manifold-features-would-be ) The discussions are still really early, but I think this is one way Manifold can have a huge impact: by pushing forward this use of futarchy and decision markets in real-world use cases!

If a private Manifold instance sounds like something you'd want for your team -- get in contact at akrolsmir@gmail.com!

Expand full comment

Know this might be a bit out of left field but I’d pay you a monthly subscription fee for a browser extension that told me whether or not a given website/Twitter account/etc was reliable or not on average if it had been ranked by Manifold.

Expand full comment

Hm, that's interesting! I think a browser extension for something like Trustpilot (https://www.trustpilot.com/) might be more appropriate for the generic website use case.

For a specific Twitter user, if they link their Twitter account in their Manifold bio, it might be feasible to have an extension bring up their trading history, so you can see if they're as good at punditry as they claim. We're probably pretty far from being able to do this at the moment, but thanks for your feedback -- always helpful to hear what users are so interested in that they'd pay!

Expand full comment

What about Kalshi?

Expand full comment

Kalshi went through the "near-impossible regulatory hurdles" and allows real money for US citizens. As far as I know, it's the only one to do so.

Expand full comment

Predictit does too.

Expand full comment

Yeah still a bit confused by Scott's persistent negative stance towards the first company to be able to create legalized prediction markets ever...

Expand full comment

Does he have a negative stance toward it? I recall him being quite excited about it.

Expand full comment

They're real money, which makes them irrelevant to this discussion of play money and reputation systems.

Expand full comment

they're fantastic if you want to predict untinteresting things that will resolve in a week, like the weather in new york

Expand full comment

Kalshi is also limited since, because of the regulatory hurdles, every question is a binary options contract offered by Kalshi. No one will ever be able to post their own questions on Kalshi without serious changes in the CFTC.

Expand full comment

on the bright side, it means no "will someone try to assassinate the president" questions

Expand full comment

Ooooh I like the per-market loan system. They should at least try it and see what happens!

Expand full comment

Yeah, I like the idea a lot too, as a way to lower the barrier to trading and encouraging activity, without causing significant inflation or giving an edge to people to who trade on everything (ala Metaculus).

As a bonus, it basically allows for 10 "free" comments!

Expand full comment

Hey there, I work for Metaculus and wanted to share my perspective on Scott's points about reputation and about how Metaculus incentivizes predictions. Tournaments have a different scoring mechanism than the rest of the platform, because there are cash prizes at stake. If someone is highly-ranked on a tournament leaderboard and wins prize money, it's because they outperformed other forecasters and contributed a lot of information with their forecasts.

Expand full comment

Do they get some other form of reputation point for winning real money or otherwise doing well?

Expand full comment

As we build out more features on the platform, we’re planning to expand the reputation system so that performance on tournaments becomes part of users’ track record pages.

Expand full comment

I was thinking some about the issue with betting on conditional markets that may well never trigger, particularly in the context of Scott's which book should I review markets, or similar. And I think at least a partial solution is the following:

If there are multiple conditional markets whose conditions for triggering are mutually exclusive, you should be allowed to use the same dollar to bet in as many of those markets as you choose.

Expand full comment

So, leverage?

Expand full comment

Absolute accuracy is usually represented by brier scores, and brier scores suck, because they don't use logarithms, so they can't appreciate the huge difference between 1% and 0.01%. I have an idea to construct a better formula. (https://en.wikipedia.org/wiki/Brier_score#Definition) Instead of just squaring (Ft-Ot), apply the transformation g(x) = -lg(1-x), where x=abs(Ft-Ot) So your penalty is ~zero if your probability estimate is close to correct, but your penalty goes to infinity as your confidence in the wrong outcome goes to 1.

Predictit is very negative-sum due to fees, and it works fine.

Positive sum sucks because it incentivizes spamming the most predictions without regard for accuracy.

Positive sum combined with bad formulas (see above) is what lets people get away with predicting 99% on questions that should be 96%. On a real money negative-sum market, you're not going to make much return by doing that. Predictit's incentives are such that 99.9% certainties often trade at 97 cents. I think negative-sum tends to under-estimate probabilities of 99% events, while positive-sum can over-estimate probabilites, but doesn't have to, if they fix the formulae (see above).

Expand full comment

I have never seen a prediction market even try to distinguish between 0.1% and 0.01%. I agree this would be a useful ability to cultivate.

Expand full comment

Betfair Exchange goes up to 1000, ie predicting 0.1% probability. Weirdly, it only goes down to 1.01 (99% probability). This is obviously meaningless when there are only two possible outcomes, but when there are more than two, then it means that very unlikely possibilities can be meaningfully predicted - while if one option has captured 99% of the probability space, then the market is effectively over anyway.

Expand full comment

I mean you could use any proper scoring rule, right? Isn't what you're suggesting just the log score? Or am I missing something?

Expand full comment

Proper just means the scoring rule is optimized by predicting the true probability of the event. Proper doesn't say anything about fairly penalizing incorrect probabilities.

Here are three scenarios:

Adam predicts 99.9% on 10 events, and 9 of them happen.

Bob predicts 50% on 10 events, and 5 of them happen

Carl predicts 90% on 10 events, and 8 of them happen

We can probably agree that Adam is the worst predictor in this bunch, then Carl, then Bob.

But according to brier scores, Adam is the best.

If you divide brier score by reference brier score, the ranking is correct but the distance between Adam and Carl is smaller than the distance between Carl and Bob. This still feels wrong.

If you use logarithmic error term and divide the score by the reference logarithmic score, then you get the correct ranking plus correct distances. Bob gets 1, Carl gets 1.08, and Adam gets 2.12

I made a spreadsheet to demonstrate all these examples:

https://docs.google.com/spreadsheets/d/1RYxqWp_hJooxYMoDW9PwfREGEuS-V418nb_o4hygGPo/edit?usp=sharing

Expand full comment

Switching to log score won't help here, because people can just learn the rule "never be more confident than 99%, no matter what". (The value 99% may need to be smaller, like 95%, depending on the scoring rule). Then Adam will be back to being the best predictor.

More explicitly, the problem is that by switching to the log score, you're successfully penalizing 99.9999% wrong predictions... but you're NOT successfully REWARDING correct 99.9999% predictions. This is for a fundamental reason: it is impossible to reward 99.9999% well enough so that people would actually care to make them (instead of just making 99% predictions), because the difference between 99.9999% and 99% only shows up ~1% of the time. So what you end up with is people rounding off their predictions to 99% always, or maybe to 95%, and you've gained nothing.

(Techincally, of course, both Brier and log-score are proper, so they technically give the correct incentive even if your confidence is 99.9999%, but the issue is that you lose very little score if you do the "wrong" thing and round down your predictions to 99% or 95% instead.)

By the way, there exist proper scoring rules that are even more aggressive than log-score, penalizing overconfidence even more. For example, a penalty of sqrt(1/x-1) if you've assigned probability x to the correct outcome is a proper scoring rule, and the penalty goes to infinity as x->0 faster than it does for the log-score.

As a final aside, your formula isn't the usual log-score, so are you even sure that it is proper? Did you check?

Expand full comment

I don't know the mathematical background beyond what I just read on wikipedia so I'm just reinventing it from scratch and I wouldn't know how to check if the rule is strictly proper.

Expand full comment

> We can probably agree that Adam is the worst predictor in this bunch, then Carl, then Bob.

Can't say I agree with this. My gut reaction is that I'd rather have Carl as my forecaster, and indeed your spreadsheet has Carl beating both Adam and Bob on log score by a little over 0.2 bits per prediction.

The impression I get from your score calculation is that you're trying to reward good calibration, but I find it more useful to think of calibration as a hack to extract probabilities from subjective belief, an instrumental goal in service of accuracy. Bob's perfect calibration gives me no information to adjust my prior away from 50%. "If you lack the ability to distinguish truth from falsehood, you can achieve perfect calibration by confessing your ignorance; but confessing ignorance will not, of itself, distinguish truth from falsehood." (https://www.lesswrong.com/posts/afmj8TKAqH6F2QMfZ/a-technical-explanation-of-technical-explanation)

Expand full comment

50% provides information. There's no reason to expect a randomly selected proposition to be anywhere near 50%. But I get you that making ten 90% predictions provides more bits of information than making ten 50% predictions, and that should be rewarded somehow.

But in order to condense scores into a 1-dimensional spectrum, this reward has to trade off against the reward for correct calibration, and I don't know what's the right ratio for that tradeoff.

Expand full comment

I'm not sure how you'd measure "bits of information" in this context, but ten accurate predictions of the form "with p~90, company X will not go bankrupt this year" is less informative than ten accurate predictions of the form, "with p~50, company Y will see its stock value increase tenfold this year".

Expand full comment

So whether the predictor is really sticking his neck out is a function of the content of the predictions, and can't be scored by a formula that just compares predicted probabilities with binary outcomes. Any absolute-accuracy formula is going to reward people equally for:

* a correct 90% prediction that the sun will rise tomorrow

* a correct 90% prediction that aliens will blow up major cities tomorrow

Seems like absolute accuracy just sucks and can't be fixed.

Market-based relative accuracy could pay the former 1.000001x his wager and pay the latter 1000000x his wager.

Expand full comment

This is log loss/log likelihood/cross-entropy commonly used in machine learning.

Expand full comment

In predictit you generally won't see good arbitrage above ~94% certainty because they charge something like 6% when your withdraw winnings, so there's no gains to be made up there.

Expand full comment

The withdrawal fee is 5%, but you can be patient and roll over your money many many times on 97% markets that should be 99.9%, before you withdraw it.

Expand full comment

Now that regulators have given the green light to some real-money prediction markets in the U.S., do we think they'll start to incorporate that into policy decisions? I feel like a fully-regulated prediction market is the only sustainable way to create real, skin-in-the-game based information signals.

Expand full comment

You also need to subsidize them and regulate them heavily enough to actual provide useful signals. Prediction markets aren't magic.

Expand full comment

Huh? Sportsbooks are unsubsidized and essentially unregulated and they work great.

Expand full comment

Those are pure gambling, not prediction engines. And in any case, there are still rules about athletes not betting on themselves or taking bribes to throw a game, etc.

Expand full comment

"Those are pure gambling, not prediction engines"

Then why are they the most accurate prediction markets in the world?

"there are still rules about athletes not betting on themselves or taking bribes to throw a game, etc."

It seems like those rules actually make the predictions LESS accurate?

Expand full comment

Alternatively/in addition to initial loans, is there a reason why these markets don't do some kind of dynamic margin depending on the implied probabilities of the market?

If I've bought 100 contracts at $0.90, then with ~90% probability I expect a $100 payout and a $10 profit. Given that my odds are so high, the market provider probably doesn't need me to put up $90 in collateral, right? At 'reasonable' 5x leverage I'd put up ~$20, at horrifying-crypto-exchange 50x leverage I'd put up $2.

I can see one issue, which is that if collateral requirements for long positions go down as the price goes up, then this makes it easier to go *even longer*; if the 'correct' price is lower, then the increased (or at least not-decreased) collateral requirements for short positions make it harder for the market to correct.

Expand full comment

One reason why your book review markets are showing probabilities significantly higher than 44% for getting 125 likes is that Yes bettors are incentivized to add more likes.

Anyone who bet Yes would very likely add a like to such a post, and maybe get friends and family to as well (or create fake accounts to add likes, if they really want to win play-money).

This is the problem with using metrics — they often stop being useful when you condition action on them.

One solution is to loop in human judgment: Will Scott believe that this book review was a relative success 1 month after the article was published?

Or, on a scale from 0 to 100, how valuable will Scott judge this book review to be one month after posting?

Or, if you want to be really free-form, you can use our new free-response markets: What will the reader reaction be to this book review? Users would then submit text answers and bet on which answer best describes how the book review was received, and then Scott would choose one winner (or multiple winners). This kind of market could give qualitative descriptions that binary and scalar markets cannot.

Expand full comment

This is exactly what I came to the comments section to say; I think it's one of my core objections to prediction markets. I should go and bet a *ton* of money on each of Scott's future book reviews getting 125+ Likes; I'm sure I can convince 125+ subscribing SSC commenters to like those future posts, especially if I tell them to do the same thing and get easy money (albeit not very much easy money by the time 125+ people do this). Now that I know Likes on ACX psots are a thing people make prediction markets about, I think I get even less meaning out of the Like count on a given post than I already did.

I'm not convinced this is evadable even for high-profile predictions like whether or not Starlink gets set up by X date or whatever. Elon Musk is definitely not above making a ton of money in a prediction market by delaying the release of Starlink until just after a date in order to make a ton of prediction market money.

Expand full comment

How are you going to convince us to 'like' the post, exactly?

Expand full comment

Scott is a Trustworthy Person, so I don't think he would game the system. I don't think he would actually bet on a prediction about himself or try to gain (reputation/play money) from it. That said, Scott is pretty rare, and seems to care more about prediction markets working than about gaining within such a market. I don't see how such a system scales, especially with real money on the line. Scott could easily game such a system, and it's obvious how he could do so.

I would be interested in hearing how user-generated questions with user-defined payouts would not be overwhelmingly scams if a prediction market became actually popular. This is significantly more concerning if/when real money were involved. There's a very good reason that the US doesn't want to allow these sites to use real money on user-generated questions.

Expand full comment

This only very remotely on topic, but one of the reasons I did not participate in any of the book review markets is that I think "Number of likes" is not a very good mesure of readers satisfaction. I'd say that number of comments is a much better proxy.

Expand full comment

Yes, I noticed that the Sadly Porn review had relatively few likes, but provoked a lot of discussion, and got many subreddit upvotes. I still don't really understand the dynamics here and one day I might try to track different posts' Substack-likes-to-Reddit-upvotes ratio and see if any patterns fall out.

Expand full comment

Might be relevant to note that I was not aware their was an ability to "like" something in substack until today.

Expand full comment

Ditto, and I'm still not clear on the point. Other social media uses likes as a way to demonstrate engagement to the algorithm so the algorithm promotes the content, but to my understanding substack doesn't do that. Is it purely as information to the author? If so that could use clarification.

Expand full comment

I guess it works as an appreciation gesture or some kind of favorite system.

Scott probably has a lot of people like me who don't use substack for anything but ACX and therefore miss the 'social network' features. Reddit likes might be a better indicator, but they're still very dependent on unrelated factors like posting time.

Expand full comment

It actually never occurred to me that my likes meant anything – if I like a popular creator's work on, say, YouTube, it's one of a hundred thousand others and has zero impact. I only use them to train the recommendation algorithm.

Now that I hear that over half of your (book review) posts get 125 likes or less, and also that it's a metric you care about, I'll definitely start liking things I enjoy more.

Expand full comment

Doesn't Substack give you some engagement metric (unique visits, visit duration etc.) that you can use?

Expand full comment

I think that a "Like" to most people suggests a positive, happy emotional reaction, which I can't imagine all that many readers having to the "Sadly, Porn" review. It's content that a lot of readers had an emotional reaction to and wanted to engage with, but it's a reaction and a type of engagement that "Like" just doesn't really describe properly.

I think Facebook used to have a similar dilemma... if someone posts to say "My personfriend got fired from their job and my dog has covids; life sucks", it feels pretty inappropriate to say that you "Like" that post. Which is probably why Facebook has transitioned to a wider range of reaction emoji now.

I haven't really used Reddit all that much, but I suspect that an "upvote" on Reddit has fewer emotional connotations for people and is more of a neutral "I appreciated this post and believe more people should see it" signal.

Expand full comment

I mean you did force substack to turn off comment likes here, a move which was generally well-recieved by the commentariat here. I think we just dislike likes.

Expand full comment

Was it well-received though? I like likes.

Expand full comment

Even if I like a post, It's kind of a crapshoot whether I'll remember to click the heart, and this might be biased by whether I'm distracted by the comments or reading links. Likes+Comments is probably a better metric than Likes alone.

Expand full comment

Ironically, I just found out the other day that getting a very high comment-to-like ratio on a twitter post is called "getting ratioed" and is apparently considered a very negative experience.

Expand full comment

Likes are a more direct boost to our ego I guess. But ACX has not exactly the same readership as Twitter...

Expand full comment

Maybe it's a "more heat than light" phenomenon, and controversial doesn't necessarily mean better?

A lot of Scott's more controversial blog posts have also been among his most liked and more famous, but the connection is tenuous.

Expand full comment

Typo: If Russia invades Ukraine, this person will win +58 points; if it doesn’t, they will win +32 points. Why does *Manifold* --> *Metaculus* allow this? They want to incentivize people to forecast.

Expand full comment

We are also still thinking about how to get better predictions for long term markets, where you note that incentives are not-so-good, like for Dwayne Johnson's presidential bid.

We talked to Robin Hanson today, and he suggested creating 3 parallel currencies which are used for short term (<1 month), medium term (1 month - 1 year) and long term (1 year+) questions. He says the shorter term currencies would be able to trade in the longer term markets, but not vice versa.

I quite like this solution and think it would work. It's another example of a zero-sum reputation solution.

Expand full comment

I sorta think that the loan thing solves for the long-term bets too-if I can bet without tying up any money, I don't need other currencies, as long as I'm willing to bet small.

Expand full comment

Yeah, the loan thing is a good idea as well!

Expand full comment

Real-money markets have the same issues with long-term markets. Professional bettors would rather bet on a short term market than lock their money up for any extended period.

The only time you get lots of money in a long-term market is when there is an option whose probability is rising fast. For instance, anyone who predicted that Obama would win the 2008 election prior to the 2004 Convention speech would make a huge profit, as the first probability surge came from that speech.

The only alternative I could think of would be paying interest to locked-up bets, but you'd either have to pay out at the end (which still locks the money up, but does increase the ROI), or you pay out interest on an unsettled prediction (based on the current market value of the prediction) and that incentivises things like "I will flip a coin in 2032, will it come up heads?" as that will stay at 50-50 until resolved and you can just grab the interest.

Expand full comment

One thing that helps with real-money markets is the ability to have "trading bets", where you can cash out of the market at the current price. You get people effectively betting on /predicting the future market price rather than the actual result.

A typical example is that Presidential prediction markets are affected strongly on price by the results of primaries, so if there isn't a market on a specific primary, you can predict the primary by betting on the overall Presidential just before the primary and then take your profit after the winner's prices rise. But you can only do that if you can take two bets, on both sides of the question, and resolve those into a profit and get you out of the market.

Expand full comment

One fairly obvious wrinkle to add to your loan strategy (which I love!) is that you can't transfer it without first paying off the loan. So if you get a M$10 loan and take a position, you can only make money by selling it for more than M$10. There would be some margin to be had there for folks who trolled around looking for questions and taking mispriced positions, but that would be highly useful! Also, you could fairly easily incorporate that into the ranking: A given user could have at least 3 distinct values by which they could be sorted in a leaderboard: current balance, current gain (abs | %age) over money put in, and a weird confidence interval-looking thing that showed how much they currently owe on loans and the value of their positions.

It's probably necessary if you have this option to make every user pay at least a little to get in-otherwise someone could make a bunch of sockpuppets, each of which would make a large number of stupid bets against their main, making it rich.

Expand full comment

Yes, I like your analysis of the loans. We'd be somewhat concerned about people printing money with fake accounts, although they can kind of do that today by creating many new accounts that start with M$ 1000.

But overall it's an intriguing idea, and the incentives are pretty good since you can still lose money eventually if you bet the wrong way. Maybe Scott is onto something here.

Expand full comment

I'd be tempted to get rid of the starting money (or at least tie it to something expensive-ish to make, like a CC, phone number, etc.), and then gate the loans with an actual small payment.

Expand full comment

Regarding the leaderboard, I think it's important to remember that assessing a forecaster's performance based on a single number has the same problems as assessing an investor's performance based only on the amount of money made. You fall victim to a number of issues, like "did they make all of their money on a single high-payoff prediction that was mostly luck?" or "Were they good only for a brief period of time when there were markets on a specific political event?" or "are they a very high variance forecaster and they just happen to be on a lucky streak?"

You need more sophisticated analysis to answer how good a forecaster is: Things like a sharpe ratio of their profits, or a graph of their winnings over time to show when they were active or not would be a start.

For less formal analysis, Kaggle's rating system (https://www.kaggle.com/progression) might provide a good starting point

Expand full comment

One way we disaggregate performance is by category. You can see your performance for different subsets of markets by topic. E.g. the leaderboards for just the ACX 2022 prediction markets: https://manifold.markets/fold/acx-predictions-for-2022

Expand full comment

I used to watch a lot of poker, and it was frustrating to watch some moderate-skill player go all-in early in a game and luck into a giant stack of chips. Once they had a chip advantage, they could lean on other players harder and push them out of the game (or, if the advantage was high enough, force other players to go all-in on near 50-50 odds multiple times).

Essentially, I am agreeing with you that you need more information than a leaderboard to determine actual skill.

Expand full comment

> they solve this by actually reputationalizing play money profits, which works

It partially works. The problem is that profits are also a function of the money you have. Suppose I have $1 and I know for certain that a market that currently has a 1% odd will actually come true. At best, I can end up with ~$100. If I have $1000 to start with, maybe there isn't enough liquidity for me to wind up with $100,000, but maybe there's enough liquidity for me to wind up with $3000.

Buying more money allows you to multiply any winnings, so it's not a good judge of how good of a predictor you are. I think a better judge would be to normalize your profit by the amount you've bought, but that unfortunately takes away Manifold's business model.

Expand full comment

Yes -- Scott called out Robert McIntyre as #1 on the leaderboard, and (I hope he'd be okay with me saying that) he's purchased M$ before. This does make it easier to post higher total profits on the leaderboard, assuming he's 51% right or better.

This is something that's actually quite tricky to get right, and we'd welcome thoughts that others have on how to set up leaderboards that are meaningful and accurate. Specifically, we're hoping to run a prediction tournament for cash prizes sometime in the future, and we'd want to know how to fairly set that up (eg limit everyone to the same amount of initial buy-in?)

Expand full comment

Do purchases made decrease the total profit number?

Expand full comment

Nope; though they are counted against the personal UI where we show your % profit. E.g. If you'd made M$100 on your initial M$1000, you'd see "+10%"; then when you buy another M$1000, you see "+5%" instead.

(Everything ofc is subject to change -- let us know what would be best for your use case!)

Expand full comment

There could be a 2nd score of how much profit divided by the amount bet. By having multiple scores, a website can incentivize multiple behaviours.

Expand full comment

On the use of positive sum markets to incentivize voting -- there's probably some ideal tradeoff point between number of bids people make vs the accuracy of each guess (i.e. where they stand on the Susan-to-Randy spectrum) which maximizes the amount of real information entering the market. I don't know how Metaculus calculates payouts, but there's presumably also some parameter (or set of parameters) controlling how positive-sum the markets tend to be. This obviously gives Metaculus the ability to tune how generous payouts are in order to optimize user behavior (assuming that people, on average, resemble rational actors enough to vote more readily when payouts are more generous and vice versa.) Of note, they *wouldn't* have this ability if Metaculus bids were denominated in dollars or whatever, since then positive-sum markets simply drive them broke. So perhaps there is a way in which play money markets can be epistemically superior after all.

Of course, this same flexibilty could be joined to the benefits of a real-money market by having bids be made in play money which can be exchanged for real money afterwards, at some rate related to the generosity of the payout system in a way that keeps the market operators financially afloat.

Expand full comment

this is exactly what metaculus does. the ratio between "absolute points" to "relative points" appreaches 0 as the number of forecasters increases.

Expand full comment

It's worth pointing out that real-money markets are negative-sum (the market takes a vig).

Also, real-money markets need to get a certain amount of attention (a few thousand dollars) before they are making any sort of useful prediction, which heavily restricts the non-sports markets that are created.

There is far more money on a first round tennis match in an obscure tournament than on any but the biggest non-sporting markets.

This is partly because people like getting results quickly, and sports generate large numbers of results in a short period of time, where most non-sporting predictions require locking up money for long periods of time. In many cases, you have to get in relatively early, as a great many predictions become near-certainties (99-1 propositions) for quite a long time before they are finally resolved.

If you have to lock up money for a long time if you want to be making a meaningful prediction, then the ROI is much worse than sports - but the predictive power of the market would be much more valuable. I think this is a hard problem to solve; if you could earn a lot more money by studying the 30th-50th ranked tennis players and predicting their early round results when they play each other, then many smart superpredictors would be incentivised to do that rather than providing information useful to society.

Expand full comment

I have a few observations about Manifold, which you may need to take with a grain of salt because I did manage to lose quite a bit of play money over the weekend.

Assuming I haven't misunderstood their Technical Guide (https://manifoldmarkets.notion.site/Technical-Guide-to-Manifold-Markets-b9b48a09ea1f45b88d991231171730c5), placing bets on Manifold is, in fact, (slightly) negative-sum. Of the bet pool, 4% goes to the market creator as a 'commision' and 1% is "burned" as a 'platform fee.

In addition to not wanting to tie up their money for long periods, another reason for people not to correct markets is elucidated here: https://kevin.zielnicki.com/2022/02/17/manifold/ - essentially, because your payout depends on the state of the market at resolution, not only on the state when you place your bid, you get less expected profit if the market moves in the correct direction as people get more information near when trading closes.

Expand full comment

You're right on both counts! The platform fee is our attempt at fighting inflation (from users joining or buying M$), and also something we're testing for eventual hypothetical crypto use case.

We also take Kevin's criticisms fairly seriously, and are considering alternative market-making mechanisms such as LMSR; see the discussion here: https://manifold.markets/kjz/will-manifolds-developers-agree-wit

Expand full comment

Note: 1% and 4% fees are on trader profits, not the bet pool. It is still technically negative sum though!

Expand full comment

Sports books try to set a "line" so that half the bets are on one side and half on the other. Over time, the line changes to keep things that way, so it is a kind of prediction market. Since the book always takes a cut, the market is negative-sum for the bettors.

Expand full comment

I am very interested to see how positive-sum systems resolve the issue of Randy-strategist bettors because of the implications for what I think is the most interesting model: positive-sum for-profit prediction markets. The idea is simple: by offering a game where the house always loses, you incentivize people to forecast. Your actual business model will be making money off of having more or less some knowledge of the future, as generated by the wisdom of crowds.

Expand full comment

I'd love to hear why the longterm prediction issue you highlight here, which is also pretty apparent to anybody who's used a real money or limited play money prediction market, is solvable.

If a prediction market can't successfully predict if the Rock will be president in two years, how can we expect prediction markets to predict if a new virus will be an issue in two years, if a potential war will be considered a good idea in two years etc.? If the idea is that these markets are accurate due to incentivizing and rewarding correctness, how can real world decision makers lean on prediction markets for their predictive value if nobody bothers to show up due to the inherently low annual return?

I am very worried that people are so fond of a decision making tool that by design only seems capable of extremely short term thinking.

Expand full comment

"Far from being a subsidy - money which it is easy for other people to get - this feels like smart money - money that other people should be scared to bet against. So how does this open the market at all?"

Scott, if you don't think that the market can beat your own prediction, why are you even betting in the first place? What is the point of asking the market for a prediction on your life events if you don't think they can outperform you?

(I agree they likely can't outperform you, but then again, I think that asking the market for predictions about your life events is stupid. You seem to think it is not stupid, so I don't understand your thought process here.)

Expand full comment

I also would like to know Scott's thoughts here. He seems very interested in user-defined questions, which makes sense to me on some level, but obviously is including very personal questions like "will this person get married?" that seem very poorly tuned for this kind of market.

The usefulness of prediction markets seems to only come from questions big enough to generate and utilize different perspectives such that biases are squeezed out in aggregate. If you take away the multiple perspectives (really, how many different perspectives can genuinely exist about whether someone will [personal life choice]?), then you're just having different people bet about the limited and probably known information going into it (i.e., the potential spouse's brother betting on the marriage question).

Expand full comment

For the conditional prediction market on book review likes, it seems a bit like the problem with assassination markets where market participants can affect the outcome themselves: people can increase (but not decrease) the chance of any book passing the like threshold by just liking the post themselves or with sockpuppet accounts - 125 likes is not hard to achieve. That asymmetry seems like it could inflate probabilities a bit.

Or maybe Scott realized this and it's a 5head play to increase likes on his twitter posts.

Expand full comment

Scott, I think there may be a fundamental disconnect in people's (or at least my) understandings of prediction markets, because you didn't mention what I think is the most obvious reason the book review predictions are all positive: people think that you're most likely to review the books that have the highest probability of getting the required number of likes. Thus, they see these not only as predictions, but also as an opportunity to pay you to review books. Honestly, when you first announced the predictions, I thought that was your intention, "get involved in prediction markets, and you might get to pick what book I review next!" Even if that wasn't your intention, the fact that the bet going one way or the other will reward people via a side channel rather than purely with the play money seems like it will always have a distorting effect, and will be unavoidable for prediction markets. It seems like the more that real life decisions are based on the results of prediction markets, the worse this distortion will become.

Expand full comment

Now that Scott has signalled that he cares about, and is paying attention to, the number of likes that his posts get, he'll probably get more likes overall. It never would have occurred to me to press the like button before, but now I'll probably start doing it (but only on the posts that I like more than the median ACX post, which seems to be the optimal strategy)

Expand full comment

I feel I picked up a subtle hint in the last posts, so I'm gonna go for it: What is your Metaculus score?

Expand full comment

>Manifold lets you buy their play money for real money, which in theory would destroy any reputational value. But they solve this by actually reputationalizing play money profits, which works:

Say that you earn 1% returns on your predictions. Your profits would still be 10x higher if you spent 10x more real money, so you had more play money to invest, right?

Meaning, this leaderboard is still pay to win, even though you can't win by being below average - unless I'm missing something?

Expand full comment

> even though you can't win by being below average - unless I'm missing something?

Multiple accounts with contradictory predictions.

Expand full comment

Scott mentions ~200 active players on one of the markets. Creating a handful of extra accounts can probably be a major factor in specific questions.

Expand full comment

Thank you for the article. I prefer how Metaculus awards points for a true and false statement. Part of being successful at predictions is to continue being around to predict, even if you were not on the money.

Expand full comment

Why can't a play money market offer separate blocks of money for each quarter of market resolution times. So the "Dwayne Johnson for President" money would be in an Autumn 2024 block (which all users would get, like with every block) but which could only be spent on questions that resolve in Autumn 2024.

Then no one would worry about tying up this money because it could only be used for things that paid out in Autumn 2024 anyway, so you would just use it whenever a good market came along, in any year, provided it resolved in Autumn 2024.

And ideally you'd be able to sell again (if someone wanted to buy at your offer price) at any time, but this is not entirely necessary.

Expand full comment

A few weeks ago you posted an article about 'why you suck', and while that was obviously hyperbolic I would point to 'Mantic Monday' as a reason I read you less and less. When I used to see an SSC article, I was excited – when I see an ACT article, I think 'I hope it's actually something interesting,' but usually it's something like this.

The blog has largely shifted from articles applying the rationalist perspective to wider issues to articles writen exclusively for the rationalist subculture and its status symbols and obsessions. This is fine! It's your subculture after all and you're on of its most prominent members, there's no reason for you not to write what you care about. But for me, this article really is about its title – the mechanics of using play money to gain reputation within a subculture just isn't particularly interesting.

These Monday posts never teach me anything about Russia or Ukraine. Knowing the aggregate opinion, even if it were accurate, isn't interesting without a discussion of the underlying reasons for the predictions. It's the equivalent of writing 'stocks go down on Russia news,' then a paragraph on which companies went down in the Dow, then one on the Nikkei, then one on the FTSE, etc., etc. Interesting only for people playing that game, and at least they play with real money, albeit usually not their own.

I don't mean this as a criticism, obviously this is interesting to others, but if that previous post was genuinely interested in how your blog is seen to have changed, the proliferation of inside-baseball rationalist articles is the big reason why I check the blog, but skip most content and don't pay to subscribe.

Expand full comment

I agree completely.

Expand full comment

I think of this as something I do in addition to everything else, not replacing it, which makes me less concerned about this. I used to aim for two effortposts a week; now I usually have two effortposts plus a Monday post on something inside basebally.

Expand full comment

I'm thinking conditional prediction markets would probably be improved a lot if you could use the same money to bet on each of the conditions (whenever the conditions are mutually exclusive).

Expand full comment

It's fascinating that your solutions for fixing what you see are errors in implementation that allow the market to be perverted for unrelated individual motives all rely on knowing what the prediction market *should* be predicting. Do you have sufficient faith in the result of fixing these errors that you'll then be able to accept on pure faith in the mechanism the predictions that *can't* be checked (by your instincts)? Because presumably to have any real value the predictions markets have to make predictions which can't be checked. How do you get to a place where you have sufficient faith in predictions that can't be checked that you bet important things -- real money, in large amounts, career and/or life choices? Do you just keep tweaking the system until it gives you results that you like for predictions that *can* be checked?

Expand full comment