I go down to Puerto Rico pretty frequently and have some good connections there. Including in the (small) tech scene. Let me know if I can help Insight Prediction founder.
What cities/parts of cities does that scene typically congregate? I've been interested in the possibility of living in Puerto Rico for a while, but I'm not sure where to scout out
The native Puerto Rican tech scene is in San Juan. Largely near Condado/Miramar/Santurce. There's an alt-scene of expats that tends to congregate in various resorts and exclaves. Those tend to have "centers of gravity" outside of the Caribbean, often New York. They tend to be in the wealthier parts of San Juan like the old city or along the beach. Or in higher end expat communities like Dorado or Fajardo.
ETA: Happy to let you know anything else you want to know about the island. Puerto Rico is big for the Caribbean but is only about the size of Connecticut. So I've got at least a sense of pretty much everywhere.
Does anybody know of a good prediction market looking at mortgage rates?
(edit) Or perhaps has an informed opinion on how high the 30-year fixed rate will go in the USA this year?
I'm planning on a 30-year fixed mortgage. I have an uncertain closing date (9 to 18 months away) and the opportunity to lock a rate in for 9 months, buy a rate-lock extension for one month, and then buy re-locks every 5 days.
This means that I could pay a few thousand dollars to make the rate that I pay on my mortgage be today's rate, for the next 9 months. If the lock-in time expire, and I don't renew the lock, the rate goes to whatever the rate is on that day. For example, if I lock in today, I pay a few thousand and I will have a 4.6% rate on mortgage as long as I close within 270 days. If the lock expires, and the rate at closing time is 5.5%, then my mortgage rate will be 5.5% and I will be out the several thousand dollars.
I can comfortably figure out the right solution given a mortgage rate, but I'm struggling trying to get a probable range of mortgage rates across the next n months. A prediction market seems like it could be a good approach.
I'm not familiar with the American market, but if you're in the UK or most of the EU your key decision will be which provider (that's easy, just use a comparison site/broker), and how long you want to fix it for (variable or 2-5 years).
The difference between the tenors *is* a market derived prediction of rates (approximately; there are also cost of capital/funding issues for banks, a peace of mind premium on the longer fixed tenors, and lender-specific idiosyncratic factors)
I may have done a poor job communicating the question - I just updated it. I'm planning on a 30-year fixed rate mortgage. I just have the option to pay money to lock in the rate today (think a stock option) and that lock is good for 270 days. I can pay extra to extend the length of the time that the rate-lock is good for. I could close on the house anywhere from 6 to 18 months from now. As long as the 30-year mortgage rate is below 6.5% , it doesn't make sense for me to buy the rate lock. If it is likely to be above 6.5%, then the rate lock starts to make more sense.
1) Right, let's zoom out. Prediction markets are, in essence, an attempt to apply lessons learned in financial markets to non-financial things. I think there might be a tendency to view prediction markets at this super-new thing, which in a sense they are, but not in their underlying logic, and if you're discussing anything finance-related it's a bit redundant, because essentially the entirety of finance is predicting future cashflows relative to what other cashflows you could get for that investment.
2.1) The US mortgage market is pretty weird due to the whole Fannie Mae/Freddie Mac thing, but if you're more interested in directional movement, you might want to look at the biggest prediction market of all - US Treasuries. Treasuries track (in a sense, *are*) overall rates well, at least directionally.
2.2) Luckily for you, one of the 'on the run' treasuries (i.e. liquid ones) is a 30 year one. This is currently yielding about 2.52%. The forwards market (i.e., purchasing a thing in the future) has the 9-month forward 30-year treasury yield at about 2.5-2.6% (with much bigger differences at the shorter tenors, which we would broadly expect). The forwards market has much less liquidity than the spot market for treasuries, but it's still way more than you'd get in a prediction market.
2.3) Is the option ('lock') provided by the same institution that provides the overall mortgage? If so, then their own predictions about rates are likely to mean that the option is priced for a bit of profit, but that they're overall likely to be similar.
2.4) Unless you have insider knowledge of major banks' mortgage policies or at the Federal Reserve, don't try to time the market for overall rates, especially for a 30-year mortgage. Predicting overall interest rates (which mortgages closely track) is not a realistic goal.
3) I'm glad I'm not getting a mortgage in the US (my mortgage is 1.11% in the UK, albeit only fixed for 2 years).
That was quite helpful - thanks! Lock is provided by the lender, but people are kind of all over the place on who's offering what (e.g. very large banks like Chase not offering locks at all, while still providing comparable rates). This is a s/medium lender.
Can you expand on why you're glad to have a fix only for 2 years and not for 30? My cursory understanding about the bond market is that rates are artificially depressed because very large players (pension funds, insurers) don't have another choice than buying bonds and decreasing rates further. This, compounded with the fact that most central banks have engaged in asset purchase programs.
However, if you do have a choice, you should shun bonds and borrow as much as you can, fixing for the longest period you're able to.
Could you recommend a book/something that would help me understand this more intuitively? I have good general understanding of finance, but bonds and how they’re priced especially continues to confuse me!
The option is more than likely hedged so says not too uch about the banks expectation (if such a thing even exists - both me my SO are economists with more than a decade of experience in banking each, just took out a huge mortgage and don't believe in that)
Mortgage rates may be the best available prediction for the mean value, but the question OP is asking depends far more on the *variance*, which I find harder to judge. Note that he is asking about an 'option' on the mortgage rates next year not a 'future'
I don't know of a good prediction market, but here are some priors on how the interest rate might move. I pulled the last 20 years of mortgage rate data from FRED. (Link: https://fred.stlouisfed.org/series/MORTGAGE30US) The first thing I checked was the 1-year change, which has a lower quartile of -79 basis points, a median of -10 basis points, and an upper quartile of 23 basis points. (100 basis points = 1% annually on your mortgage.)
But that analysis comes from 20 years where interest rates were mostly falling. There are good reasons to think that has come to an end, with the federal reserve raising rates. So, ignore the median. Instead, look at the size of the range between the lower 25% and upper 75%, which is 102 basis points. So, a rough model would be to model the rate a year out as the current rate, plus or minus 0.51%.
For modelling the rate out N months, I made a graph for this. (Note that I switched here from inter-quartile range to standard deviation.) See here: https://i.imgur.com/qprmgDg.png For example, if you look at the point at the 3 month mark, I subtracted each rate from what it was 3 months prior to that, and then took the standard deviation of those differences. If you want to know what the rate will be in 3 months, you can model it as a normal distribution with a mean of the current rate and a standard deviation of 33 basis points.
> The only difference I can find is that Insight requires two media sources to resolve positively and Metaculus only one - surely people don’t think there’s a 40% chance troops will enter Kyiv but only one source will report it?
The difference also includes the chance that they don't enter Kyiv, but one source reports that they did anyway.
Is there a 40% chance that a media source reports erroneously that Russian troops have entered Kyiv?
That definitely seems possible. But doesn't it also require that the person who posted the question resolve it in that direction? Seems likely that the question was posted in good faith and likely to be resolved at least somewhat accurately?
How pinned down is the definition of "enters Kiev"? If the Russians send troops in to an outer suburb and they get pushed back that same day, but they technically set foot in metropolitan Kiev, did they "enter" the city?
It seems quite possible to me that different people disagree strongly on what exact line to use here.
I looked at the comments section for the Metaculus question on "will Russian soldiers enter Kyiv". It's basically an argument over whether >100 soldiers entered Kyiv between Feb 25 and Feb 27 and whether they counted under the question criteria. Sometimes you can predict the future, sometimes you can't even accurately predict the past : )
Admittedly this doesn't explain the discrepancy between the two sites, which use similar wording. The graph of Insight pricing looks weird to me, it rose to near 100% around Feb 25, then briefly dropped to nine cents on the dollar, bounced back up, then fell down with almost no increase in total wagers. Not sure what causes such swings, but clearly not many people are betting on this question, which will be hard to resolve.
"I’m hoping to get people together next year, come up with a standard question set, and give it to as many platforms (and individuals!) as possible to see what happens."
The best time to do this was at the start of this year. The second best time is now! Why wait until the end of the year? Just that it might be a sort of a Schelling point? You could also pick June as a mid-year thing and coordinate around that.
The market "When will programs write programs for us?" is an interesting example. The headline, and how most of the people that I've talked to interpreted it, implied something very different from how it was graded. How people interpreted it was more along the lines of "when can programs replace a significant fraction of programming work", and we're still far away from that almost a year later.
Of course, the fault there is in the predictors for not reading the full description, but it does suggest that the problem is not in prediction AI progress but in question reading. If your takeaway from it was "predictors are too conservative on AI progress, GAI is coming sooner than people think", you're not taking away the right lesson. I think the right lesson here is "people are bad at predicting when the question title is ambiguous, even if there's a specific description in the body".
It seems interesting to note that the list of examples of tasks the Metaculus question wants a computer to solve are beginner-level learning-to-program homework problems (the first one is literally Hello World), but this fact wasn't advertised IN the question--you have to follow the link and read the list in order to see that.
I suspect it might be possible to meet the question's criteria with no machine learning at all, just by creating a large library of pre-generated solutions to common beginner-level homework assignments that are indexed by a pseudo-natural-language key. (The question allows problems to be human-translated into something friendlier to the computer, as long as the translation is shorter than the original. Also, the question only requires you to succeed on 2 out of 5 trials.) Would depend on exactly how the test questions were generated.
Might even be possible to meet the question's criteria just by having a program that uses Google to search the web for pre-existing solutions. (And maybe some text-replacement rules for converting similar programs with different numbers/text.)
Can I get a sanity check here? I occasionally place bets on Kalshi to see if I do as well in real-life markets as I do on the reputation based ones. Thus far I've done ok but really confused by the action on this question:
My modeling predicts that covid cases under <25,000 by April 1 is highly likely, probably by Friday unless BA2 creates an uptick (we're currently at about 27000 cases and dropping) but the market currently has it at 23%. There's $25000 in open interest so liquidity isn't an issue. Is there something I'm missing?
Delete if this is too self-serving but thought it may slot under prediction market goofiness.
I guess this is actually a good example of a prediction market being right then! Today’s number was a marked increase. I didn’t think the BA2 spike would happen for another week or two tbh. Going to go back to the drawing board on my modeling, I think my error was not tracking NYC closely enough.
>The only difference I can find is that Insight requires two media sources to resolve positively and Metaculus only one - surely people don’t think there’s a 40% chance troops will enter Kyiv but only one source will report it?
Seems more likely the error would be in the opposite direction. If they don't enter the city but at least one news source says they did, especially given existence of propaganda and fog of war
Some more weird stuffs about Aver: their Discord server has 19k+ members yet isn't really... active. That's a lot of big numbers for one platform we never heard of before. I hope someone will prove my suspicion wrong!
I'm excited about all the sketchy-looking crypto prediction platforms - it means more potential arbitrage opportunities.
So how much moderation is enough moderation for a prediction market site?
(edit to add: Manifold Markets doesn't feel like it has enough moderation.)
I would like it if the various glowfic markets were required to be categorized in a way that makes them less visible. Also sports betting. And anything involving root vegetables.
Hopefully some form of community moderation can solve that. The holy grail is "prediction-market based moderation", though you probably need two incommensurate currencies to pull that off. Roughly, if there is >95% certainty that a comment is bad, it is automatically pulled for an hour.
On the other hand, I signed up for Metaculus a few hours ago and submitted the "Turkey 2022 Name Change" question, and it still hasn't been approved. So sometimes less is more?
This is Eric (one of the people running the forecasting contest.) Happy to answer questions about the aggregation method we used! I also want to second Scott's invitation for others to try their hand at aggregating the data, with a few caveats:
- Consider blinding yourself to the particular questions and only looking at the numbers. Otherwise you could end up "cheating" by e.g. choosing an aggregation method that assigns a particularly high probability to the Russia/Ukraine war question (which we now know will resolve positively).
- You can approach this in two ways. First, you could come up with an aggregation method that (like ours) *only* uses Scott's predictions and the contest participants' predictions and not any other data (e.g. "superforecaster or not") about the participants. Or alternatively, you can make your best guess about how to use the extra data to make a better aggregation method. I think both of these are interesting problems!
- I think there's a ton of luck in terms of which aggregation methods will work well, just because most aggregation methods will tend to give similar numbers and the sample of questions just isn't very large. Accordingly, there's a pretty decent chance your aggregation method will do worse than ours even if it's better, or do better than ours even if it's worse. But perhaps we'll take the approaches that do well this year and see how well they do next year!
Geometric mean of the odds is equivalent to taking the arithmetic mean of the log-odds (the result would be specified as log-odds in the second case), which seems like a reasonable justification for using it.
The EA forum link discussing that aggregation method doesn't seem to mention this fact at all, and I can't tell whether everyone thinks it's too obvious to mention, or no one noticed it or what.
Yeah, it's equivalent, and that's how I think about it. I agree that this provides a good justification, as basically this means that you're averaging each forecaster's "Bayesian evidence".
If you folks want to make easy money, bet on the Oscar prediction market. It's ridiculously straightforward. The only public prediction that I made (in 2011) was pretty terrible by my standards and I got only 15 out of 24 categories right (but all the major ones excluding supporting actress) [1, 2]. But on a good year I could do up to 21 out of 24, before getting really bored of it.
It's not seen as a game here, but as a means to find true information about important complex issues. This war and where it will lead is important, complex, and usual sources have strong agendas.
I understand you're upset about it, but to put this into perspective, a lot of the people that are into prediction markets (on this blog, at least) are also into effective altruism. As a rule, there's a lot of money going from those same people toward helping out people directly affected by the war.
Minor case in point: I am betting (on Metaculus), but I've already put ~25% more than a full month's paycheck into helping out. This entire thing has me very upset, but I'm all the more interested in people's predictions, because it helps me figure out things like "how long will this likely last", which in turn lets me partition out my finances so I can still keep donating later on, instead of either burning everything now, or sitting on my money too long to be able to actually help out.
To reiterate, I'm very upset about this war; friends of close friends are directly affected. However, I'm not upset at people betting about it in prediction markets; even if they were just doing it because they want a few dollars for themselves (some might!), they're helping people like me make decisions. (And I imagine they're also helping people in much better positions to help than me to make decisions, too.)
I find the idea of wagering on the Ukrainian war deeply disturbing. I am a bit ashamed of myself, waiting for someone else to mention it first for fear of looking foolish. I suppose my position is not rational.
Thankfully it is not rational. Betting is not scientific but it is excused here in the name of rationality. You are speaking out because you are human. This place is a boys' club with supersized egos.
"You are speaking out because you are human." I agree with this and I don't think anyone should be ashamed of feeling queasy about prediction markets about war. War is a terrible thing and it's normal to want no one to benefit from it on the sidelines, especially if one doesn't see any obvious benefit in what is being done.
Mind you, independent of the emotions involved here and their validity, I do encourage you get to know the people you're judging before you dismiss them outright, and to consider that prediction markets might have benefits you too approve of. I touched on an example a bit further up this page. But that doesn't at all mean you're not allowed to feel queasy.
Still, I need to point out the obvious: "This place is a boys' club with supersized egos." This is neither true nor kind. I don't know if it's necessary, but I don't think so - I don't think this accusation is going to achieve anything.
Hi Neike. It is entirely true and I speak from experience. When I addressed astrological forecasting on tis forum and cited scientific published studies I was attacked badly with few exceptions and by people who know nothing of the subject: and yes, not one of them were women. When running out of arguments I was told that the study cited was not on a serious subject: it was on Amazon review misspellings (chosen btw because of free and massive data availability and for limiting variables) as if not having a "more serious" research subject somehow invalidates the methodology. Betting odds in Las Vegas, however, were treated very respectfully here apparently approved by the boys as an appropriate subject. Making money on the murders of Ukraine is heartless and can only be justified on the altars of reason without compassion even if those betting have good odds as they are "smart" and "capable of cognitive decoupling" as one of the links below explains. Who are these betters and whom do they benefit? Geopolitical experts or military advisors? You don't find placing money on how many cities will fall and how soon somewhat sociopathic? Someone below offered that betting on the Oscars is easier and more profitable for easy money. Obviously, I find your use of your profits admirable but they are more efficient ways to help people in Ukraine if that is someone's sole goal.
"It is entirely true and I speak from experience." It's not good form to judge an entire forum by its most vocal members; I'd ask you please not do that. I barely comment, because I rarely have anything insightful to ask, but I'm definitely part of this community, and have been for years. I've seen many outpourings of support in open threads for people who needed advice and help, and I generally trust that if I need good advice, I can come here and get it.
Is my experience less valid than yours? I'm honestly not trying to accuse you of anything here, you've clearly had a bad experience and I'm sincerely sorry you did, but do I encourage you not to tar the whole community with the same brush. (Astrology is certainly a topic people here have kneejerk reactions about.)
"Obviously, I find your use of your profits admirable but they are more efficient ways to help people in Ukraine if that is someone's sole goal."
Sorry, it looks like I need to clarify what I'm doing: I'm not making profits, I'm only using Metaculus, which doesn't involve money, just internet points. The money is just my paycheck; I have the savings to absorb losing a few of those, so I'm putting them into charities to help out. Metaculus and other prediction markets are just helping me to figure out e.g. whether I should spend another paycheck's worth of money *now* (from my savings), or prefer a more steady trickle of large (up to quarter-paycheck) weekly donations over a long time.
I participate in Metaculus because I want to improve my intuition for certain topics. It's useful to know when one is wrong and when one isn't, and having a website that tells you "nope, you were wrong about this, by this much" or "yep, you got this right!" helps a lot in getting a better feel for what's true and what's not over the long run. I don't "care" about the internet points, but I do care about predicting reality correctly; predicting reality correctly lets me make informed decisions about how to be more helpful.
By making a statement like "Making money on the murders of Ukraine is heartless", you're making a lot of assumptions about the motivations of the people involved. As mentioned, there's a significant overlap with EA here. If you can acknowledge that I'm not heartless (even though I predict in Metaculus), can you not imagine you're also wrong about the average prediction market user's psyche?
> Betting is not scientific but it is excused here in the name of rationality.
Are you familiar with the concept of "prediction markets"?
Winning or losing *one* bet, that is mostly a question of luck. But if two people make hundred bets, and one of them wins 80% of them, and the other loses 80% of them, chances are high that the former simply has better knowledge.
Also, betting is tax on bullshit. Imagine that one person says "I am absolutely sure it will rain tomorrow." And the other person says: "In that case, let's make a bet; I will give you $100 if it rains tomorrow, and you give me $100 if it doesn't. If you are absolutely certain, that means free money for you." Except... suddenly the former person is no longer so sure. (As a rule of thumb, anytime someone is "absolutely sure" but refuses to bet, they are actually not sure.)
> This place is a boys' club with supersized egos.
People with supersized egos who make lot of bets tend to lose their money.
I think the claim is that *wagering* on the war should in some sense be thought of as no more disturbing than *reading news* about the war. Neither of them actually changes the course of the war itself, and both claim to make you more informed about the war. The claim here is that wagering is the way to give people the discipline to read carefully, and understand what their information does and doesn't say.
Wagering on something that you *can* affect the outcome of can lead to all sorts of perverse motivations (which is why athletes aren't allowed to bet on their own sporting events, and which would be even worse if it were war we were talking about). But when it's just distant observers, the claim is that wagering on war is the way to get the best information in one place.
I feel like this perspective is... lacking in practicality. If we could gather a bunch of smart people to altruistically predict how long the war will last, that would be useful for planning/budgeting reasons. But getting people to contribute time and effort out of the goodness of their hearts is very difficult!
Prediction markets (try to) leverage human greed to good ends - to discover true information that would otherwise be difficult to get at, because there's no "force" pushing the right people or resources to working on the problem otherwise. If money can make it happen and it would be good to know, then that seems like a good thing.
Suppose Alice is always talking about how blue-eyed people commit twice as many murders as the population average and blue-eyed Bob objects that this perpetuates harmful stereotypes that blue-eyed people are violent. The decoupling framework tells us that this is because Bob is a low-decoupler and Alice is a high-decoupler. But Bob is always talking about how brown-eyed people are more unemployed than the population average and brown-eyed Alice objects that he is perpetuating harmful stereotypes that brown-eyed people are lazy.
People often demand context where it benefits them and neglect it where it does not, this does not fit within the decoupling model's proposal that people operate on some consistent level of contextualization.
I don’t claim decoupling can explain all behavior, just that it’s one way to understand why people might bet on the outcome of a war and yet not be monsters.
I am still dubious about the idea that "gamblers", aggregated, are better predictors.
Is there really any research backing that proposition?
Is the wisdom of a crowd of gamblers more wise than the wisdom of the crowd.
The theory seems to be that having skin in the game makes one a better predictor, but aren't there 1000s of gamblers in bankruptcy.
Markets are only sort of efficient. And markets are also susceptible to corruption and manipulation. Even with some transparency and the SEC, there is still undetected insider trading and madoff scams. Is it really possible that an online prediction market where some one can bet as igottip234 can be a good thing?
Why does a known gambler like Nate Silver refuse to publicly state whether he bets on his own predictions when it is known that his predictions will affect odds in other markets?
Well, at the very least, we have existence proofs that prediction markets can be wildly off (like anything involving US politics, and particularly, Trump).
In the long term the bad gamblers go broke while the good gamblers take their money, then you have a market where most of the money is being thrown around by people with a proven track record which ought to mean something. Most prediction markets are too new, too small and too restricted for this to work, but it's one of those things that looks appealing in theory.
The actual theory is that lots and lots of opinion tend in aggregation to lead to a pretty good prediction.
The wisdom of the crowds is not at all about gambling.
Gambling as a way to create a crowd of guesses is a slightly different thing.
Markets seems to be a way to create price discover it is similar to the wisdom of the crowds but might be theoretically slightly different. Markets can also lead to speculative bubbles which is different than the wisdom of the crowds. So there is some evidence that gambling introduces a error into what might just be sampling theory.
So there is the question which I will ask again: Is a crowd of gamblers better at predicting than a crowd?
Let's also address the iterative effect of "public" gambling,
Is a private auction (all bids are private) better at prediction than a public continuous auction?
Everybody seems to be saying x% but my gut says y% so I adjust to the crowd and alter my bid as if the crowds guesses are information.
Everyone ends up in Abilene when no one actually wanted to go to Abilene.
One more thing about the "skin the game" theory. A millionaire betting $100 has quite a lot less skin in the game than a pauper betting $100. So the skin in the game theory would have need to account for how much skin are we really talking about.
I really doubt that anybody is coming up with a market that requires the predictor to provide a copy of her tax return with each bet/prediction.
I see that the linked Metaculus problem specifies resolution criteria of passing at least 2 out of 5 trials from a given list. Did anyone actually test Codex on those problems? How many did it pass?
Anyway, assuming that Codex did pass some of those, I think the problem is that people were estimating dates based on their interpretation of the question (when will AI become useful for code generation) rather than the actual resolution criteria (when will AI get good at copypasting leetcode solutions from Stack Overflow).
The market was open for months after Codex came out but I can't tell how much the market moved because the forecast timeline inexplicably stops at December 2019, two years before it was actually resolved. What's going on with that? If nothing moved post-Codex then that seems to firmly establish it as "People were interpreting the question based on the headline, not the much easier resolution criteria".
In what way can OpenAI’s Codex be said to be a program writing programs that a compiler cannot?
Both are programs that you give a high level specification to and it attempts to return a lower level program that accomplishes that spec.
The list from the links is of course much higher level and more freeform of a specification than, say, C code. But is it higher level as compared to its products? Is the difference between a paragraph and Python more than the difference between C and machine code? Good compiled and interpreted code already do read like their spec.
Codex is more useful than prior art at generating/copypasting leetcode solution based on a *natural language* description of the problem. It is still utterly useless, and possibly actively harmful, when it comes to real world programming tasks.
A question is proposed - Will Ukraine successfully drive out Russian forces?
Currently the market might say 0.5% yes.
If territory begins to be taken back by Ukrainian forces that percentage will rise. If we get Russian forces in full retreat it might go to 95% and then as the last Russian troop leaves someone will win.
Where are we judging that the prediction market added any value?
We judge that the prediction market added value if there's some point at which someone sees some minor news story about some new style of Turkish drone the Ukrainian troops are getting, and uses that to raise the market price, and then a few days later the value of the drone becomes more obvious to the major news sources.
If no one has any sources other than the front pages of the major international newspapers, then the market isn't going to do much useful.
That newspaper-only world is not the world in which we live, though, and it hasn't been for more than a decade. So what value do prediction markets add in this world?
Well if prediction markets of war are not games or about money but advice and wisdom offered to high places as I am told ---that is exactly what I would like to know. How would a bunch of people's aggregate bet in SV would be helpful to know how many cities will fall to Putin next month or if he has Parkinsons more than the views of strategical advisors and journalists in the region who quite obviously have no clue.
That’s exactly the point. In the newspaper only world, the market adds nothing. But when there are many sources of information available, the market lets people express the value of the bits of unique information that they’ve been paying attention to and understanding the value of.
> Even in late 2020, just before the question stopped accepting new predictions, the forecast was January 2027. The real answer was six months later, mid-2021, when OpenAI released Codex. I don’t want to update too much on a single data point, but this is quite the data point. If I had to cram this into the narrative of “not systematically underappreciating speed of AI progress”, I would draw on eg this question about fusion, where the resolution criteria (ignition) may have been met by an existing system - tech forecasters tend to underestimate the ability of cool prototypes to fulfill forecasting question criteria without being the One Amazing Breakthrough they’re looking for.
I think it's a much less spectacular data point than it might look, and the fusion example is a good example of why. If you think forecasters know how much more progress needs to be done, and were wrong about how fast that progress will get made, then the forecasters were off by an order of magnitude, which is a lot. But if you think forecasters knew how fast progress would happen, and were wrong about how much more needed to be done, then progress was 6 years closer to Codex than they thought it was, and expressing their error as a ratio instead wouldn't make sense, since they'd be consistently 6 years off according to this model (unless they learn more about how much progress needs to be done before getting Codex), but the ratio of their 6-year-too-late prediction to the actual time until Codex will change, asymptoting to infinity the moment before Codex is completed.
I go down to Puerto Rico pretty frequently and have some good connections there. Including in the (small) tech scene. Let me know if I can help Insight Prediction founder.
What cities/parts of cities does that scene typically congregate? I've been interested in the possibility of living in Puerto Rico for a while, but I'm not sure where to scout out
The native Puerto Rican tech scene is in San Juan. Largely near Condado/Miramar/Santurce. There's an alt-scene of expats that tends to congregate in various resorts and exclaves. Those tend to have "centers of gravity" outside of the Caribbean, often New York. They tend to be in the wealthier parts of San Juan like the old city or along the beach. Or in higher end expat communities like Dorado or Fajardo.
ETA: Happy to let you know anything else you want to know about the island. Puerto Rico is big for the Caribbean but is only about the size of Connecticut. So I've got at least a sense of pretty much everywhere.
Does anybody know of a good prediction market looking at mortgage rates?
(edit) Or perhaps has an informed opinion on how high the 30-year fixed rate will go in the USA this year?
I'm planning on a 30-year fixed mortgage. I have an uncertain closing date (9 to 18 months away) and the opportunity to lock a rate in for 9 months, buy a rate-lock extension for one month, and then buy re-locks every 5 days.
This means that I could pay a few thousand dollars to make the rate that I pay on my mortgage be today's rate, for the next 9 months. If the lock-in time expire, and I don't renew the lock, the rate goes to whatever the rate is on that day. For example, if I lock in today, I pay a few thousand and I will have a 4.6% rate on mortgage as long as I close within 270 days. If the lock expires, and the rate at closing time is 5.5%, then my mortgage rate will be 5.5% and I will be out the several thousand dollars.
I can comfortably figure out the right solution given a mortgage rate, but I'm struggling trying to get a probable range of mortgage rates across the next n months. A prediction market seems like it could be a good approach.
Mortgage rates *are* the prediction market for mortgage rates.
Shall I specify a meta prediction? Or perhaps compare thee to a Summer's day?
I'm not familiar with the American market, but if you're in the UK or most of the EU your key decision will be which provider (that's easy, just use a comparison site/broker), and how long you want to fix it for (variable or 2-5 years).
The difference between the tenors *is* a market derived prediction of rates (approximately; there are also cost of capital/funding issues for banks, a peace of mind premium on the longer fixed tenors, and lender-specific idiosyncratic factors)
I may have done a poor job communicating the question - I just updated it. I'm planning on a 30-year fixed rate mortgage. I just have the option to pay money to lock in the rate today (think a stock option) and that lock is good for 270 days. I can pay extra to extend the length of the time that the rate-lock is good for. I could close on the house anywhere from 6 to 18 months from now. As long as the 30-year mortgage rate is below 6.5% , it doesn't make sense for me to buy the rate lock. If it is likely to be above 6.5%, then the rate lock starts to make more sense.
1) Right, let's zoom out. Prediction markets are, in essence, an attempt to apply lessons learned in financial markets to non-financial things. I think there might be a tendency to view prediction markets at this super-new thing, which in a sense they are, but not in their underlying logic, and if you're discussing anything finance-related it's a bit redundant, because essentially the entirety of finance is predicting future cashflows relative to what other cashflows you could get for that investment.
2.1) The US mortgage market is pretty weird due to the whole Fannie Mae/Freddie Mac thing, but if you're more interested in directional movement, you might want to look at the biggest prediction market of all - US Treasuries. Treasuries track (in a sense, *are*) overall rates well, at least directionally.
2.2) Luckily for you, one of the 'on the run' treasuries (i.e. liquid ones) is a 30 year one. This is currently yielding about 2.52%. The forwards market (i.e., purchasing a thing in the future) has the 9-month forward 30-year treasury yield at about 2.5-2.6% (with much bigger differences at the shorter tenors, which we would broadly expect). The forwards market has much less liquidity than the spot market for treasuries, but it's still way more than you'd get in a prediction market.
2.3) Is the option ('lock') provided by the same institution that provides the overall mortgage? If so, then their own predictions about rates are likely to mean that the option is priced for a bit of profit, but that they're overall likely to be similar.
2.4) Unless you have insider knowledge of major banks' mortgage policies or at the Federal Reserve, don't try to time the market for overall rates, especially for a 30-year mortgage. Predicting overall interest rates (which mortgages closely track) is not a realistic goal.
3) I'm glad I'm not getting a mortgage in the US (my mortgage is 1.11% in the UK, albeit only fixed for 2 years).
That was quite helpful - thanks! Lock is provided by the lender, but people are kind of all over the place on who's offering what (e.g. very large banks like Chase not offering locks at all, while still providing comparable rates). This is a s/medium lender.
Can you expand on why you're glad to have a fix only for 2 years and not for 30? My cursory understanding about the bond market is that rates are artificially depressed because very large players (pension funds, insurers) don't have another choice than buying bonds and decreasing rates further. This, compounded with the fact that most central banks have engaged in asset purchase programs.
However, if you do have a choice, you should shun bonds and borrow as much as you can, fixing for the longest period you're able to.
Enjoying your comments in this thread.
Could you recommend a book/something that would help me understand this more intuitively? I have good general understanding of finance, but bonds and how they’re priced especially continues to confuse me!
The option is more than likely hedged so says not too uch about the banks expectation (if such a thing even exists - both me my SO are economists with more than a decade of experience in banking each, just took out a huge mortgage and don't believe in that)
Mortgage rates may be the best available prediction for the mean value, but the question OP is asking depends far more on the *variance*, which I find harder to judge. Note that he is asking about an 'option' on the mortgage rates next year not a 'future'
I don't know of a good prediction market, but here are some priors on how the interest rate might move. I pulled the last 20 years of mortgage rate data from FRED. (Link: https://fred.stlouisfed.org/series/MORTGAGE30US) The first thing I checked was the 1-year change, which has a lower quartile of -79 basis points, a median of -10 basis points, and an upper quartile of 23 basis points. (100 basis points = 1% annually on your mortgage.)
But that analysis comes from 20 years where interest rates were mostly falling. There are good reasons to think that has come to an end, with the federal reserve raising rates. So, ignore the median. Instead, look at the size of the range between the lower 25% and upper 75%, which is 102 basis points. So, a rough model would be to model the rate a year out as the current rate, plus or minus 0.51%.
For modelling the rate out N months, I made a graph for this. (Note that I switched here from inter-quartile range to standard deviation.) See here: https://i.imgur.com/qprmgDg.png For example, if you look at the point at the 3 month mark, I subtracted each rate from what it was 3 months prior to that, and then took the standard deviation of those differences. If you want to know what the rate will be in 3 months, you can model it as a normal distribution with a mean of the current rate and a standard deviation of 33 basis points.
Alternate ways you could model this: you could look at what experts project. This website polled experts 10 days ago and found most of them thought rates would rise in the near future. https://themortgagereports.com/32667/mortgage-rates-forecast-fha-va-usda-conventional
Thanks a ton for this!
> The only difference I can find is that Insight requires two media sources to resolve positively and Metaculus only one - surely people don’t think there’s a 40% chance troops will enter Kyiv but only one source will report it?
The difference also includes the chance that they don't enter Kyiv, but one source reports that they did anyway.
Is there a 40% chance that a media source reports erroneously that Russian troops have entered Kyiv?
That definitely seems possible. But doesn't it also require that the person who posted the question resolve it in that direction? Seems likely that the question was posted in good faith and likely to be resolved at least somewhat accurately?
Sounds like a 40% good-faith + fake-news metric, no?
How pinned down is the definition of "enters Kiev"? If the Russians send troops in to an outer suburb and they get pushed back that same day, but they technically set foot in metropolitan Kiev, did they "enter" the city?
It seems quite possible to me that different people disagree strongly on what exact line to use here.
I looked at the comments section for the Metaculus question on "will Russian soldiers enter Kyiv". It's basically an argument over whether >100 soldiers entered Kyiv between Feb 25 and Feb 27 and whether they counted under the question criteria. Sometimes you can predict the future, sometimes you can't even accurately predict the past : )
Admittedly this doesn't explain the discrepancy between the two sites, which use similar wording. The graph of Insight pricing looks weird to me, it rose to near 100% around Feb 25, then briefly dropped to nine cents on the dollar, bounced back up, then fell down with almost no increase in total wagers. Not sure what causes such swings, but clearly not many people are betting on this question, which will be hard to resolve.
I prefer the term "postdict" for predicting the past.
"I’m hoping to get people together next year, come up with a standard question set, and give it to as many platforms (and individuals!) as possible to see what happens."
The best time to do this was at the start of this year. The second best time is now! Why wait until the end of the year? Just that it might be a sort of a Schelling point? You could also pick June as a mid-year thing and coordinate around that.
The market "When will programs write programs for us?" is an interesting example. The headline, and how most of the people that I've talked to interpreted it, implied something very different from how it was graded. How people interpreted it was more along the lines of "when can programs replace a significant fraction of programming work", and we're still far away from that almost a year later.
Of course, the fault there is in the predictors for not reading the full description, but it does suggest that the problem is not in prediction AI progress but in question reading. If your takeaway from it was "predictors are too conservative on AI progress, GAI is coming sooner than people think", you're not taking away the right lesson. I think the right lesson here is "people are bad at predicting when the question title is ambiguous, even if there's a specific description in the body".
It seems interesting to note that the list of examples of tasks the Metaculus question wants a computer to solve are beginner-level learning-to-program homework problems (the first one is literally Hello World), but this fact wasn't advertised IN the question--you have to follow the link and read the list in order to see that.
I suspect it might be possible to meet the question's criteria with no machine learning at all, just by creating a large library of pre-generated solutions to common beginner-level homework assignments that are indexed by a pseudo-natural-language key. (The question allows problems to be human-translated into something friendlier to the computer, as long as the translation is shorter than the original. Also, the question only requires you to succeed on 2 out of 5 trials.) Would depend on exactly how the test questions were generated.
Might even be possible to meet the question's criteria just by having a program that uses Google to search the web for pre-existing solutions. (And maybe some text-replacement rules for converting similar programs with different numbers/text.)
For your last point, you may be interested in stacksort: https://gkoberger.github.io/stacksort/
(sigh) You've finally convinced me to join Metaculus and start recording my own predictions.
Can I get a sanity check here? I occasionally place bets on Kalshi to see if I do as well in real-life markets as I do on the reputation based ones. Thus far I've done ok but really confused by the action on this question:
New COVID-19 case average by April 1 - https://kalshi.com/events/SCASER3/markets/SCASE-029
My modeling predicts that covid cases under <25,000 by April 1 is highly likely, probably by Friday unless BA2 creates an uptick (we're currently at about 27000 cases and dropping) but the market currently has it at 23%. There's $25000 in open interest so liquidity isn't an issue. Is there something I'm missing?
Delete if this is too self-serving but thought it may slot under prediction market goofiness.
Yeah, it's BA2: https://twitter.com/jburnmurdoch/status/1504497750741815299
It's likely that ~28k is the lowest that we'll see for a while: https://covid.cdc.gov/covid-data-tracker/#trends_dailycases
I guess this is actually a good example of a prediction market being right then! Today’s number was a marked increase. I didn’t think the BA2 spike would happen for another week or two tbh. Going to go back to the drawing board on my modeling, I think my error was not tracking NYC closely enough.
Looking over the prediction contest data, most of the averages and aggregates are similar to Scott's original prediction.
The one that is dramatically different is whether inflation will be < 3%. Scott says 80%, the average is 51%, and the aggregate is 35%.
>The only difference I can find is that Insight requires two media sources to resolve positively and Metaculus only one - surely people don’t think there’s a 40% chance troops will enter Kyiv but only one source will report it?
Seems more likely the error would be in the opposite direction. If they don't enter the city but at least one news source says they did, especially given existence of propaganda and fog of war
Some more weird stuffs about Aver: their Discord server has 19k+ members yet isn't really... active. That's a lot of big numbers for one platform we never heard of before. I hope someone will prove my suspicion wrong!
I'm excited about all the sketchy-looking crypto prediction platforms - it means more potential arbitrage opportunities.
So how much moderation is enough moderation for a prediction market site?
(edit to add: Manifold Markets doesn't feel like it has enough moderation.)
I would like it if the various glowfic markets were required to be categorized in a way that makes them less visible. Also sports betting. And anything involving root vegetables.
Hopefully some form of community moderation can solve that. The holy grail is "prediction-market based moderation", though you probably need two incommensurate currencies to pull that off. Roughly, if there is >95% certainty that a comment is bad, it is automatically pulled for an hour.
On the other hand, I signed up for Metaculus a few hours ago and submitted the "Turkey 2022 Name Change" question, and it still hasn't been approved. So sometimes less is more?
You can't defer moderation problems just by selling moderatorship to the highest bidder. That just increases the incentives for bad behavior!
This is Eric (one of the people running the forecasting contest.) Happy to answer questions about the aggregation method we used! I also want to second Scott's invitation for others to try their hand at aggregating the data, with a few caveats:
- Consider blinding yourself to the particular questions and only looking at the numbers. Otherwise you could end up "cheating" by e.g. choosing an aggregation method that assigns a particularly high probability to the Russia/Ukraine war question (which we now know will resolve positively).
- You can approach this in two ways. First, you could come up with an aggregation method that (like ours) *only* uses Scott's predictions and the contest participants' predictions and not any other data (e.g. "superforecaster or not") about the participants. Or alternatively, you can make your best guess about how to use the extra data to make a better aggregation method. I think both of these are interesting problems!
- I think there's a ton of luck in terms of which aggregation methods will work well, just because most aggregation methods will tend to give similar numbers and the sample of questions just isn't very large. Accordingly, there's a pretty decent chance your aggregation method will do worse than ours even if it's better, or do better than ours even if it's worse. But perhaps we'll take the approaches that do well this year and see how well they do next year!
Geometric mean of the odds is equivalent to taking the arithmetic mean of the log-odds (the result would be specified as log-odds in the second case), which seems like a reasonable justification for using it.
The EA forum link discussing that aggregation method doesn't seem to mention this fact at all, and I can't tell whether everyone thinks it's too obvious to mention, or no one noticed it or what.
Yeah, it's equivalent, and that's how I think about it. I agree that this provides a good justification, as basically this means that you're averaging each forecaster's "Bayesian evidence".
Nuno brought it up here (https://www.lesswrong.com/posts/mpDGNJFYzyKkg7zc2/aggregating-forecasts#oC5pJvoFFTXwxbK3F) and so I don't think anyone else has bothered to bring up the same point again. (Although personally I always try and use the phrase "average log-odds" rather than "geometric mean of odds")
If you folks want to make easy money, bet on the Oscar prediction market. It's ridiculously straightforward. The only public prediction that I made (in 2011) was pretty terrible by my standards and I got only 15 out of 24 categories right (but all the major ones excluding supporting actress) [1, 2]. But on a good year I could do up to 21 out of 24, before getting really bored of it.
[1] Post of the predictions before the Oscars (in Italian, sadly): http://www.fiveobstructions.com/?p=2292
[2] Aftermath post: http://www.fiveobstructions.com/?p=2302
My goodness you all don’t find the Ukrainian war casting @ betting on how many will be butchered utterly tasteless? There are no words…
It's not seen as a game here, but as a means to find true information about important complex issues. This war and where it will lead is important, complex, and usual sources have strong agendas.
By betting? The next agenda is betting on the Oscars….
I understand you're upset about it, but to put this into perspective, a lot of the people that are into prediction markets (on this blog, at least) are also into effective altruism. As a rule, there's a lot of money going from those same people toward helping out people directly affected by the war.
Minor case in point: I am betting (on Metaculus), but I've already put ~25% more than a full month's paycheck into helping out. This entire thing has me very upset, but I'm all the more interested in people's predictions, because it helps me figure out things like "how long will this likely last", which in turn lets me partition out my finances so I can still keep donating later on, instead of either burning everything now, or sitting on my money too long to be able to actually help out.
To reiterate, I'm very upset about this war; friends of close friends are directly affected. However, I'm not upset at people betting about it in prediction markets; even if they were just doing it because they want a few dollars for themselves (some might!), they're helping people like me make decisions. (And I imagine they're also helping people in much better positions to help than me to make decisions, too.)
I find the idea of wagering on the Ukrainian war deeply disturbing. I am a bit ashamed of myself, waiting for someone else to mention it first for fear of looking foolish. I suppose my position is not rational.
Thankfully it is not rational. Betting is not scientific but it is excused here in the name of rationality. You are speaking out because you are human. This place is a boys' club with supersized egos.
"You are speaking out because you are human." I agree with this and I don't think anyone should be ashamed of feeling queasy about prediction markets about war. War is a terrible thing and it's normal to want no one to benefit from it on the sidelines, especially if one doesn't see any obvious benefit in what is being done.
Mind you, independent of the emotions involved here and their validity, I do encourage you get to know the people you're judging before you dismiss them outright, and to consider that prediction markets might have benefits you too approve of. I touched on an example a bit further up this page. But that doesn't at all mean you're not allowed to feel queasy.
Still, I need to point out the obvious: "This place is a boys' club with supersized egos." This is neither true nor kind. I don't know if it's necessary, but I don't think so - I don't think this accusation is going to achieve anything.
Hi Neike. It is entirely true and I speak from experience. When I addressed astrological forecasting on tis forum and cited scientific published studies I was attacked badly with few exceptions and by people who know nothing of the subject: and yes, not one of them were women. When running out of arguments I was told that the study cited was not on a serious subject: it was on Amazon review misspellings (chosen btw because of free and massive data availability and for limiting variables) as if not having a "more serious" research subject somehow invalidates the methodology. Betting odds in Las Vegas, however, were treated very respectfully here apparently approved by the boys as an appropriate subject. Making money on the murders of Ukraine is heartless and can only be justified on the altars of reason without compassion even if those betting have good odds as they are "smart" and "capable of cognitive decoupling" as one of the links below explains. Who are these betters and whom do they benefit? Geopolitical experts or military advisors? You don't find placing money on how many cities will fall and how soon somewhat sociopathic? Someone below offered that betting on the Oscars is easier and more profitable for easy money. Obviously, I find your use of your profits admirable but they are more efficient ways to help people in Ukraine if that is someone's sole goal.
"It is entirely true and I speak from experience." It's not good form to judge an entire forum by its most vocal members; I'd ask you please not do that. I barely comment, because I rarely have anything insightful to ask, but I'm definitely part of this community, and have been for years. I've seen many outpourings of support in open threads for people who needed advice and help, and I generally trust that if I need good advice, I can come here and get it.
Is my experience less valid than yours? I'm honestly not trying to accuse you of anything here, you've clearly had a bad experience and I'm sincerely sorry you did, but do I encourage you not to tar the whole community with the same brush. (Astrology is certainly a topic people here have kneejerk reactions about.)
"Obviously, I find your use of your profits admirable but they are more efficient ways to help people in Ukraine if that is someone's sole goal."
Sorry, it looks like I need to clarify what I'm doing: I'm not making profits, I'm only using Metaculus, which doesn't involve money, just internet points. The money is just my paycheck; I have the savings to absorb losing a few of those, so I'm putting them into charities to help out. Metaculus and other prediction markets are just helping me to figure out e.g. whether I should spend another paycheck's worth of money *now* (from my savings), or prefer a more steady trickle of large (up to quarter-paycheck) weekly donations over a long time.
I participate in Metaculus because I want to improve my intuition for certain topics. It's useful to know when one is wrong and when one isn't, and having a website that tells you "nope, you were wrong about this, by this much" or "yep, you got this right!" helps a lot in getting a better feel for what's true and what's not over the long run. I don't "care" about the internet points, but I do care about predicting reality correctly; predicting reality correctly lets me make informed decisions about how to be more helpful.
By making a statement like "Making money on the murders of Ukraine is heartless", you're making a lot of assumptions about the motivations of the people involved. As mentioned, there's a significant overlap with EA here. If you can acknowledge that I'm not heartless (even though I predict in Metaculus), can you not imagine you're also wrong about the average prediction market user's psyche?
> Betting is not scientific but it is excused here in the name of rationality.
Are you familiar with the concept of "prediction markets"?
Winning or losing *one* bet, that is mostly a question of luck. But if two people make hundred bets, and one of them wins 80% of them, and the other loses 80% of them, chances are high that the former simply has better knowledge.
Also, betting is tax on bullshit. Imagine that one person says "I am absolutely sure it will rain tomorrow." And the other person says: "In that case, let's make a bet; I will give you $100 if it rains tomorrow, and you give me $100 if it doesn't. If you are absolutely certain, that means free money for you." Except... suddenly the former person is no longer so sure. (As a rule of thumb, anytime someone is "absolutely sure" but refuses to bet, they are actually not sure.)
> This place is a boys' club with supersized egos.
People with supersized egos who make lot of bets tend to lose their money.
I think the claim is that *wagering* on the war should in some sense be thought of as no more disturbing than *reading news* about the war. Neither of them actually changes the course of the war itself, and both claim to make you more informed about the war. The claim here is that wagering is the way to give people the discipline to read carefully, and understand what their information does and doesn't say.
Wagering on something that you *can* affect the outcome of can lead to all sorts of perverse motivations (which is why athletes aren't allowed to bet on their own sporting events, and which would be even worse if it were war we were talking about). But when it's just distant observers, the claim is that wagering on war is the way to get the best information in one place.
I feel like this perspective is... lacking in practicality. If we could gather a bunch of smart people to altruistically predict how long the war will last, that would be useful for planning/budgeting reasons. But getting people to contribute time and effort out of the goodness of their hearts is very difficult!
Prediction markets (try to) leverage human greed to good ends - to discover true information that would otherwise be difficult to get at, because there's no "force" pushing the right people or resources to working on the problem otherwise. If money can make it happen and it would be good to know, then that seems like a good thing.
No, it's not tasteless, let alone utterly so.
'Betting' doesn't seem meaningfully different from rewarding or punishing anyone for making good or better predictions.
Please read: https://www.reddit.com/r/slatestarcodex/comments/8fnch2/high_decouplers_and_low_decouplers/
The decoupling framework is bad, an example:
Suppose Alice is always talking about how blue-eyed people commit twice as many murders as the population average and blue-eyed Bob objects that this perpetuates harmful stereotypes that blue-eyed people are violent. The decoupling framework tells us that this is because Bob is a low-decoupler and Alice is a high-decoupler. But Bob is always talking about how brown-eyed people are more unemployed than the population average and brown-eyed Alice objects that he is perpetuating harmful stereotypes that brown-eyed people are lazy.
People often demand context where it benefits them and neglect it where it does not, this does not fit within the decoupling model's proposal that people operate on some consistent level of contextualization.
I don’t claim decoupling can explain all behavior, just that it’s one way to understand why people might bet on the outcome of a war and yet not be monsters.
I am still dubious about the idea that "gamblers", aggregated, are better predictors.
Is there really any research backing that proposition?
Is the wisdom of a crowd of gamblers more wise than the wisdom of the crowd.
The theory seems to be that having skin in the game makes one a better predictor, but aren't there 1000s of gamblers in bankruptcy.
Markets are only sort of efficient. And markets are also susceptible to corruption and manipulation. Even with some transparency and the SEC, there is still undetected insider trading and madoff scams. Is it really possible that an online prediction market where some one can bet as igottip234 can be a good thing?
Why does a known gambler like Nate Silver refuse to publicly state whether he bets on his own predictions when it is known that his predictions will affect odds in other markets?
Well, at the very least, we have existence proofs that prediction markets can be wildly off (like anything involving US politics, and particularly, Trump).
In the long term the bad gamblers go broke while the good gamblers take their money, then you have a market where most of the money is being thrown around by people with a proven track record which ought to mean something. Most prediction markets are too new, too small and too restricted for this to work, but it's one of those things that looks appealing in theory.
Are there really such things as "good" gamblers taking money from "bad" gamblers?
The house always wins in the long run, but they aren't really gambling at all.
The actual theory is that lots and lots of opinion tend in aggregation to lead to a pretty good prediction.
The wisdom of the crowds is not at all about gambling.
Gambling as a way to create a crowd of guesses is a slightly different thing.
Markets seems to be a way to create price discover it is similar to the wisdom of the crowds but might be theoretically slightly different. Markets can also lead to speculative bubbles which is different than the wisdom of the crowds. So there is some evidence that gambling introduces a error into what might just be sampling theory.
So there is the question which I will ask again: Is a crowd of gamblers better at predicting than a crowd?
Let's also address the iterative effect of "public" gambling,
Is a private auction (all bids are private) better at prediction than a public continuous auction?
Everybody seems to be saying x% but my gut says y% so I adjust to the crowd and alter my bid as if the crowds guesses are information.
Everyone ends up in Abilene when no one actually wanted to go to Abilene.
One more thing about the "skin the game" theory. A millionaire betting $100 has quite a lot less skin in the game than a pauper betting $100. So the skin in the game theory would have need to account for how much skin are we really talking about.
I really doubt that anybody is coming up with a market that requires the predictor to provide a copy of her tax return with each bet/prediction.
This seems like the important criticism to me. There's all sorts of good theory around prediction markets, but the empirics haven't yet borne it out.
I see that the linked Metaculus problem specifies resolution criteria of passing at least 2 out of 5 trials from a given list. Did anyone actually test Codex on those problems? How many did it pass?
Anyway, assuming that Codex did pass some of those, I think the problem is that people were estimating dates based on their interpretation of the question (when will AI become useful for code generation) rather than the actual resolution criteria (when will AI get good at copypasting leetcode solutions from Stack Overflow).
Edit: From the comments, it looks like people did test Codex and it did pass all the trials.
The market was open for months after Codex came out but I can't tell how much the market moved because the forecast timeline inexplicably stops at December 2019, two years before it was actually resolved. What's going on with that? If nothing moved post-Codex then that seems to firmly establish it as "People were interpreting the question based on the headline, not the much easier resolution criteria".
In what way can OpenAI’s Codex be said to be a program writing programs that a compiler cannot?
Both are programs that you give a high level specification to and it attempts to return a lower level program that accomplishes that spec.
The list from the links is of course much higher level and more freeform of a specification than, say, C code. But is it higher level as compared to its products? Is the difference between a paragraph and Python more than the difference between C and machine code? Good compiled and interpreted code already do read like their spec.
Programming is the art of specifying what you want so precisely that even a machine can do it.
Codex is more useful than prior art at generating/copypasting leetcode solution based on a *natural language* description of the problem. It is still utterly useless, and possibly actively harmful, when it comes to real world programming tasks.
Hum…let me see if I understand how this works.
A question is proposed - Will Ukraine successfully drive out Russian forces?
Currently the market might say 0.5% yes.
If territory begins to be taken back by Ukrainian forces that percentage will rise. If we get Russian forces in full retreat it might go to 95% and then as the last Russian troop leaves someone will win.
Where are we judging that the prediction market added any value?
We judge that the prediction market added value if there's some point at which someone sees some minor news story about some new style of Turkish drone the Ukrainian troops are getting, and uses that to raise the market price, and then a few days later the value of the drone becomes more obvious to the major news sources.
If no one has any sources other than the front pages of the major international newspapers, then the market isn't going to do much useful.
That newspaper-only world is not the world in which we live, though, and it hasn't been for more than a decade. So what value do prediction markets add in this world?
Well if prediction markets of war are not games or about money but advice and wisdom offered to high places as I am told ---that is exactly what I would like to know. How would a bunch of people's aggregate bet in SV would be helpful to know how many cities will fall to Putin next month or if he has Parkinsons more than the views of strategical advisors and journalists in the region who quite obviously have no clue.
That’s exactly the point. In the newspaper only world, the market adds nothing. But when there are many sources of information available, the market lets people express the value of the bits of unique information that they’ve been paying attention to and understanding the value of.
I wrote a little script that compiles a daily newsletter with the biggest Metaculus changes: https://newspredictions.substack.com/p/news-predictions-2022-03-23?showWelcome=true&s=w
It's very similar to the Metaculus Twitter Bot but, well, a newsletter.
> Even in late 2020, just before the question stopped accepting new predictions, the forecast was January 2027. The real answer was six months later, mid-2021, when OpenAI released Codex. I don’t want to update too much on a single data point, but this is quite the data point. If I had to cram this into the narrative of “not systematically underappreciating speed of AI progress”, I would draw on eg this question about fusion, where the resolution criteria (ignition) may have been met by an existing system - tech forecasters tend to underestimate the ability of cool prototypes to fulfill forecasting question criteria without being the One Amazing Breakthrough they’re looking for.
I think it's a much less spectacular data point than it might look, and the fusion example is a good example of why. If you think forecasters know how much more progress needs to be done, and were wrong about how fast that progress will get made, then the forecasters were off by an order of magnitude, which is a lot. But if you think forecasters knew how fast progress would happen, and were wrong about how much more needed to be done, then progress was 6 years closer to Codex than they thought it was, and expressing their error as a ratio instead wouldn't make sense, since they'd be consistently 6 years off according to this model (unless they learn more about how much progress needs to be done before getting Codex), but the ratio of their 6-year-too-late prediction to the actual time until Codex will change, asymptoting to infinity the moment before Codex is completed.
From Fox News:
“U.S. and European officials have been working to seize excessive yachts belonging to Putin's oligarch allies and moored in various NATO countries.”
The use of the word “excessive” in that paragraph makes me chuckle.
https://reason.com/2022/03/23/belgium-decriminalizes-prostitution/
How refreshingly rational.
Two months before invasion, metaculus gave ~30% odds of such an event. Even two weeks before the war started, it was still under 50% odds. https://www.metaculus.com/questions/8898/russian-invasion-of-ukraine-before-2023/