When I click on the links to download the data I get a tab that pops open briefly but no option to actually download the data (Chrome, Windows 10). Checking out the URLs I managed to get the actual data files.
I'm also not sure what went wrong, but I've added the full text of the links into the post, and attached the XLSX to Substack (it doesn't seem to accept .csvs)
I don't care for myself, but just wanted to note that it is very easy to personally identify people from, say, a meetup group located anywhere that is not CA (and many of the survey questions are very personal)
Combing through the answers is very interesting, I have to say. There is some usual 'too this side/too that side/biting your tongue in order not to offend X or Y' comments about "what would you change about ACX" so I did some very rough sorting based on survey questions. Without further ado, let me present:
THE GREAT BIG HUGE ARE WE ALL RIGHT WINGERS OR LEFT WINGERS ON HERE SORTING AND SHUFFLING GO-ROUND!
(Based on American political party affiliation, so all we non-Americans may be cosplaying Rosa Luxemburg or António de Oliveira Salazar in our spare time, but tough luck, it's USA! USA! USA!)
Democrats - 272
Libertarians - 14
Not Registered - 234
Other Third Party - 8
Republicans - 67
And counting up all the left wing versus right wing on "political affiliation" (left - liberals, Marxists, social democrats; right - alt-right, conservatives, libertarians, neo-reactionaries) that gives us 341 for the right to 707 for the left, a victory for truth, justice, and niceness for all.
Well that was easy, wasn't it? No we are not right-wingers, the right side of history conquers all, like love!
But hold hard a moment there, young lion! It's never that easy, now is it?
Based on how has hanging round the unsavoury bowels of this site changed our pristine opinions of yore, it is a different matter:
More left: 144
More right: 214
No change: 259
Other, not easily described: 446
So if we are shifting in our views, it looks like it tends to be more towards the right. O creeping extremism, who shall save us now? 😁
I think it'd only be easy for anyone in a country outside America. I forget if the survey asked for locality to the state/city level, but the data he released only lists country from what I'm seeing.
"There’s no such thing as cheating, short of time travel or murdering competitors" -- so murdering people other than competitors, such as subjects of questions, is not cheating either?
Hm, the obvious person to assassinate for the prediction market would be Putin, and I'm guessing it'd be hard to legally transfer money to someone in a Russian prison, especially if Russia fell apart after the assassination (although I guess you'd be executed if caught anyway)
I think you only wind up in a Russian prison (as opposed to an unmarked Russian grave) if you cause a terrible traffic accident on your way to attempt to assassinate Putin and don't accidentally tell the police where you were actually going.
You are, of course, aware, that in a community like this, more people will be trying to hack your process than will be trying to generate good answers.
People will be asking questions like: given the information that I am reasonably confident of, how do I create a small enough set of potential response combinations that I can submit them all, that maximizes the expected value of the best performing response?
This is not necessarily about making good predictions.
I should note that I ignored the perverse incentives when playing in Blind Mode. Not just the "if you want to come first in a prediction contest, rather than to maximise your expected score, some degree of overconfidence is optimal" thing, but also the "predict against things which are likely to stop Scott paying you" thing (there are a couple of questions which are unlikely to resolve true without resulting in Global Thermonuclear War).
Hi David, thanks for bringing up your concern, if I have an error I'll work to resolve it ASAP. I just took a look at the raw data sheet and noticed I had filtered column BD to hide blanks, which I've now unfiltered. That won't affect the results on other sheets but would keep people from being able to find their predictions in that sheet, could that have been why things didn't line up? Aside from filtering I made only one other adjustment to the .xlsx of raw data I downloaded, replacing "=#NULL!" values with blanks.
Oops, I forgot that you woudn't be able to change the selector without being able to edit. If you go to File ->Make a Copy you can create a copy that you can change the selection on.
You can't just say that without telling us which questions you think we're all wrong on.
The ones that look clearly wrong to me are #2 (I'm pretty sure Russia would use nukes to prevent this; Putin's explicitly drawn that redline and he can probably get away with using nukes in that kind of semi-defensive manner more than he can get away with breaking that promise), and #14 (the question amounts to "will the PRC stop lying about case numbers", since any decent caseload will hit it but their current lies won't, and I don't think that's particularly incentivised).
Actually, hard maxing your variance does not maximize your chance of winning. Rather the opposite (given the number of questions and players, anyway).
I did some analysis of the game theory surrounding this. Suppose you have a toy game where some number of players P is trying to predict the outcome of N coin flips, winner take all.
Obviously the 'correct' prediction is 0.5 probability for all flips, and that is unlikely to win assuming people play optimally. But the amount of variance you want to buy at the cost of correctness depends on the number of players (or, really, the number of players using game theory). It ends up being a mixed strategy with most of the correct play(s) being close to 'predict <log 2 P> flips with 0 or 1 probability'. With 3000 players and 50 flips, you'd want to predict around a dozen flips so long as N>12.
In the actual prediction game it becomes rather murkier because you would want to have some idea of how large the natural variance (based on different estimates) was, along with the actual advantage of good predictors. But the overall conclusion that just choosing 1 or 99 for all questions is not a winning strategy remains (unless the number of players becomes very large, like, far larger than Earth's population).
Do we know what scoring system will be used? That strategy don't sounds right based on how Scott was rating his own predictions. have less than 99% (respectively, more than 1%) of your 99% prediction come true should tank the result quite a bit.
Scott suggested Brier scoring was likely, but he hadn't committed to a specific system IIRC.
In practice it doesn't make too much difference to the optimal strategy for someone who is just trading correctness for variance. Using scoring based on log probability makes the penalty for being confidently wrong much higher, which does mean that the odds of someone who chose all 1 and 99 winning drop by orders of magnitude. But they were already extremely low.
In a coin flip prediction contest with optimal play, much of the expectation of winning comes from missing 0 or 1 predictions (under Brier scoring), and using log scoring changes that so that even missing 1 prediction is likely too much. This changes the number of flips someone would want to predict downwards by 1 or so, but it's still in the vicinity of log 2 P.
More generally, adding overconfidence only helps if there is already some consensus, close to correct probability that the orthodox predictions are somewhat clustered around. The limit of that model is the coin flip prediction game. In that situation, the scores of predictors not using game theory don't have a super wide distribution (indeed, it's fixed at 0.25 x the number of flips at the limit).
Philosophically this really comes down to the question of what kind of edge good predictors have. If all the events are genuinely unknowable or stochastic, then 'good prediction' is just estimating dice-like outcomes with precision, and the advantage of a good predictor is not very large. If on the other hand a good predictor is one who can identify some subset of events and say 'aha, the consensus estimate of 70% is incorrect, this won't happen (or has only a 2% chance of happening)', then the advantage is much larger, and just juicing your variance isn't going to give much chance of beating such a player, unless you also have a similar edge.
The distribution of scores (which I hope Scott publishes ... though they can always be calculated, if he provides spreadsheet data for the predictions) should be informative in terms of distinguishing between these two models of reality. But looking at the variation in blind predictions makes me suspect that there is already so much variance in the predictions of people who are playing it straight, that deliberately increasing your own simply isn't going to do very much to increase your odds of winning. Indeed the number of people who are Doing It All Wrong by choosing *all* 1 and 99 values leads me to say that players trying to use game theory are going to have collectively reduced chances of winning, because it isn't trivial to choose the right amount of overconfidence.
I entered the Prediction Contest mostly to calibrate how wrong my gut answers were, rather than to try and get them right. So Full Mode loses most of my interest (plus it sounds like...work, potential cash and internet points notwithstanding).
The variance is still fascinating though. Some topics frequently cleave the community at its joints, so I expect that on things like AI risk. Others, it makes me wonder at what kinda priors are running underneath the hood. "What do they know that I don't?" "Why so (un)confident?" (Iirc there was only a single question I felt truly deserved a 1-or-99 estimate., and only a couple true-50s either.)
Even as someone who doesn't plan on putting much effort into it, I think Full Mode will be interesting. In particular I'm interested to see how my gut answers and wild guesses compare to a field that [I predict] will mostly consist of people who are really trying, especially on topics that I think I know a lot about and the questions that make me wonder "who is even all that unsure about this? Do they know something I don't or is it the other way around?" That last question becomes much more interesting to me as the average level of topic knowledge among respondents increases.
Never forget the lizardman constant. I misread one of the questions and put in a "guaranteed" answer where it's far from such, only realising when I went back looking for places I disagreed with everyone else.
"Will prediction markets say Ron DeSantis is the most likely Republican nominee for President in 2024?"
Will *prediction markets* say it? Pretty sure they will.
Will he *be* the nominee? Different question, and it depends if he feels that this is a good time to go for it, or would it be better to wait it out, build up his qualifications for campaign, and go in 2028. I think he might wait it out.
I don't think that the people who bet on prediction markets are the same set of people who nominate Republican candidates, and I think it's a case of "what is the current biggest GOP name we hear bandied about, especially as a boogeyman for liberals?" re: will the prediction markets name him.
I also think DeSantis would be foolish to sit 2024 out hoping for a better shot later. Nobody ever gets a better shot than DeSantis has right now. People get bored and move on, and if he doesn't run now then they won't still be talking about him in 2028 and 2032 just like we've all forgotten Marco Rubio and Paul Walker and a dozen other people that were briefly discussed as the Next Big Thing.
I'm not that confident of either, because this is still pretty early in the election cycle for a contested nomination. I can imagine Trump acting as the effective incumbent for the GOP (though I don't think that's certain), but if he doesn't then it's historically been pretty hard to predict a front-runner this far out.
So you have a strong belief that you know better than the markets, which means you should be able to profit from this superior knowledge. I wonder why so many people are so wrong about this (to the extent that there aren't enough enlightened folks to come along and collect their handsome reward for correcting a mispricing like this).
If I had serious capital to commit to a speculative investment that won't pay off for 18 months, I'd find some interesting stock to buy. Maybe buy a fixer-upper duplex and become a landlord. Putting it into a online market run by nerd enthusiasts that could easily do an FTX and evaporate my cash entirely[1] would be just about the last thing I'd consider.
------------------
[1] Although to be sure the AUSA of the Southern District of New York would be certain to vigorously wag his finger at the miscreants, and the Times would do a somber chinstroker on Who Could Have Seen This Coming? in order to make the fleeced sheep feel better about their stupidity.
I think the SDNY are doing a bit more than finger-wagging, somebody seems to be mad as hell about Sam's Gang and some of them (Ellison, Wang) have taken the sensible step of cutting deals and agreeing to shove all the blame on Sam:
"Sam had put up the steps, and was preparing to jump upon the box, when he felt himself gently touched on the shoulder; and, looking round, his father stood before him. The old gentleman’s countenance wore a mournful expression, as he shook his head gravely, and said, in warning accents —
‘I know’d what ‘ud come o’ this here mode o’ doin’ bisness. Oh, Sammy, Sammy, vy worn’t there a alleybi!’
Yep. A lot of well-off donors to New York Democrats lost money, so someone needs to pay. But it's interesting that they cut a deal with Ellison. That still suggests some significant degree of gingerness here. They're not treating it like a mafia case, and *all* these people need to go to jail. Nor do I think they really, really need her testimony to hang SBF. What I surmise is going on is that powerful people want SBF to cop a plea and quietly go away to jail for a while, without this case becoming a three-ring circus during the trial, OJ style. Too many red-faces among those important pillar o' the community people. So the purpose of the deals with the other players is to isolate SBF, to crank up the pressure on him to avoid the trial and cut a deal. It's going to be a fairly shitty deal, so they need a lot of pressure, a lot of sense of "you just can't win this."
My own view is that the deal with Ellison is partly because this entire affair is such an unholy mess. Whether or not Bankman-Fried was a scammer from day one is up for debate, but the impression I get is that the entire set-up is a horrible, tangled mess. So they need Ellison or somebody who was at a high-up level to help them figure out what the hell was going on.
Bankman-Fried is playing dumb, but I think there is also an element of truth in that he has no idea exactly what was going on where. John J. Ray's testimony to the congressional committee is very informative (and as entertaining as his bankruptcy filing) - there is such a Gordian knot of companies that were owned by about three people and robbing Peter to pay Paul that simply finding out even how many people worked there is a difficult question:
There's about four entities that all want a piece of Bankman-Fried: the SDNY, the SEC, the CFTC and the Bahamas. So while there are people who got large political donations and wish it would all go away, I don't think it will.
Maybe. They already arrested SBF and charged him, so presumably they think they understand pretty well what happened -- or at least, enough that they could clearly point to a law broken by a specific set of actions. They would need that to file charges in the first place. They might certainly be interested in uncovering more crimes, to pile more charges on, but they've already charged him with enough to send him up the river for a long time. I would guess their main concern now is building the case, so the jury can be convinced even in the face of whatever the defense has to offer by way of alternate explanation, justification, questioning the evidence.
It may indeed be useful to cut a deal with Ellison for that purpose. Since she was heavily involved, she could otherwise invoke her Fifth Amendment rights and say nothing. But if they immunize her against certain charges, she becomes obligated to say what she knows about those events -- her Fifth Amendment rights vanish.
I of course can't speak for Deiseach, and personally any confidence I might have had re my ability to predict future events has been gone for at least eight years.
But even if I were more confident, my feelings about messing about in new kinds of markets that I don't fully understand and that other people are studying much closer than me, whether crypto, prediction, or other, are pretty much summed up by:
"One of these days in your travels, a guy is going to come up to you and show you a nice brand-new deck of cards on which the seal is not yet broken, and this guy is going to offer to bet you that he can make the Jack of Spades jump out of the deck and squirt cider in your ear. But, son, do not bet this man, for as sure as you are standing there, you are going to end up with an earful of cider."
The one thing I'm confident of is that I don't know enough about (or have enough trust in) either market to go seriously fishing in them with my own money. Even if I someday see what looks like a market offering a future dollar against a penny paid for the proposition "The Sun Will Rise Tomorrow".
Or rather, per the quote, *especially* if that's what it looks like they're offering.
Explaining why I'm not betting on this requires me to give out more personal knowledge than I think you have any right to know, so the short answer is "not betting".
I'm not setting out to have superior knowledge, just that (1) I don't trust what I've seen of these markets (2) I think it's too early to be speculating on who the Republican candidate will be (3) I do think there is a disconnect between the people who bet on these markets, because they're aware of them, and the people who will be making decisions about candidates to back for the other side.
Do you think Biden is going to run in 2024? What about his age? If not Biden, then who - Kamala? Mayor Pete? Someone else? There's a gap there between "candidate who looks from the outside like they're the best pick" and "candidate who really has a chance of making it past the primaries and the debates and the entire circus".
Two points: first of all, the markets actually agree that RdS is more likely not to be the candidate than to be the candidate; he's at about 41% at both Predictit and Betfair. He's the most likely of the people named, but that's not the same prediction.
But more generally, this seems to be an attitude of "how dare you be so arrogant as to disagree with the almighty prediction market 18 months out?" which I see no basis for holding. This has been discussed here before so those arguments in both directions are available, so I'll just mention this:
DeSantis's qualifications are as good as they're going to get. Florida is doing well, he turned a big and growing purple state deep red, which has all the party insiders and moneybags chortling with glee ("Do it again with Pennsylvania and North Carolina! w00t!"). He was just re-elected by an enormous margin, and he already has national visibility, thanks in part to Trump's porcine bellowing and DeSantis's ability to pick widely-reported fights with the right kind of lefties -- having the mayor of New York or the governor of Massachusetts shake their fists at him is probably worth a million Republican primary votes each. Waiting is not really his friend -- it's just a chance for random bad luck to hurt his chances.
I think the most important decision he needs to weigh is: just how vulnerable is Joe Biden? It's rare for Americans since ~1940 to shorten a President's term, and doing so twice in a row has arguably never happened, depending on how you want to classify Ford's defeat in 1976. Probably this depends mostly on whether inflation subsides on its own, without a brutal recession -- or if the necessary recession is sufficiently short that it's Morning In America by the spring of '24, although DeSantis needs to make his decision well before then. If the Biden economy in late '23 looks like the Carter economy in late '75 (complete with the same feckless saccharine moralizing, or Biden's patented ex-NJ boxer with TBI aphasic swagger) then I think DeSantis is totally in. The one thing he most doesn't want to have happen is some *other* Republican to win the White House in '24, and then he has a nasty primary slog in '28, or has to wait until '32, which is a long time. DeSantis is a young man, certainly, but you got to seize the brass ring when it's offered, because it might not come around again at all.
Edit: also, the Senate elections in '24 are unusually unfriendly to Democrats. They're defending 23 of the 33 seats up, of which ~2-8 are in states where Biden's victory was slim to absent. A victorious Republican President with good coattails would probably bring a decent-sized Senate Republican majority and an enlarged House Republican majority. A solid control of both houses of Congress would be a sweet, sweet deal for any Republican President wanting to shake things up and be remembered with bitter curses by half a generation of journalists, Chief Diversity Officers, and Hollywood actors. This probably adds to the temptation to give it a shot.
I downloaded the data and ran some correlations. What I found most surprising was how few things were correlated. The political things were correlated with each other, but most things were pretty uncorrelated. I am interested in seeing what there is with the more full dataset. Anyway, here's some funny correlations I found: (they are all pretty small so don't read into them too much)
I don't fly in planes or drive, I'm not married/no children, I'm not high IQ, I'm religious, I don't support abortion and I have had COVID.
On the other hand I am vaccinated, I don't take LSD, I do have some anxiety, I am humanities not STEM, I don't believe humanity will go extinct due to AI and I love bread and pasta (and spuds, how can we forget the spuds?) 😁
If you answered the survey and got those results linked to your answers, you can track yourself down in the spreadsheet based on your details (that's what I did).
What is the best way to find out what my own predictions actually were? Or, equivalently, how can I figure out which line on the spreadsheet represents me?
If I have time I'd like to make a data notebook of this (Quarto) and put it on Kaggle, also maybe run the R Aggrecat package https://osf.io/preprints/metaarxiv/74tfv and share it --
1. Does that make sense, or is that against the spirit of the competition
I jotted down my answers in a file as I submitted them. Now I see they're not in the spreadsheet of submissions. I assume this is user (me) error, but noting it here just in case it's a symptom of a larger issue.
This means I've missed out on my $0.15 expected gain from my shot at the prize money. Or maybe not. Anyone who's participating in the "Full Mode" predictions and wants the added substantial advantage of knowing my predictions can pay me $0.15. Just reply to this comment and we'll make arrangements. I only need one response to break even! I even accept contingency payments...you pay only if you win!
In the interest of truth in advertising, the correlation between my answers and the mean of all answers is 0.79. I wonder what the correlation would be for an optimally valuable additional set of answers? 1.00 corresponds to useless, so I've probably overvalued my answers at $0.15.
When I click on the links to download the data I get a tab that pops open briefly but no option to actually download the data (Chrome, Windows 10). Checking out the URLs I managed to get the actual data files.
XLSX: http://slatestarcodex.com/Stuff/2023blindmode_predictions.xlsx
CSV: http://slatestarcodex.com/Stuff/2023blindmode_predictions.csv
EDIT:
Weird, even those links still open a tab for me that just closes. I can download the data if I:
1. Open the link in a new window
2. Click on the URL in the browser bar at the top of the screen
3. Press 'ENTER'
Weird Substack behaviour?
I get a message telling me it's a potential security risk and that the file uses an insecure connection. Both from the original links and yours.
I'm also not sure what went wrong, but I've added the full text of the links into the post, and attached the XLSX to Substack (it doesn't seem to accept .csvs)
It works for me, using Edge on Windows 10. I'm surprised at the difference, given Edge is now using the Chromium engine.
Some browsers don't like downloading things over http that were linked from a site using https. Changing the links to https should fix this.
Verified, this works: https://slatestarcodex.com/Stuff/2023blindmode_predictions.csv
I don't care for myself, but just wanted to note that it is very easy to personally identify people from, say, a meetup group located anywhere that is not CA (and many of the survey questions are very personal)
Likewise, I thought the ACX survey linkage was for Scott’s analysis only (although I guess there’s no going back now.)
This is the prediction contest, not the ACX Survey
Download the file, he correlated the ACX answers to the predictions.
orly. My apologies.
Combing through the answers is very interesting, I have to say. There is some usual 'too this side/too that side/biting your tongue in order not to offend X or Y' comments about "what would you change about ACX" so I did some very rough sorting based on survey questions. Without further ado, let me present:
THE GREAT BIG HUGE ARE WE ALL RIGHT WINGERS OR LEFT WINGERS ON HERE SORTING AND SHUFFLING GO-ROUND!
(Based on American political party affiliation, so all we non-Americans may be cosplaying Rosa Luxemburg or António de Oliveira Salazar in our spare time, but tough luck, it's USA! USA! USA!)
Democrats - 272
Libertarians - 14
Not Registered - 234
Other Third Party - 8
Republicans - 67
And counting up all the left wing versus right wing on "political affiliation" (left - liberals, Marxists, social democrats; right - alt-right, conservatives, libertarians, neo-reactionaries) that gives us 341 for the right to 707 for the left, a victory for truth, justice, and niceness for all.
Well that was easy, wasn't it? No we are not right-wingers, the right side of history conquers all, like love!
But hold hard a moment there, young lion! It's never that easy, now is it?
Based on how has hanging round the unsavoury bowels of this site changed our pristine opinions of yore, it is a different matter:
More left: 144
More right: 214
No change: 259
Other, not easily described: 446
So if we are shifting in our views, it looks like it tends to be more towards the right. O creeping extremism, who shall save us now? 😁
I mean, if Scott is a centrist, and there's twice as many left wingers as right wingers, and he moves everybody to the center...
Need to disentangle any putative effect of hanging around a substack from simply getting older.
I think it'd only be easy for anyone in a country outside America. I forget if the survey asked for locality to the state/city level, but the data he released only lists country from what I'm seeing.
Oh great. It originally had state included. He must have removed it since the original post
"There’s no such thing as cheating, short of time travel or murdering competitors" -- so murdering people other than competitors, such as subjects of questions, is not cheating either?
AFAICT this is correct - you would still be allowed to collect your $500 from prison (one of the great usecases of cryptocurrencies!)
Why would you need cryptocurrency? I don't think there would be anything stopping Scott from submitting the $500 to your commissary fund.
Hm, the obvious person to assassinate for the prediction market would be Putin, and I'm guessing it'd be hard to legally transfer money to someone in a Russian prison, especially if Russia fell apart after the assassination (although I guess you'd be executed if caught anyway)
I think you only wind up in a Russian prison for an unsuccessful assassination *attempt* against Putin.
I think you only wind up in a Russian prison (as opposed to an unmarked Russian grave) if you cause a terrible traffic accident on your way to attempt to assassinate Putin and don't accidentally tell the police where you were actually going.
Next year’s ACX survey should include “How many crimes have you committed to alter the payouts of prediction markets you have entered?”
You are, of course, aware, that in a community like this, more people will be trying to hack your process than will be trying to generate good answers.
People will be asking questions like: given the information that I am reasonably confident of, how do I create a small enough set of potential response combinations that I can submit them all, that maximizes the expected value of the best performing response?
This is not necessarily about making good predictions.
I think that's part of the game, and why there are different prizes for people who complete over a certain percentage of prediction questions.
I should note that I ignored the perverse incentives when playing in Blind Mode. Not just the "if you want to come first in a prediction contest, rather than to maximise your expected score, some degree of overconfidence is optimal" thing, but also the "predict against things which are likely to stop Scott paying you" thing (there are a couple of questions which are unlikely to resolve true without resulting in Global Thermonuclear War).
Same, I’m mostly using this as an opportunity to test how calibrated my predictions are, not to win the contest.
If you want to take a brief look at the predictions, I put together a Google sheet with some simple summaries. One tab has some percentiles for all questions, the other has a selector to view the histogram for any individual question. https://docs.google.com/spreadsheets/d/1jiRhSp7MmPsJp64oMlBHiHqtlm7Ix_u2tWi0XTX_Y7w/edit?usp=sharing
Hi David, thanks for bringing up your concern, if I have an error I'll work to resolve it ASAP. I just took a look at the raw data sheet and noticed I had filtered column BD to hide blanks, which I've now unfiltered. That won't affect the results on other sheets but would keep people from being able to find their predictions in that sheet, could that have been why things didn't line up? Aside from filtering I made only one other adjustment to the .xlsx of raw data I downloaded, replacing "=#NULL!" values with blanks.
Possibly. I now see my entry, thatnks!
Phew, that's a relief! I appreciate you commenting to let me know something was off, I wish I had noticed that filtering earlier.
Thanks that's helpful, although I couldn't make the selector on the histogram work.
I am pretty excited about how WRONG you all are. Can I just get paid now? lol
Oops, I forgot that you woudn't be able to change the selector without being able to edit. If you go to File ->Make a Copy you can create a copy that you can change the selection on.
ah yes thank you sir
You can't just say that without telling us which questions you think we're all wrong on.
The ones that look clearly wrong to me are #2 (I'm pretty sure Russia would use nukes to prevent this; Putin's explicitly drawn that redline and he can probably get away with using nukes in that kind of semi-defensive manner more than he can get away with breaking that promise), and #14 (the question amounts to "will the PRC stop lying about case numbers", since any decent caseload will hit it but their current lies won't, and I don't think that's particularly incentivised).
Good points. I haven't really looked at how everybody voted in a detailed way yet. Just my weak attempt at humor.
Pretty wild how many questions have a significant number of people at both 99 and 1.
Yet for the most part the averages come out reasonable (by which I mean, close to my own answers!).
it makes sense if you're trying to max out your probability of winning instead of your expected score.
Actually, hard maxing your variance does not maximize your chance of winning. Rather the opposite (given the number of questions and players, anyway).
I did some analysis of the game theory surrounding this. Suppose you have a toy game where some number of players P is trying to predict the outcome of N coin flips, winner take all.
Obviously the 'correct' prediction is 0.5 probability for all flips, and that is unlikely to win assuming people play optimally. But the amount of variance you want to buy at the cost of correctness depends on the number of players (or, really, the number of players using game theory). It ends up being a mixed strategy with most of the correct play(s) being close to 'predict <log 2 P> flips with 0 or 1 probability'. With 3000 players and 50 flips, you'd want to predict around a dozen flips so long as N>12.
In the actual prediction game it becomes rather murkier because you would want to have some idea of how large the natural variance (based on different estimates) was, along with the actual advantage of good predictors. But the overall conclusion that just choosing 1 or 99 for all questions is not a winning strategy remains (unless the number of players becomes very large, like, far larger than Earth's population).
Do we know what scoring system will be used? That strategy don't sounds right based on how Scott was rating his own predictions. have less than 99% (respectively, more than 1%) of your 99% prediction come true should tank the result quite a bit.
Scott suggested Brier scoring was likely, but he hadn't committed to a specific system IIRC.
In practice it doesn't make too much difference to the optimal strategy for someone who is just trading correctness for variance. Using scoring based on log probability makes the penalty for being confidently wrong much higher, which does mean that the odds of someone who chose all 1 and 99 winning drop by orders of magnitude. But they were already extremely low.
In a coin flip prediction contest with optimal play, much of the expectation of winning comes from missing 0 or 1 predictions (under Brier scoring), and using log scoring changes that so that even missing 1 prediction is likely too much. This changes the number of flips someone would want to predict downwards by 1 or so, but it's still in the vicinity of log 2 P.
More generally, adding overconfidence only helps if there is already some consensus, close to correct probability that the orthodox predictions are somewhat clustered around. The limit of that model is the coin flip prediction game. In that situation, the scores of predictors not using game theory don't have a super wide distribution (indeed, it's fixed at 0.25 x the number of flips at the limit).
Philosophically this really comes down to the question of what kind of edge good predictors have. If all the events are genuinely unknowable or stochastic, then 'good prediction' is just estimating dice-like outcomes with precision, and the advantage of a good predictor is not very large. If on the other hand a good predictor is one who can identify some subset of events and say 'aha, the consensus estimate of 70% is incorrect, this won't happen (or has only a 2% chance of happening)', then the advantage is much larger, and just juicing your variance isn't going to give much chance of beating such a player, unless you also have a similar edge.
The distribution of scores (which I hope Scott publishes ... though they can always be calculated, if he provides spreadsheet data for the predictions) should be informative in terms of distinguishing between these two models of reality. But looking at the variation in blind predictions makes me suspect that there is already so much variance in the predictions of people who are playing it straight, that deliberately increasing your own simply isn't going to do very much to increase your odds of winning. Indeed the number of people who are Doing It All Wrong by choosing *all* 1 and 99 values leads me to say that players trying to use game theory are going to have collectively reduced chances of winning, because it isn't trivial to choose the right amount of overconfidence.
It's not really that significant a number of people; there aren't any questions where 5th percentile = 1 and 95th percentile = 99.
I entered the Prediction Contest mostly to calibrate how wrong my gut answers were, rather than to try and get them right. So Full Mode loses most of my interest (plus it sounds like...work, potential cash and internet points notwithstanding).
The variance is still fascinating though. Some topics frequently cleave the community at its joints, so I expect that on things like AI risk. Others, it makes me wonder at what kinda priors are running underneath the hood. "What do they know that I don't?" "Why so (un)confident?" (Iirc there was only a single question I felt truly deserved a 1-or-99 estimate., and only a couple true-50s either.)
Even as someone who doesn't plan on putting much effort into it, I think Full Mode will be interesting. In particular I'm interested to see how my gut answers and wild guesses compare to a field that [I predict] will mostly consist of people who are really trying, especially on topics that I think I know a lot about and the questions that make me wonder "who is even all that unsure about this? Do they know something I don't or is it the other way around?" That last question becomes much more interesting to me as the average level of topic knowledge among respondents increases.
Never forget the lizardman constant. I misread one of the questions and put in a "guaranteed" answer where it's far from such, only realising when I went back looking for places I disagreed with everyone else.
The beginning of the survey says "Five Quick Demographic Questions" but only three demographic questions follow
Thanks, fixed.
The CSV link points also to the XLSX data
Thanks, fixed.
Hmmm, thing is, for this one:
"Will prediction markets say Ron DeSantis is the most likely Republican nominee for President in 2024?"
Will *prediction markets* say it? Pretty sure they will.
Will he *be* the nominee? Different question, and it depends if he feels that this is a good time to go for it, or would it be better to wait it out, build up his qualifications for campaign, and go in 2028. I think he might wait it out.
I don't think that the people who bet on prediction markets are the same set of people who nominate Republican candidates, and I think it's a case of "what is the current biggest GOP name we hear bandied about, especially as a boogeyman for liberals?" re: will the prediction markets name him.
Have you maxed out your bets on this position? Easy money if you really believe that.
You are asking me to do money things on computer? Get thee behind me, Satan! 😁
I do believe it. But I don't bet in real life because I am rubbish at money management and would lose my shirt *and* socks.
I think some of them are the same people.
I also think DeSantis would be foolish to sit 2024 out hoping for a better shot later. Nobody ever gets a better shot than DeSantis has right now. People get bored and move on, and if he doesn't run now then they won't still be talking about him in 2028 and 2032 just like we've all forgotten Marco Rubio and Paul Walker and a dozen other people that were briefly discussed as the Next Big Thing.
I'm not that confident of either, because this is still pretty early in the election cycle for a contested nomination. I can imagine Trump acting as the effective incumbent for the GOP (though I don't think that's certain), but if he doesn't then it's historically been pretty hard to predict a front-runner this far out.
So you have a strong belief that you know better than the markets, which means you should be able to profit from this superior knowledge. I wonder why so many people are so wrong about this (to the extent that there aren't enough enlightened folks to come along and collect their handsome reward for correcting a mispricing like this).
If I had serious capital to commit to a speculative investment that won't pay off for 18 months, I'd find some interesting stock to buy. Maybe buy a fixer-upper duplex and become a landlord. Putting it into a online market run by nerd enthusiasts that could easily do an FTX and evaporate my cash entirely[1] would be just about the last thing I'd consider.
------------------
[1] Although to be sure the AUSA of the Southern District of New York would be certain to vigorously wag his finger at the miscreants, and the Times would do a somber chinstroker on Who Could Have Seen This Coming? in order to make the fleeced sheep feel better about their stupidity.
I think the SDNY are doing a bit more than finger-wagging, somebody seems to be mad as hell about Sam's Gang and some of them (Ellison, Wang) have taken the sensible step of cutting deals and agreeing to shove all the blame on Sam:
https://nymag.com/intelligencer/2022/12/sbf-lieutenants-caroline-ellison-and-gary-wang-plead-guilty.html
To quote "The Pickwick Papers":
"Sam had put up the steps, and was preparing to jump upon the box, when he felt himself gently touched on the shoulder; and, looking round, his father stood before him. The old gentleman’s countenance wore a mournful expression, as he shook his head gravely, and said, in warning accents —
‘I know’d what ‘ud come o’ this here mode o’ doin’ bisness. Oh, Sammy, Sammy, vy worn’t there a alleybi!’
Yep. A lot of well-off donors to New York Democrats lost money, so someone needs to pay. But it's interesting that they cut a deal with Ellison. That still suggests some significant degree of gingerness here. They're not treating it like a mafia case, and *all* these people need to go to jail. Nor do I think they really, really need her testimony to hang SBF. What I surmise is going on is that powerful people want SBF to cop a plea and quietly go away to jail for a while, without this case becoming a three-ring circus during the trial, OJ style. Too many red-faces among those important pillar o' the community people. So the purpose of the deals with the other players is to isolate SBF, to crank up the pressure on him to avoid the trial and cut a deal. It's going to be a fairly shitty deal, so they need a lot of pressure, a lot of sense of "you just can't win this."
My own view is that the deal with Ellison is partly because this entire affair is such an unholy mess. Whether or not Bankman-Fried was a scammer from day one is up for debate, but the impression I get is that the entire set-up is a horrible, tangled mess. So they need Ellison or somebody who was at a high-up level to help them figure out what the hell was going on.
Bankman-Fried is playing dumb, but I think there is also an element of truth in that he has no idea exactly what was going on where. John J. Ray's testimony to the congressional committee is very informative (and as entertaining as his bankruptcy filing) - there is such a Gordian knot of companies that were owned by about three people and robbing Peter to pay Paul that simply finding out even how many people worked there is a difficult question:
https://www.youtube.com/watch?v=YQdvfBZ0VbQ
There's about four entities that all want a piece of Bankman-Fried: the SDNY, the SEC, the CFTC and the Bahamas. So while there are people who got large political donations and wish it would all go away, I don't think it will.
Maybe. They already arrested SBF and charged him, so presumably they think they understand pretty well what happened -- or at least, enough that they could clearly point to a law broken by a specific set of actions. They would need that to file charges in the first place. They might certainly be interested in uncovering more crimes, to pile more charges on, but they've already charged him with enough to send him up the river for a long time. I would guess their main concern now is building the case, so the jury can be convinced even in the face of whatever the defense has to offer by way of alternate explanation, justification, questioning the evidence.
It may indeed be useful to cut a deal with Ellison for that purpose. Since she was heavily involved, she could otherwise invoke her Fifth Amendment rights and say nothing. But if they immunize her against certain charges, she becomes obligated to say what she knows about those events -- her Fifth Amendment rights vanish.
I of course can't speak for Deiseach, and personally any confidence I might have had re my ability to predict future events has been gone for at least eight years.
But even if I were more confident, my feelings about messing about in new kinds of markets that I don't fully understand and that other people are studying much closer than me, whether crypto, prediction, or other, are pretty much summed up by:
"One of these days in your travels, a guy is going to come up to you and show you a nice brand-new deck of cards on which the seal is not yet broken, and this guy is going to offer to bet you that he can make the Jack of Spades jump out of the deck and squirt cider in your ear. But, son, do not bet this man, for as sure as you are standing there, you are going to end up with an earful of cider."
The one thing I'm confident of is that I don't know enough about (or have enough trust in) either market to go seriously fishing in them with my own money. Even if I someday see what looks like a market offering a future dollar against a penny paid for the proposition "The Sun Will Rise Tomorrow".
Or rather, per the quote, *especially* if that's what it looks like they're offering.
Explaining why I'm not betting on this requires me to give out more personal knowledge than I think you have any right to know, so the short answer is "not betting".
I'm not setting out to have superior knowledge, just that (1) I don't trust what I've seen of these markets (2) I think it's too early to be speculating on who the Republican candidate will be (3) I do think there is a disconnect between the people who bet on these markets, because they're aware of them, and the people who will be making decisions about candidates to back for the other side.
Do you think Biden is going to run in 2024? What about his age? If not Biden, then who - Kamala? Mayor Pete? Someone else? There's a gap there between "candidate who looks from the outside like they're the best pick" and "candidate who really has a chance of making it past the primaries and the debates and the entire circus".
Two points: first of all, the markets actually agree that RdS is more likely not to be the candidate than to be the candidate; he's at about 41% at both Predictit and Betfair. He's the most likely of the people named, but that's not the same prediction.
But more generally, this seems to be an attitude of "how dare you be so arrogant as to disagree with the almighty prediction market 18 months out?" which I see no basis for holding. This has been discussed here before so those arguments in both directions are available, so I'll just mention this:
https://www.forbes.com/sites/georgecalhoun/2022/11/14/the-un-wisdom-of-crowds-prediction-markets-failed-their-midterm-exams/?sh=4ba72278179a
DeSantis's qualifications are as good as they're going to get. Florida is doing well, he turned a big and growing purple state deep red, which has all the party insiders and moneybags chortling with glee ("Do it again with Pennsylvania and North Carolina! w00t!"). He was just re-elected by an enormous margin, and he already has national visibility, thanks in part to Trump's porcine bellowing and DeSantis's ability to pick widely-reported fights with the right kind of lefties -- having the mayor of New York or the governor of Massachusetts shake their fists at him is probably worth a million Republican primary votes each. Waiting is not really his friend -- it's just a chance for random bad luck to hurt his chances.
I think the most important decision he needs to weigh is: just how vulnerable is Joe Biden? It's rare for Americans since ~1940 to shorten a President's term, and doing so twice in a row has arguably never happened, depending on how you want to classify Ford's defeat in 1976. Probably this depends mostly on whether inflation subsides on its own, without a brutal recession -- or if the necessary recession is sufficiently short that it's Morning In America by the spring of '24, although DeSantis needs to make his decision well before then. If the Biden economy in late '23 looks like the Carter economy in late '75 (complete with the same feckless saccharine moralizing, or Biden's patented ex-NJ boxer with TBI aphasic swagger) then I think DeSantis is totally in. The one thing he most doesn't want to have happen is some *other* Republican to win the White House in '24, and then he has a nasty primary slog in '28, or has to wait until '32, which is a long time. DeSantis is a young man, certainly, but you got to seize the brass ring when it's offered, because it might not come around again at all.
Edit: also, the Senate elections in '24 are unusually unfriendly to Democrats. They're defending 23 of the 33 seats up, of which ~2-8 are in states where Biden's victory was slim to absent. A victorious Republican President with good coattails would probably bring a decent-sized Senate Republican majority and an enlarged House Republican majority. A solid control of both houses of Congress would be a sweet, sweet deal for any Republican President wanting to shake things up and be remembered with bitter curses by half a generation of journalists, Chief Diversity Officers, and Hollywood actors. This probably adds to the temptation to give it a shot.
I downloaded the data and ran some correlations. What I found most surprising was how few things were correlated. The political things were correlated with each other, but most things were pretty uncorrelated. I am interested in seeing what there is with the more full dataset. Anyway, here's some funny correlations I found: (they are all pretty small so don't read into them too much)
Educated behavior:
1. Getting married
2. Flying on planes
3. Having children
4. Driving cars
5. Being high IQ
Uneducated behavior:
1. Believing humanity will go extinct due to AI
2. Considering suicide
3. Supporting Donald Trump
4. Enjoying ACX meetups
5. Being a STEM person (as opposed to humanities)
Vaccinated behavior:
1. Trusting the mainstream media
2. Supporting abortion
3. Supporting feminism
4. Being an effective altruist
5. Flying on planes
Unvaccinated behavior:
1. Supporting Donald Trump
2. Being a Republican
3. Having worse long COVID
4. Being religious
5. Living in the suburbs
Risky behavior:
1. Flying on planes
2. Taking LSD
3. Believing nuclear weapons will be used in 2023
4. Being in a relationship
5. Getting COVID
Safe behavior:
1. Having anxiety
2. Trusting the mainstream media
3. Living in the suburbs
4. Ignoring emails
5. Enjoying bread and pasta
As ever, I am a radical centrist based on this!
I don't fly in planes or drive, I'm not married/no children, I'm not high IQ, I'm religious, I don't support abortion and I have had COVID.
On the other hand I am vaccinated, I don't take LSD, I do have some anxiety, I am humanities not STEM, I don't believe humanity will go extinct due to AI and I love bread and pasta (and spuds, how can we forget the spuds?) 😁
Is there any way that I can review my blind mode answers ?
If you answered the survey and got those results linked to your answers, you can track yourself down in the spreadsheet based on your details (that's what I did).
What is the best way to find out what my own predictions actually were? Or, equivalently, how can I figure out which line on the spreadsheet represents me?
If I have time I'd like to make a data notebook of this (Quarto) and put it on Kaggle, also maybe run the R Aggrecat package https://osf.io/preprints/metaarxiv/74tfv and share it --
1. Does that make sense, or is that against the spirit of the competition
2. Has anyone done this/is anyone doing this
3. Anyone want to collab on this?
I jotted down my answers in a file as I submitted them. Now I see they're not in the spreadsheet of submissions. I assume this is user (me) error, but noting it here just in case it's a symptom of a larger issue.
This means I've missed out on my $0.15 expected gain from my shot at the prize money. Or maybe not. Anyone who's participating in the "Full Mode" predictions and wants the added substantial advantage of knowing my predictions can pay me $0.15. Just reply to this comment and we'll make arrangements. I only need one response to break even! I even accept contingency payments...you pay only if you win!
In the interest of truth in advertising, the correlation between my answers and the mean of all answers is 0.79. I wonder what the correlation would be for an optimally valuable additional set of answers? 1.00 corresponds to useless, so I've probably overvalued my answers at $0.15.
For next time, it would be helpful to get the blind mode answers in a Google Sheet as well, just for those who might want to get a quick sense of it.
I can't see my entry. Nobody with my age and religion. I'm sure I submitted correctly.