175 Comments

1. Will an existing, validated personality assessment tool or validated questions be used in the upcoming survey to assess personality?

2. If it became widely known that people with X (personality) trait are poor predictors (or at least widely known by prediction model participants), will those people self-select out of prediction markets, especially those associated with actual money (some people already decline to participate in polls and some people don't "gamble")?

3. Could it be that some personalities predict some things well, but other things poorly? Would that correlation be looked for?

Expand full comment

I'm using that as a stand-in for "the kind of thing on the ACX Survey". See past surveys (eg https://docs.google.com/forms/d/e/1FAIpQLScGVSQbvDiqGMoTQbOP4Fyj07rQ3c50i58cuNIy8rpY0QIa8A/viewform) for examples of what that might be.

Expand full comment

I edited my answer while Scott was answering (face-palm).

My first original question (I think?!), was how would the personalities be assessed?

Scott, straighten me out if this was wrong.

Expand full comment

Your definition of "major US political figure" in #24 would exclude Donald Trump (or Mike Pence). Did you mean to do that?

Expand full comment

No but it's not worth changing the question around for this now.

Expand full comment

How is the contest scored please?

Hugely excited to participate!

Edit: Sorry, I see the contest rules are deliberately ambiguous ("probably Briar score"), probably to prevent me from doing exactly the sort of gaming of the system I was planning :) However just to say that on my own forecasting tournament Briar and log -loss produce quite different results some years

Expand full comment

was the 2022 one not official?

Expand full comment

It was official for Eric and Sam, but I'm not going to do anything particular with the results, and I don't think other prediction markets officially participated (except sort of Manifold).

Expand full comment

didnt a lot of readers participate? will you be releasing how well calibrated we are?

Expand full comment

Yes, though most of this will be asking Eric and Sam to do the hard work and publishing what they find.

Expand full comment

Might I suggest giving a slightly below-average score for skipped questions? Maybe 40th percentile or so? As someone who'll probably skip some questions but hopefully not too many I don't want to feel like I'm incentivized to skip questions I'm less confident on.

Expand full comment

On the other hand, as someone with no vested interested in either US or UK politics and no intuition on how their political systems work, I skipped a lot of questions. I don't mind being penalised for it, honestly, I didn't participate because I think I'm good at this, I participated out of curiosity, but I'm not sure it ought to be penalised?

Expand full comment

Yeah, some questions I felt like I had no idea, and 5 minutes of research wouldn't help at all.

Expand full comment

I feel not predicting things should count against someone in a prediction competition. It seems reasonable that the person who wins the prediction competition oughtn't be someone who says "dunno" to a bunch of questions.

Expand full comment

I didn't skip, but I gave 50% as a "heck if I know" guess for questions I had no idea about and which would have taken longer than five minutes to properly research.

Expand full comment

I did the same thing. For some reason 50% seemed like a better solution than leaving it blank.

Expand full comment

Having the skipping be 50th percentile encourages people not to cheat. If it was 40th percentile, I figure some people may look up a market for a question rather than just skipping it

Expand full comment

Worth noting here that the market consensus would almost certainly outperform the average participant, so this incentive is still there.

Expand full comment

How do you plan to prevent the blind mode prizes from going to people who cheated by spending more than 5 minutes on research or looking at Manifold or Metaculus?

https://manifold.markets/IsaacKing/in-the-acx-2023-prediction-contest

Expand full comment

Honor system; I will be naming the names of winners and I don't think $500 is enough incentive to cheat. If I notice that someone's answers are exactly identical to the markets I'll take notice. And I think there are enough people, and the markets are imperfect enough, that even if someone tried this strategy it might not win.

Expand full comment

$500 may not be enough incentive, but being seen by the community as a good forecaster might be

Expand full comment

If they don't mind the stigma of being a dirty rotten cheater, then they're not good forecasters 😁

But this contest is not going to find good forecasters, it'll find people who are good at working out strategies to maximise their placings. So if they get into the Top Three, it won't matter if they really believe that (say) the Third World War is going to happen by February 2023, they'll only care if that is the answer that gets them a higher placing than the truth.

This is why I think prediction markets are nonsense to find out 'truth'. If there's a reward to be made, people will game answers to get the reward. If there's no reward, people are not going to bother participating (except for fun). Would you set national policy on housing/health care/foreign affairs by how a bunch of Wordle players decided it should go?

Expand full comment

Actual prediction markets or contests which give prizes in proportion to a proper scoring rule fix this, but your criticism on prizes like the one Scott is giving is correct.

Expand full comment

"Will any new country join NATO?"

Does this question include Sweden and Finland as "new countries" or no? They've signed accession protocols so their membership is imminent, but they aren't strictly speaking members of NATO yet.

Expand full comment

Yes.

Expand full comment

So, is this one factually resolved as yes, given that Finland has joined?

Expand full comment

For new countries joining NATO, I assume that's beyond Finland and Sweden? I'm pretty sure they're not formal members yet so it might be good to clarify if they count.

Expand full comment

F&Sweden it is about, mostly. Turkey and Hungary have not yet agreed/ratified. Hungary is expected to do in January 2023 - Turkey ... Erdogan is less predictable than Orhan even. Plus technicalities after ratification. - But, yeah, pretty sure it will happen in 2023. Any others? Ukraine? Much less likely, esp. as even Sweden was not the expected shoo-in.

Expand full comment
Comment deleted
Dec 17, 2022Edited
Comment deleted
Expand full comment

Indeed, thus both have to ratify the admission of FI+SW to NATO. All other NATO-members already did so, most within a few days, the US in 4 weeks, Slovakia after 8 weeks . https://www.nato-pa.int/content/finland-sweden-accession

Expand full comment
Comment deleted
Dec 18, 2022Edited
Comment deleted
Expand full comment

Actually, I am Mark. Not remark. And I do that kinda typo, too. ;) (Plus I can't ever resist making a bad joke.) Wishing you a new year full of peace and joy. May your holidays be full of warmth and cheer!

Expand full comment

7. Does this have to be an accident? As opposed to say an intentional strike, or sabotage? What about intentional mismanagement leading to a predictable but plausibly deniable "accident"? If accident just means nuclear related event leading to evacuation (due to the nuclear part as opposed to say some fighting/strikes in the area), I think my predictions are different.

Expand full comment

Good point, I have changed this to "issue". If there's a non-accidental issue I may disqualify this question.

Expand full comment

Small request on the survey, based on looking at last year's — I don't remember ever taking a formal IQ test, and I don't have my SAT score, but I still have my ACT score on hand. Can we use ACT as an entry in that section? (If not, should I just leave the entire intelligence section blank?)

Expand full comment

I'll figure this out when I do the survey; I think in the past there's been a section for ACTs.

Expand full comment

If you're collecting IQ test data, will you have a category for "we don't do official IQ tests round here but here is my Crappy Online Test result"? I have one, which is minorly crappy but based on a cut-down version of the Cattell Culture Free Intelligence Test, and the result feels intuitively within the ballpark of where I'd estimate my mathematical/pattern matching ability to lie.

Expand full comment

This should be fun! I entered your 2022 contest, and was inspired to host my own contest for friends that was derived from the questions you sent out + more fun/dumb cultural questions about sports, James Bond, etc to try to capture a category of friends who wouldn't typically try this kind of thing. Every few months I send out an email that provides a rundown of the questions that resolved, the questions in the news and the current leaderboard. I plan to do the same this year but make it more community-oriented by crowdsourcing at least a few questions from participants, maybe we'll have a t-shirt or something dumb like that. There isn't really a way to scale this without losing some specificity (it takes time and it's more fun amongst people who know each other), but I do think there's more buy-in knowing that it's a year-long thing, that your friends are doing it and that there's some minimal interaction with the content.

I wish there was some way to easily plug in the data I collect back into the data you all are collecting.

Expand full comment

Oh also, at least 10% of my friends messed up the scoring (submitted the opposite of what they thought) despite aggressive attempts at explaining the system.

Expand full comment

> For the forecasting questions, you just have to give a percent chance, from 1 - 99.

Since people are rounding to the nearest integer percentage, a response of 0 or 100 is not necessarily irrational, and I think you should allow it.

Expand full comment

The main reasons I did this were:

1. I don't know what scoring rule we'll use, and it might be one that assigns infinite negative penalty for wrong answers of zero. I think people would be angry if this happened unexpectedly and I don't want to explain it to them.

2. If I allow answers below 1, then someone is going to put in 0.5 to a question that seems about even odds, and I'm not going to know whether they meant it to be 50% or 0.5%. I would rather just make this failure mode impossible.

3. I tried to choose things that were at least somewhat uncertain. I think if you put below 1% or above 99%, you are wrong.

Expand full comment

I don't mind having expected score of negative infinity if it means I can improve my max score by ~1%. Since you only reward the best predictor, I am incentivized to optimize my chance of winning rather than my expected score.

Expand full comment

"I don't know what scoring rule we'll use ... "

Hmm. I thought competent experimental design requires that you make such a decision IN ADVANCE of the experiment.

Expand full comment

Theoretically, as long as Scott chooses one of the many strict proper scoring rules, people are still incentivized to guess their true probability.

This is complicated slightly by the fact that due to the skewed payout structure of the contest, you want to maximize some complicated combination of both variance and expected value. But really: most people here are not doing it for the money, so I doubt this matters too much.

Expand full comment

Some things are truly 0%. It's a mistake to impose your own forecasts on the participants rather than just letting the results come out naturally

Expand full comment

>3. I tried to choose things that were at least somewhat uncertain. I think if you put below 1% or above 99%, you are wrong.

Or they know something you don't?

Expand full comment

TBH none of those questions sound like anyone could reasonably answer < 2% or > 98% based on publicly available evidence alone.

Expand full comment

Fun guessing game but I expect to be completely wrong on everything.

Expand full comment

Perfect! Just answer the opposite of what you truly believe on every question and you'll get a great score.

Expand full comment

I had a great CS professor who gave tests in true/false format and said if you got <10% you'd get an automatic A!

Expand full comment

TW: Very Pedantic. The questions are written as questions, not statements, but the answers are probability percentages which (to my understanding) correspond with an affirmative statement. I find it a bit unintuitive but I assume this is by some convention I'm not aware of. The 2022 survey had questions in the form of a statement

E.g. Question 1. "Will A be B of C". The actual statement I am giving a probability for is instead "A will be B of C".

Last year's was fun, I took a look back and think I did alright. I was a bit too bullish on the GOP and the economy, but correctly bearish on Crypto

38. Applepay (Apple) already allows users to pay using USDC, a stablecoin. Would this resolve to true assuming it doesn't break the partnership?

Expand full comment

I believe Tesla also accepts payment in Dogecoin. I would expect the question to resolve to positive if this remains the case on the resolution date, but it would be nice to confirm.

Expand full comment

50 and arguably 47 already resolved as true in 2022. For 50, Imagen Video (https://imagen.research.google/video/) is currently the best, but Meta has a similar model. For 47, early in the Ukraine - Russia war, there was a deepfake of Zelenskyy calling for Ukranian soldiers to surrender, and I find it highly probable someone was upset by this, though I can't name a specific person.

Expand full comment

There is also this:

https://www.theguardian.com/world/2022/jun/25/european-leaders-deepfake-video-calls-mayor-of-kyiv-vitali-klitschko

> The mayors of several European capitals have been duped into holding video calls with a deepfake of their counterpart in Kyiv, Vitali Klitschko.

Does this count? But the mayors were only "harmed" in having egg on their face and being in a negative article about deepfakes.

Expand full comment

This is super cool. As a machine learning scientist I recommend cross entropy loss as a scoring mechanism. Cross entropy loss is a principled and mathematically pleasing way to evaluate probability estimates. Let x be the probability you assigned to the true eventual outcome. The cross entropy loss of this forecast is -log(x). So if you said x is 100% your loss is zero. If you said x is 0% your loss is infinite. (Perhaps why Scott didn't allow 0% or 100% predictions - if you're wrong then in theory no one would ever be able to trust you again.) Cross entropy loss is how all the best machine learning models are trained and it would be a great way to score this competition as well.

Expand full comment

That was fun. Kinda. 5 min research did a lot to form my answers - in the few cases, I did so. :)

... - deleted not to violate blind mode -

Expand full comment

Reading this comment technically violates Blind Mode. So

(1) Please don't comment on your specific predictions here;

(2) Scott, please create a thread for people to share comments once they've entered their blind mode predictions, since the impulse to do so is almost irresistible, and move the bulk of Mark's comment over there.

Expand full comment

I agree! Let's not give any answers here, and only ask for question clarification.

Expand full comment

It won`t take 5 min to read my info about the train station, I figure. So ... - but otoh: I see your point - and oblige.

Expand full comment

43. <i>Will a new version of COVID be substantially able to escape Omicron vaccines?

This question will resolve positive if in the opinion of the judges the scientific consensus is that getting all currently-recommended vaccines, including the two original vaccines and the Omicron booster, decreases risk of the new variant by less than 50%.</i>

This seems underspecified. What risk do you mean? The risk of being hospitalized? The risk of dying? The risk of contracting the virus? The risk of transmitting the virus? And do you mean absolute risk reduction or relative? If a new variant emerges that is far less lethal and results in far less hospitalizations, but is totally unaffected by the vaccine, did risk increase or decrease ?

Expand full comment

We probably shouldn't be nitpicking, but I agree, this one seems quite vague to me.

Because, depending on the criteria, Omicron already “escapes” the vaccines—CDC data doesn’t indicate robust sterilizing immunity in most cases. It also varies a whole lot over age ranges.

https://www.cdc.gov/mmwr/volumes/71/wr/mm7148e1.htm?s_cid=mm7148e1_w#T1_down

Expand full comment

To be fair, the injected polio vaccine doesn't produce sterilizing immunity either, but can't be said to be ineffective.

That aspect can pose a problem when population vaccination levels drop, if the virus is introduced and passed on asymptomatically till it reaches the unvaccinated. (But as long as vaccination levels stay high, it's safer to use.)

With Covid, the mRNA vaccines probably reduce transmission by 20-40%, but so far are most effective against hospitalization anf severe disease. Those last seems like the criteria to measure escape against at this point.

(There's work on next gen vaccines hoped to be better against transmission. But at current likely levels of funding and the vanishing unlikelihood of a second Warp Speed, even if they're possible they won't be seen in 2023.)

Expand full comment

Was about to ask the same. Usually newspapers give three different percentages about vaccines: risk reduction of

- infection

- severe disease (aka hospitalization)

- death

Which of the three is meant?

Expand full comment

I am also waiting for an answer to this question.

The recent variants (e.g. BQ1.1 and BXX) very likely dodge the omicron BA.5 vaccine in terms of symptomatic infection. It's plausible they dodge for hospitalization too. It would help if Scott said what he believes the scientific consensus to be right now -- what does the scientific consensus supposedly say about BQ1.1 and BXX?

Another complication: I view it as very likely that there will be updated boosters by 1/1/2024 (but they'd still be called "omicron booster"). Does, say, BXX count as dodging omicron boosters if it turns out to dodge the current BA5 boosters but not some future BXX booster?

Yet another complication: most people already had previous omicron infection. Are we measuring the efficacy of boosters in the previously-infected population or in the immunonaive population?

Basically I'm quite bearish about vaccine efficacy in the future, but I don't understand resolution criteria and I'm afraid it will resolve against me due to weird interpretations of the scientific consensus.

Expand full comment

The boosters have been doing reasonably well on ER visits, hospitalization, and death during the fall, which is a mix of mostly BA.5, BQ.1, and BQ1.1.

https://yourlocalepidemiologist.substack.com/p/fall-bivalent-boosters-science-update-0a9?r=96jx7&utm_medium=ios&utm_campaign=post

We're always going to be chasing data on the most recent variants, which we'll generally have only when they're on the wane. But a recent booster of any sort seems to be helpful and the closer the match, the better.

Expand full comment

The studies mentioned in this article are all predominantly about BA.5. It is true that one study covered Sept 2022 to mid Nov 2022, a period which had a bit of BQ1 and BQ1.1 at the end; but it did not do a separate analysis on those variants, and according to the study itself, BA.5 predominated.

So all those results are consistent with the vaccines only being effective against BA.5 and not against BQ1 at all. We just have no information about efficacy against BQ1, except for antibody studies which would imply a ton of immune evasion.

I do expect the vaccines to help, to be clear. But (1) not against infection, only against hospitalization/death, and (2) even there, the efficacy is likely modest, not like the 90% we had at the beginning (I would not be surprised at 50%, though it could also be higher, depending on time since last infect/booster).

(I still recommend getting all boosters. I got my omicron BA5 booster. Later I also got omicron (BA5 probably, given timing), but it was reasonably mild.)

Expand full comment

Also, is it the protection as measured soon after the booster or later? This can make a huge difference.

https://www.nejm.org/doi/full/10.1056/NEJMoa2119451

Expand full comment

Yep, this one gave me trouble. My answers are *very* different depending on whether 'risk'' = risk of infection or risk of death.

Expand full comment

Also, is the decrease in risk measured as compared to an (increasingly hypothetical) immunologically naive person? Compared to the average person among those who haven't got the omicron booster (but who may have got the original vaccine and/or one or more earlier bouts of covid)? Compared to the average unvaccinated person (who has probably caught covid at some point)?

Expand full comment

Wish some of this was specified better. Meta has an ar headset coming out too already. Does new Covid variant count as a new pandemic?

Expand full comment

This is a winner-takes-all game without punishment for bad guesses. Doesn’t this push us away from making our best-calibrated guess toward a more extreme higher-risk strategy?

Expand full comment

This is a potential problem! Maybe he should give everyone who does better than average a payout that is somehow proportional to their score?

Expand full comment

I've mulled it over a bit, and I think I should probably just play my best-calibrated guesses, given there are 50 questions. I dunno--maybe there are some interesting correlations in the nuclear->Russia/Putin/NATO questions that would benefit a high-risk strategy.

I'd be interested to hear your opinion.

Expand full comment

Am I supposed to try and win the contest or make the most honest/accurate predictions? They aren't the same thing.

Expand full comment

Oh try to win (IMHO)

Expand full comment

Pick the girl who everyone else thinks is the most beautiful.

Expand full comment

If you think you can win, try for that (this seems to be what most people here are trying).

If you think you have no chance of winning, try for accuracy.

If you don't think you're likely to be accurate, be honest (that's my strategy) 😀

Expand full comment

Sorry if this is a stupid question but why aren't they the same thing? (If you can answer in a way that doesn't give away your winning strategy)

Expand full comment

No.

In a contest with many players, going for more extreme less accurate predictions will have a better result. Say the real chance of each think happening is either 75 or 25. And you are perfect and guess the right answer every time. You look uncertain.

Meanwhile I guess 100 or 0. If there are lots of people playing, the winner is more likely to have pursued my strategy and been lucky, than your strategy. The more players there are the more "false certainty + luck" you likely need to win.

Expand full comment

Ok, so "I" would get a higher score *on average* than the people following your strategy, but what you're saying is that doesn't matter because it's approximately a winner-takes-all competition, and it's likely that somebody will have got enough lucky guesses to get a higher score than "me"?

This doesn't seem right, though, if there is a reasonably large number of questions. The probability of guessing every question right goes down exponentially as the number of questions increases, whereas the number of people adopting a guessing strategy only goes up linearly with the number of contestants. For 50 questions where you have a 75% chance of being right, the odds of getting them all right is 0.75^50, which is less than 10^-6. I accept that you don't need to get *every* question right under your system to beat hypothetical perfect me, but my intuition is that the exponential beats the linear.

(I also think this depends on the scoring system Scott decides to go with.)

Expand full comment

It does depend on the scoring system, but I think you need to be pretty cagey to have one that isn't going to reward exaggerating your confidence.

Yes there are a lot of outcomes, but you just need to perform the "best". If you guess 75/25 and I guess 100/0, and the real answer is 50/50 each time, despite your estimates being WILDLY better, it doesn't take too many of me before one of me beats you (once again depending on how exactly we score).

Expand full comment

Some pedantic nitpicking about questions:

Question 9 is very specific about nuclear weapons, then question 10 is exceptionally vague. It's not clear if an act of terrorism involving a dirty bomb would qualify for question 10 and this could have a significant effect on the answer. It's also not clear what "used" means in question 10. You could argue that technically, Russia is already "using" its nuclear weapons in war, since their existence and its threatening posture is limiting the Western response.

In question 49, an "average big tech company employee" probably works in Amazon warehouse fulfillment or Meta content moderation. This probably should be limited to software engineers.

Expand full comment

Question 19 (Will the Supreme Court rule against Affirmative Action) needs much more clarification, in my opinion.

The range of outcomes that might be considered to resolve to true range from: "It is illegal to overtly and explicitly admit someone on the basis of their race, but it is perfectly fine to do so as long as you don't say you are doing it," to "any evidence whatsoever of preference on the basis of race alone is actionable."

Expand full comment

I agree. Perhaps "explicitly overrule any part of Grutter v. Bollinger" would be sufficiently concrete and similar in spirit.

Expand full comment

Rather than going off a predefined list of predictions, it would be more fun to let people come up with their own unlikely predictions. You could then choose, let's say, 20 most interesting ones and put them up for a vote. The winner would be the person whose prediction was judged as least likely to occur by voters, while also being judged as "interesting" by at least 50% of voters and you.

Expand full comment

Can someone explain (or link me to) some potential ways to score something like this?

Expand full comment

Here are 2 common ways:

1. For each question, take the probability you assigned to the "wrong" answer and square it. Take the average of your scores on all questions. Best possible score is 0, worst possible score is 1.

2. For each question, take the log of the probability you assigned to the "wrong" answer (this log is negative). Add the scores of all individual questions. Best possible score is 0, worst possible score is -∞.

Expand full comment

I think the logarithmic rule involves the probability you assigned to the right answer.

Expand full comment

You're right.

Expand full comment

I basically guessed most of the non-political questions. If I somehow win, either prediction is nonsense or God decided to briefly speak through me.

Expand full comment

This probably won't be relevant, but by "midnight" do you mean 0000 hours on the specified day?

Expand full comment

Seems likely Scott means 00:00:00 on day X. If he didn't, the end date of range of dates describing "2023" would be a full 24 hours into 2024, which he probably doesn't mean.

For what it's worth, I noticed "midnight on blah date" in (Australian) government legislation recently and looked it up. It turns out that, in legal convention here at least, midnight on day X refers to "the last moment of day X prior to 00:00:00 the next day. So "midnight <today's date>" is in the future - not 12AM this morning.

This matches colloquial usage, which is good, but the maths student in me notes that, if moments are real numbers, there is no last moment of the day, and the supremum is the next day, so it's all a bit unsatisfying. The legal definition should just be something like "midnight day X is defined as 00:00:00 on day X+1". Grumble grumble.

Expand full comment

The legislature could satisfy even you by inserting "measureable" before "moment." That would also future-proof the law, since as timekeeping got better the "last measureable moment" could automatically creep closer and closer to 00:00:00 of the next day.

Expand full comment

For question 47: "The harm must come directly from the victim believing the deepfake..."

So, to clarify, this specifically means the VICTIM (person harmed) has to be the one believing the deepfake, not anyone else?

For instance, suppose Alice makes a deepfake video of Bob committing a crime (that Bob did not actually commit), shows this video to the police, the police arrest Bob, and Bob is convicted of the crime. Is it the case that this does not count as positive resolution, because the harm did not come from **Bob** believing the deepfake (it came from the police and jury believing the deepfake)?

Expand full comment

I suspect the aim is to say that “Bob” must actually be harmed and think he was harmed. Not that some third party is concerned for Bob on his behalf.

Expand full comment

Think of the Hustler trial. Falwell claimed (and may well have believed) that the allegation he committed incest with his mother was harmful and damaging to him, the decision was that it was not, because no reasonable person would believe this was other than a parody.

So Falwell believed himself to be harmed (even if simply on grounds of "people will be repeating this false story about me") but the Supreme Court did not:

https://en.wikipedia.org/wiki/Hustler_Magazine_v._Falwell

Expand full comment

Re question 13, on wars other than the Russia-Ukraine. If the Russia-Ukraine war escalates to WWIII and starts to involve many other participants, does it count as the other war or as the same war?

Expand full comment

Also do civil wars count?

Expand full comment

Oh this is fun, When can we talk about these questions, and resolutions... where the data comes from to decide. I feel that for some of the questions, the final result source might be more important than the question. I also feel like some of the questions need more details.

For question 14 I need units clarification. Is this 25 million for a year, or 25 million in one day, week.. forever or instantaneously?

Expand full comment

Decided to participate for the lolz. Answered most of the questions, but using my gut instinct and mostly binning into one of a few categories:

- 75 is "almost sure"

- 50 is "tossup"

- 30 is "probably not"

- 10 is "almost sure not"

Skipped the ones I don't have any background to answer even with a gut guess.

Meta-prediction--I'm unlikely to be in the top 10 forecasters. Maybe 10% chance?

Expand full comment

Given the number of likely participants, many of whom will have given thought to gaming their responses to avoid clumping with others (as I did not), I'd put my own chances at 1% by the "must be an integer" rules, and rather less than that in reality.

Expand full comment

The talk about gaming in the comments is interesting, since I think that demonstrates a failure mode for "prediction markets as ways of finding out the truth and setting policy by results".

If you take the Platonic ideal prediction market where people are *only* interested in finding out THE TRUE PROBABILITY, and everyone is studious and answers honestly, then fine.

But even with a (relatively) small prize on offer here, even if it's only for the bragging rights, people are still "Yeah, I was going to game my answers depending on how you score this". They're not looking for true probabilities or "will this event really happen?", they're trying to estimate how the bulk of others will answer and then what they need to do to make sure they place above them.

So it would be all the same if the questions were not "Will the Ukraine take back the Crimea?" but "Will we discover a herd of unicorns in Broceliande?" or "How many wishes will the genie of this lamp I bought in a quaint old curiosity shop grant me?"

Expand full comment

That failure mode applies only to this kind of 1) one-shot, 2) skewed returns (no possibility to lose money) setting. Long-lasting prediction market with real money shouldn't have these problems, right?

Expand full comment

Gaming for top placement still requires that you actually be pretty good at prediction. In particular, just using a naive variance-maxing strategy where you assign 1 and 99 to things you think are less and more likely than 50% likely is quite unlikely to work; you would need to be judicious about how much accuracy you wanted to trade for variance...

Expand full comment

I went much the same way - 50 is "maybe/maybe not" since I have no idea which way it's likely to go and both alternatives seem equally likely.

Anything 55-70 was "It could happen, I don't think it will but I'm definitely not sure it won't and there is a chance it might"

Above 70 was "Pretty good chance that will happen"

45 and below was "Pretty good chance it won't happen, even if there is a slim chance it might/it could". The lower on that end was the more "probably not/fairly sure it won't happen".

Expand full comment

(45) I may be misunderstanding, but is Google Glass not an example?

Expand full comment

Whenever I take a test, I want to know how I did.

So, please provide a method come 2023 of retrieving my submissions (presumably based on email address or bespoke ID code) so that I can compare them to the resolutions.

Expand full comment