200 Comments
Comment deleted
Mar 15, 2021
Comment deleted
Expand full comment

Did you read OBA? You might disagree with Matt Yglesias’s conclusions and prescriptions, but he’s certainly not a moron.

Expand full comment
Comment deleted
Mar 15, 2021
Comment deleted
Expand full comment

Have you read "Coming Apart" by C. Murray? You might like it.

Expand full comment

While I don’t have strong opinions on immigration, I do think you’re glossing over some of the hard parts of breaking the cycle of poverty.

First off, you suggest getting a good K-12 education. This in and of itself is a hard problem. Many predominantly slave-descended African American serving schools are not good educational institutions. And turning them into good institutions is both difficult on an individual level, as well as it’s hard to scale up. We could hypothetically just pay teachers tons more, but that probably wouldn’t increase the supply of good teachers, mainly because we don’t really know how to produce them.

Secondly, your results of K-12 education are heavily impacted by your home life. Generally if you come from a low income household, at the very least probably both your parents will be busy and tired from working their jobs. Obviously there are much worse cases, but I don’t know the statistics and would rather not resort to stereotyping. From my experience in a lower-income rural predominantly white school, at least a third of students had a home life that was detrimental to their education.

Finally, you’re ignoring the financial aspects of people dropping out of school. Some people can’t wait to start working until they graduate high school because they have to help support their families. Until we eliminate this cause, you’re still going to have low-income students forced out at higher rates.

Also, as a question: From my impressions of history, many of the ethnic groups you mentioned such as the Irish or Chinese immigrants were also poor, uneducated, and despised about on the same level as the black population. Why wouldn’t they just split the number of available jobs and all stay in relative poverty?

Expand full comment
Comment deleted
Mar 15, 2021
Comment deleted
Expand full comment

I agree with you on all of those points you just made. To note, I did not mean to imply slavery only exists in the US, I was only trying to use your terminology.

My core point was that you referred to the long term solution of better education as “easy.” I was raising the point that while it is important, it certainly does not appear to be easy.

Expand full comment
Comment deleted
Mar 16, 2021
Comment deleted
Expand full comment

Okay, makes sense. Thanks for clarifying your point.

Expand full comment

What if we paid parents money if their children did well in school (in hard-to-game ways?)

Expand full comment
Comment deleted
Mar 16, 2021
Comment deleted
Expand full comment

"Get C's or above" is too easy to game, once the subsidy-financed PTA figures out how to kick back some of that loot to the teachers who always give C's or above.

Expand full comment

I generally support paying student to go to school, to me it's a possible model to combat a lot of poverty-related schooling issues. I do think giving it to the students or the parents both cause serious risks (namely kids will waste money, but low quality parents will waste it too). I do worry that it would be an expensive solution though...

Expand full comment

You need customers to create jobs. One billion people is a lot of customers. And if you read Yglesias's book, you'll see he's just as welcoming of the ultimate uneducated immigrant, aka, American babies, as he is of child or adult immigrants from other parts of the world.

Expand full comment
Comment deleted
Mar 16, 2021
Comment deleted
Expand full comment

You can make the same point without adding that last sentence, which drags down the quality of your comments a lot.

Expand full comment

Well for the record, I struggled pretty hard to find any "quality" at all -- unless you're judging quality by the trumpian standards of which loudmouth at the end of the bar calls the most people stupid.

"Only I can fix it." Yeah, right.

Expand full comment
Comment deleted
Mar 16, 2021
Comment deleted
Expand full comment

Reminds me of a woman in the grocery store last spring when I asked that she pull up her mask from around her neck. She complied, confused, then turned back to announce, "I'm not even from around here!" (As if realizing for the first time that she was surrounded by big-city-liberal zombies.)

Say whatever you want, Amy (this place is certainly not predominantly big-city liberals), but you might get some shit for it if it amounts to little more than fist-pounding, along with (to my ears) a bullyish torrent of absolutist declarations.

BTW, does Joe Rogan still hang out with Alex Jones?

Expand full comment

'There are currently 330 million customers; however, the Black American unemployment is consistently around 11% to 16%. What's gonna happen when you have one billion customers? Do you think that the labor force stay the same number of workers, so everyone will be able to find a job, and the unemployment rate will go down to 1-2%?' To defend pro-immigration policies from the view that they'll lead to unemployment problems getting worse, you don't need to think that immigration will *lower* unemployment, but just that the rate of employment will remain the same.

Expand full comment
Comment deleted
Mar 15, 2021
Comment deleted
Expand full comment

I think there is also a clear division of personalities - traders/portfolio managers need to be comfortable taking risks, most researchers/analysts I met are not (so much)

Expand full comment

They, like you, downplay this and every other report as “not counting” and yet, eppur si muove...

Expand full comment

What's the argument that dormant Iraqi weapons programs were a reasonable and generally acknowledged causus belli?

Expand full comment

To repeatedly claim that no WMD were ever found in Iraq and thus all allegations were false is itself false. Additionally, components for the production of WMD were found, and there is strong circumstantial evidence that stockpiled Iraqi WMDs were evacuated prior to the invasion. That doesn’t change the fact that major intel agencies also had bad information, but it doesn’t support the imaginary narrative that Scott promulgates over and over again that no WMD were ever found and it was all a lie/false pretext.

Expand full comment

Since the causus belli was "Saddam had an active WMD program." And the reality was "Saddam had no active WMD program." It seems "Saddam has no WMD" is a reasonable shorthand for a true claim, though I suppose it's always better spell things out to avoid pedants.

Expand full comment

It has been a long time since I’ve thought about this issue but I thought the rationale was that the first Iraq war had been suspended on certain conditions, including particular inspection-related obligations. I think it was more or less undisputed that Iraq had violated those obligations by refusing access to inspectors. If that’s right then the rationale was that the conditions on which the suspension of hostilities had been predicated were no longer operative. Then the argument would be that the impact of uncertainty regarding current WMD activity under conditions where inspections obligations had been violated should not fall on the non-breaching parties. And intelligence assessments about ongoing activities were more related to the urgency of action rather than the justification for taking it. But as I say, this is all from faded memory. I understood the operative resolutions and agreements at the time but would have to re-educate myself if I wanted to remember the detailed arguments.

Expand full comment

The legality of the Iraq war has its own wikipedia article, which I think is a reasonable tool for figuring out the general consensus on reality.

https://en.wikipedia.org/wiki/Legality_of_the_Iraq_War

Your argument seems to also be mentioned in a related wikipedia page more concisely, which I'll quote from: https://en.wikipedia.org/wiki/Iraq_and_weapons_of_mass_destruction#Legal_justification

> Most member governments of the United Nations Security Council made clear that after resolution 1441 there still was no authorization for the use of force. Indeed, at the time 1441 was passed, both the U.S. and UK representatives stated explicitly that 1441 contained no provision for military action

So yes, it was determined that they were in breach of inspection obligations, but that alone was not enough to "resume hostilities". The previous conflict was not put on pause, but rather ended and those conditions were given, but not under penalty of the war restarting.

That's what I gleam from the wikipedia article at least, which I expect to be a little less correct than an expert on the subject, and much more trustworthy/correct than an internet commenter with no specific relevant credentials

Expand full comment

There were weapons inspectors on the ground in Iraq at the start of the second Iraq war. These inspectors had to be airlifted out as the invasion started.

Expand full comment

The casus belli had a few parts.

1) Saddam was working towards WMDs, and would be a serious risk if he ever got them.

2) Saddam was in violation of various UN mandates, sanctions regimes, etc., which meant there was a legal casus belli under international law.

3) Saddam was a human rights disaster, and sufficiently evil that his removal was justified on grounds of "good riddance"(to paraphrase).

Side note: #1 isn't the same as saying that he *already had* massive stockpiles. Heck, by the de facto settlement with North Korea, we can see that a big WMD stockpile would have meant that he'd have been safe from invasion. Clearly, nobody thought he had functioning nukes yet, they just worried that he was trying to acquire them in the future.

#3 was very obviously true, and not disputed by anyone serious - the only argument there is if it's sufficient to be a casus belli. #2 was almost as obvious at the time - even if he wasn't really trying to build WMDs, he was trying to create strategic ambiguity around his programs to intimidate rivals, and that ambiguity was created by dodging inspectors rather frequently.

#1 is the one that's in dispute. From what we've seen since the invasion, it's clear that he had no significant stocks of functioning WMDs, just a few leftovers in forgotten bunkers. There were at least a few attempts to maintain the capability to develop them in future if the sanctions ever lifted, but from what I can tell, they weren't terribly effective attempts. Burying centrifuges in the backyards of your nuclear physicists is at least worthy of mention, but you aren't going to want to rely on them still spinning years later.

I supported the war at the time, and for some years thereafter, though not any more(tl;dr, I had way too much faith in the ability of the US to stand up a functioning Iraqi state and society once the war was over). And from what I can tell, I think that there was enough truth in #1 to be within the margin of confirmation bias(i.e. that it was honestly believed by decision-makers), but there's not enough truth that it really stands up as a casus belli with 20/20 hindsight. It's only kinda true, and that's not enough to start a war over. And if that's not true enough for a war, then #2 is a pretty weak reed of a casus belli too.

#3 is still entirely apt, and watching that bastard hang was the single best moment of the whole war. But seeing Saddam Hussein in a noose doesn't justify a war all by itself. It was good, but not *that* good.

Expand full comment

I'm a little sad. I thought Bush in Iraq was a good non-controversial example with which to explain the overall point. I clearly failed.

It's hard to explain a concept without using any examples, but then the examples somehow always eclipse the point...

Expand full comment

If #3 were adequate as a casus belli, then when are the bombs going to start raining down on Saudi Arabia?

Expand full comment

A casus belli is a justification for war.

But having a justification for something does not mean you actually *want* to do that thing. I may be justified in firing an employee who screws up, that doesn't mean I will actually do so if they are an otherwise good employee. I may be justified in breaking off a friendship with someone who insults me, but no one would find it strange if I forgave them instead.

Ukraine has a crystal clear casus belli against Russia. What could be a better justification for war then "Well they actually invaded us and stole half our country". They still didn't go to war with Russia. It needs no explaining why.

By the way, #3 is generally not seen as sufficient for a casus belli. Whether it should be is a question of philosophy. For myself, I'll just say that it would be nice to live in a world most countries were well behaved enough that it could be used as one.

Expand full comment

I am aware of what a casus belli is. My point is that if being a "human rights disaster" were sufficient as a casus belli, then we have chose not to make war on by far the worst human rights violator in the region.

Expand full comment

The other problem with #3 is that a single Big Bad Evil Guy firmly in charge might be preferable to a power vacuum and lots of Smaller Bad Evil Guys all fighting each other. See: Libya after Qaddafi fell.

Expand full comment

A *dormant* Iraqi chemical weapons program would have been a poor cause for war. An *active* Iraqi chemical weapons program would have been a direct and explicit violation of the agreement that ended the *last* war. Deliberately violating peace treaties is generally considered a legitimate reason to go to war, and if the term of the peace treaty that one is violating is the "no forbidden superweapons" part then it is reasonable to believe they have designs involving the use of those weapons.

Iraq didn't have an active chemical weapons program in 2003, but the leaders on both sides were doing their best to convince everyone that they did, so we got a war.

Expand full comment

I thought the U.S. administration was trying to convince everyone that Iraq had an active nuclear weapons program — not that they had made them yet but were working on them. We knew they had had chemical weapons, although not whether they were still producing them. "Weapons of mass destruction" is conveniently ambiguous between the two.

Expand full comment

The article explicitly states that these weapons were old, made before 1991, and not from a then-active program of building them. Yes, if this were a prediction market and it were phrased as "chemical weapons will be found in Iraq", it would pay out, but the issue under contention was always whether there was an active program of building these (because that was the claimed casus belli), not whether there might be old ones lying around. So, you know, hopefully such a prediction market would have *not* phrased it in such a way!

Expand full comment

I 100% agree about hoping a prediction market would structure the proposition differently, but what I am taking issue with is Scott and so many others’ retcon that there were no WMD or evidence of their production or stockpile found after the invasion of Iraq. Even CNN dissembles here and tries to turn it around on the US. If we were foolish enough to hand them to the Hussein regime, make the attack about that, but to paint it as “NBD these chems don’t count because we say so” is disingenuously avoiding the issue.

Expand full comment

> I am taking issue with is Scott and so many others’ retcon that there were no WMD

This is ridiculous. The "retcon" is that the existence of *any* WMDs was relevant to the discussion at hand. Scott's reference to this issue is criticisms of a pundit class that justified a war on WMDs. Everyone who lived through that period knows, in general, what he's referring to even if the specifics of "active WMD program" vs. "decades-old stockpiles of WMDs" eludes them.

Your criticism amounts to, "You should've spent more words elaborating in order to be technically correct, even though everyone already knew what you meant and the technicalities that I'm upset about are irrelevant to your overall point." Which, fair enough if that's your thing, but calling it a "retcon" is absurd.

Expand full comment

These are leftovers from the Iraq/Iran war. Not some new weapons program.

Sometimes Belgian farmers unearth 100 year old mustard gas shells with their ploughs (Google "iron harvest"). Are we going to war with Belgium next?

Expand full comment

I feel like we’d have to take a number after the handful of Euro countries that haven’t gone to war with Belgium, yet. Snark aside, there was and still remains evidence that the Hussein regime didn’t destroy everything and had likely dual-use facilities available (since many chemical weapons are small steps off of widely used agricultural chemicals.) https://www.pbs.org/wgbh/pages/frontline/shows/gunning/etc/arsenal.html <— VX doesn’t “keep” forever after mixing, but it’s precursors are relatively stable and can be handled surprisingly safely to transport. If I could put a small sum on “predictions about future declassification” I would predict we will find out that there were indeed CW stockpiles, but they were removed to Syria (in 2003 much less solidly aligned with Iran) prior to US invasion. Even a grossly incompetent Republican Guard could move some trucks into a friendly territory.

Expand full comment

Any day now, the truth will come out! All we need is some evidence!

Expand full comment

On top of what everyone else mentioned, it's worth pointing out that the term "WMDs" itself is a (probably deliberate) equivocation that conflates small-scale chemical weapons with nuclear bombs. The Bush Administration clearly wanted the public to be thinking in terms of nuclear weapons, using "mushroom cloud" imagery and talking about Iraq looking to buy uranium ore from Africa.

Given all that, I'm a lot more comfortable rounding off to "WMD was a lie" even if technically there were some old chemical warheads lying around.

Expand full comment

The only WMDs that mattered were nuclear. Chem and bio are in a completely different league.

Expand full comment

Chemical, sure. I'm not sure about bio, though. More people have died of COVID-19 than died from the two atomic bombs dropped on Japan. How many people could someone kill by infecting themselves with smallpox and hanging around in airports?

Expand full comment

A few thousand. Millions over a year or two if you were really lucky, and if your opposition was really incompetent. COVID has killed barely half a million people in the USA.

You could kill millions in forty-five minutes with only a dozen primitive atomic bombs and ICBMs. With 1960s tech and an economy the size of modern Iraq (which had a higher GDP per capita than 1950s America, btw), you could probably kill a hundred million Americans in 45 minutes after a twenty-year program.

Expand full comment

Covid-19 has about a 1% fataility rate at most; smallpox had a 30% one. A hypothetical virus as contagious as SARS-COV-2 and as lethal as smallpox, with no pre-existing vaccine, would have killed 15 million Americans instead of 500,000 and 60 million worldwide. HIV has killed approximately 30 million people worldwide so far. Estimated deaths from the 1918 flu pandemic are 50 million worldwide. Potential deaths from bioterrorism don't seem to be that different from nuclear terrorism.

Expand full comment

Another point of comparison: a simulation of a massive nuclear attack by Israel against Iran resulted in about 20 million fatalities.

Source: https://conflictandhealth.biomedcentral.com/articles/10.1186/1752-1505-7-10/tables/4

Expand full comment

Only in technothrillers and conspiracy theories does "biological warfare" normally mean releasing an actively infectious disease and hoping it infects only the enemy. That's the biological-warfare equivalent of the cobalt bomb; nasty and theoretically possible but nobody actually does it because it's too easy to score an own goal. Actual biological warfare programs are almost always targeted at diseases whose vector can be more directly controlled, e.g. Anthrax whose spores can be cultivated en masse in your production facility but whose victims are unlikely to go on and infect other victims. So, functionally equivalent to chemical weapons but with a lower LD50 and a longer time of action.

Expand full comment

Agreed; most actually existing biological warfare programs weren't trying to make "If I go down I'm taking you with me" doomsday devices to deter attacks. But they could have.

Expand full comment

Prior to Operation Iraqi Freedom, physicist Richard Muller argued that the findings of inspections after the first Gulf War made it impossible to trust supposed negative evidence from UN inspector teams. They had completely missed the main Hussein nuclear program using Calutrons. He predicted an invasion was inevitable.

https://www.technologyreview.com/2002/02/07/235255/springtime-taxes-and-the-attack-on-iraq/

Expand full comment

Thanks !

Also maybe explains why the US dropped not just one but two nuclear bombs : to compare their effects!

Expand full comment

0% implied chance of UER between 4 and 5% feels disqualifying to me!

Also worth noting that some of the markets (e.g. inflation, sports) listed have deeper and better existing markets than prediction markets (TIPS and sports betting, namely).

Expand full comment

I think you misread; the implied chance of UER between 4% and 5% is >=60%.

Expand full comment

Whupps, you are correct. I hereby disqualify myself!

Expand full comment

On his own Substack, the misreadings of this particular pair of predictions generated some very large comment threads.

Expand full comment

Slight correction: That should say 60%, not >=60%. I mean, it is >=60%, but that's being under-specific. For some reason I thought we couldn't say more than that, but we absolutely can.

Expand full comment

In response to "we should cherish people who are often extremely wrong but occasionally right when nobody else is; these people would lose a lot of money", I think it would be easy to make the case that they would be quite profitable (or perhaps at least break-even), as the payoff for their successes would be significantly more than their losses (given that everyone disagrees with them, they would theoretically be able to find extremely favorable odds to bet on), so even if they are right only 5% of the time, that could easily be >=20x payout for when they are (and indeed there are quite a few investing strategies, both in prediction markets and traditional markets, that do just this: make small losses for months on end until some improbable event occurs, then making it all back plus some. In many cases this is basically what hedging is as well).

Expand full comment

That’s what the “overconfident” bid is; if the market thinks something is 85% likely and you think it’s 80% likely, trade at 80% even if it means buying the 20% against.

If you’re right, 80% of the time you’ll lose money on the proposition.

Expand full comment

I believe this is exactly how Nassim Taleb got rich.

Expand full comment

I was gonna say the same thing: that's the old Taleb strategy.

Expand full comment

This is what I was thinking when I was reading the original post. Indeed, which is the more important signaling mechanism here? Would we rather get a little more certainty about an event everyone already agrees is likely, or get some insight into an event that nobody thinks is likely?

Expand full comment

In fact, I'd argue this hypothetical person MUST be able to make money on some hypothetical prediction market, or else they aren't actually adding any value and shouldn't be cherished. (A random number generator is occasionally right when nobody else is. This does not make listening to it valuable.)

Scott's example (from the older SSC post he linked) of a magic box that generates elegant scientific hypotheses that are sometimes right but sometimes wrong is ONLY valuable as long as it's right *more often than chance*. A box that generates ALL hypotheses is just the Library of Babel ( https://en.wikipedia.org/wiki/The_Library_of_Babel ) and its information-theoretical payload would be zero.

If you are right *more often than chance* about things that no one else is right about, then you should be able to make money on a prediction market that robustly covers those topics. (Of course, there's no guarantee that such a prediction market actually exists. But you'd have at least a hypothetical money-maker.)

Expand full comment

There are plenty of ways of adding value that don't have any direct cash value in terms of predictions. The classic example from the history and philosophy of science is Copernicus - the Copernican model of the planets (including the Earth) moving in circles (with epicycles) around the Sun was *worse* for predicting the positions of the planets in the sky than the Ptolemaic model where the planets and Sun move in circles (with epicycles) around the Earth. The big thing Copernicus had going for him at the time was that he could explain why the major planets (Mars, Jupiter, Saturn) all go into retrograde precisely when they were in opposition to the sun (because that's when the Earth's orbit pulls ahead of the planet's own orbit) while the Ptolemaic system just needed one specific epicycle for each, that happened to be perfectly aligned with the period of the sun's orbit. And of course, about 70 years later, it turned out that Copernicus's idea of putting the sun at the center was extremely helpful when Kepler replaced the circles-plus-epicycles with ellipses, and finally got better predictions than the Ptolemaic model.

Saying that we shouldn't cherish someone until they are already able to make money in their predictions is saying we shouldn't cherish Copernicus.

Expand full comment

I don't think you are thinking of "predictions" in the same way as me (or as Scott, judging from his example of the hypothesis-generating box).

If you are claiming to add value to the discussion by proposing that the planets orbit the sun, then I would say that the thing you are predicting is "the planets orbit the sun". If you were right about THAT, then that represents a winning bet in some hypothetical prediction market, whether or not you can predict the planets' exact positions. (Just like a correct prediction that "it will rain on Thursday" could win money without predicting exactly where every raindrop will fall.)

If you can say novel things like "the planets orbit the sun" and be right more often than chance, then there is some hypothetical prediction market where you could make money (on average) based off that talent.

(The practical difficulties in creating and running that prediction market might be substantial; I am not suggesting that we *actually* use prediction markets for *everything*. My point is just that your average accuracy needs to outperform a uniform credence over all possible hypotheses, or else you are contributing no more than a random number generator.)

Expand full comment

Right - I'm thinking of predictions as operationalizable things that someone can make a finite-time bet on, which I think is essential to the basic motivation for prediction markets.

Expand full comment

That 20x is slightly reminescent of the https://xkcd.com/882/ spoof of p-hacking (visavis the "traditional" 95%-confidence threshold).

From the perspective of one bidder, if they made many "contrarian" bids at odds of 1/20, and came to win only 1 bid for every 21 such bids they'd made, then that one win may be nice but their payout still doesn't cover it.

From the perspective of those who skim the vast sea of bidders in order to identify those bidders that are making various 1/20 bids *and* are winning them more often than one time out of twenty - odds are that they *will* succeed to find such bidders. But what will be the basis for asserting they did not simply find "green jelly beans" in the vast sea of bidders?

Expand full comment

Well, I think if you scan to find those with a historical record of beating the odds, and make the assumption that some of these are 'green jelly beans' who got lucky and some have some genuine additional information to contribute... you then have to have an observation period. Over the observation period, the lucky jelly beans will lose their good track records and the actually-useful-contrarians will mostly maintain their track records. The longer you observe before trusting/betting on the apparent contrarians, the more lucky jelly beans will fall out and the purer your sample will be. You will, by misfortune, lose a few useful contrarians from the batch too, but a large enough sample to start with should tolerate this effect ok.

Expand full comment

This all assumes that people are static. Someone might've been genuinely good at picking out contrarian takes ten years ago and then lost their skill.

Expand full comment

Not sure if you've seen this before, but for good measure: TipRanks analyzes stock analysts recommendations. https://www.tipranks.com/

Expand full comment

The Brier score, Brier skill score, and concept of climatologic forecasts having accuracy but no skill are all relevant here.

In essence: suppose you know that it rains 1/5 days in a given place. Then, in the absence of any other information, you should predict 20% chance of rain for a given date. This forecast will be completely accurate but not skillful. Accuracy here is defined as: of all your 20% forecasts, how many days had rain? If it's 20%, you were perfectly accurate. But since the climate in this case is know, your skill as a forecaster depends on how accurately you deviate from the climatologic prediction. (There's also power, which is how close your forecasts are to 0 or 1.)

EU Met Cal used to have a spectacularly good set of articles explaining all of this (at eg https://www.eumetcal.org/resources/ukmeteocal/temp/msgcal/www/english/msg/ver_prob_forec/uos2/), but it seems to be off the web with no Archive. If I can find a working link, I will post.

Expand full comment

To elaborate: use of the appropriate scoring tool avoids the sunrise-prediction method of running up one's score. The climate value would simply be set at 100%, making the prediction that the sun will rise tomorrow accurate, powerful, but utterly unskillful.

Expand full comment

How do you establish the "climate values" for some arbitrary prediction like "Joe Biden ends the year with his approval rating above 50%"?

Expand full comment

Whatever the historical average approval rating is for all presidents at the end of their first year for as long as it's been measured.

Expand full comment

How are you choosing the reference class? Why just presidents, rather than all politicians? Are you limiting it to US presidents? Why not limit it only to Democrat presidents, or only to presidents in a certain age range?

Why base things on the *absolute* approval rating rather than the *change* in approval rating compared to the start of their term? The prediction "Joe Biden's approval rating will increase by at least X" would have been logically equivalent, for some value of X (possibly negative)--would the baseline "climate value" have been different for that prediction?

If the historical data has an obvious trend over time, are we supposed to take that into account? If Joe Biden has a history of higher approval ratings than his average peer, are we supposed to take that into account?

What about predictions with no obvious reference class at all, like "Apple releases new iMacs powered by Apple silicon" or "The EU ends the year with more confirmed Covid-19 deaths than the US"?

Expand full comment

If there's no historical referent, probably impossible to use a weather/climate analogy.

On the others, weather prediction faces similar barrier questions, they're just a bit easier to answer. Why choose to ask if it will rain in a 24 hr period, and not over 2 days? Or a week? Why ask a yes/no rain question instead of >0.25" or >0.5"? What geographical boundaries to draw around the dataset - should we look only at that single weather station, or include stations within 10 miles? 50 miles? 1000 miles?

Some of these questions are answered by forecast utility (we care if it rains on a given day because days are meaningful to how we go about our lives) and some on prediction utility.

At the end of the day, the goal is to have a baseline to score skill against. So if there's a basic flaw in the climate value (like a rising baseline over time), savvy forecasters will observe that and it will show as skill in their scores. This even happens with weather - predicting climate value +0.1 degree for daily temperature each day in 2022 is likely to have some positive skill for most of the globe!

Expand full comment

Your questions don't seem similar to mine. Yours are about what sort of prediction we ought to make; mine are about how you determine the baseline "climate value" for a *given* prediction.

I'm not asking whether someone ought to predict Joe Biden's approval rating after 1 year or 2 years or 6 months, I'm asking GIVEN that the prediction was for 1 year, how do we get the baseline "zero skill" prediction that we're scoring it against?

> "So if there's a basic flaw in the climate value (like a rising baseline over time), savvy forecasters will observe that and it will show as skill in their scores"

That's exactly the problem. If your goal is to prevent people from running up their scores by making easy predictions like "the sun will rise tomorrow", then you need to somehow discount ALL easy predictions, not just a few of them. If there's a basic flaw in your "climate value", then your forecasters can exploit it to score free points. The top-scoring predictors will just be whoever made the largest number of predictions that exploited that flaw, rather than the ones with the greatest actual insight.

Expand full comment

Do any of those other things then. I wasn't coming up with some great thought out proposal for how to estimate a base rate, just pointing out that any event with precedent has a base rate. These aren't time-invariant processes, so we're always stuck with a flawed estimate, but it's something. Weather isn't time-invariant, either.

Expand full comment

Harder to do than with weather.

One answer the same: set boundaries on a historical record. Adam's answer is where I'd start: what was the approval rating for all presidents at the end of the first year of their term. Setting the boundaries is where it gets difficult. First terms only, or count year after reelection too? How far back to go?

Alternative: use the prediction market! Let the market make a prediction through buying and selling for 1-2 months. Then close bets, let pundits see the market value, and let the market value be the climate. Skill for the pundit is when they differ from the market.

Expand full comment

I'm confused. The Brier score is just the sum of the squares of the errors of your predictions. But what you seem to be describing is more related to a "calibration" score, which *doesn't* incentivize people to make more extreme predictions the way the Brier score does (except in the parenthetical about "power"). I'd definitely be interested in learning more if you have it, but I'm used to both the Brier (quadratic) scoring rule and the logarithmic scoring rule.

I'm not aware of a scoring rule that has a way to compare people who make predictions on different sets of claims, or a way to "objectively" measure the "difficulty" of claims, so that we can give more points to the people that are closer on the "harder" ones.

Expand full comment

The Brier skill score is one way to compare the accuracy of predictions. It's essentially a difference in Brier scores between two models.

Let's use the approval rating question. Over the last 20 years, Presidents have had a mean of about 48% - I'm going to assume that this maps to the median, though I haven't checked the actual numbers. Let's posit that the distribution of answers to the question "Is the President's approval rating greater or lower than 50% on a given day?" is 55% no, 45% yes. So our climate prediction is 45%.

Now, let's say the outcome is No. The Brier score for the climatic prediction is (0.45-0)^2=0.2025.

If I predicted 45% yes, my Brier score for this prediction is the same (0.2025), and my Brier skill score is *0* (0.2025-0.2025)/0.2025 - I predicted the same thing the climate did, so I demonstrated no skill.

If I predicted 10% yes, my Brier score is (0.1-0)^2=0.81. My Brier skill score is now (0.2025-0.01)/0.2025 = *0.95*. Because my prediction was better than the climate, I get a good Brier skill score.

If I predicted 70% yes, my Brier score is (0.7-0)^2=0.49 (NB: Better Brier scores are smaller.) My Brier skill score is (0.2025-0.49)/0.2025 = -1.42. Because my prediction was better than the climate, I get a bad Brier score.

Brier skill scores can be combined for sets of predictions. Predictions that add no information to the baseline (the sun will come up tomorrow) generate skill scores of 0, so do not move the needle either way. Predictions the wrong way vs. baseline reduce Brier skill scores. Predictions the right way increase them.

Setting baseline can be tough, but it's a problem similar (identical?) to that faced by sports bookmakers in setting spreads.

The Brier skill score is far from perfect and it's also not the only option for measuring forecast skill. I mention it to illustrate that measures of skill exist, and that they can address at least the simple concerns raised by Mr. Alexander. The broader point is that measuring forecast skill to avoid the obvious pitfalls of juking the stats is a problem considered in great depth by meteorology, so those who want to measure pundit performance on prediction would do well to start with weather.

Expand full comment

Dang, last "better" was supposed to be "worse" in the 70% yes case. No excuse. "Because my prediction was *WORSE* than the climate, I get a bad Brier score."

Expand full comment

I think this solves the issue of comparing forecasters of Phoenix weather against forecasters of Dallas weather against forecasters of New York weather, and forecasters of presidential approval rating against forecasters of prime ministerial approval rating. But I don't think it solves the bigger problem of comparing forecasters of economic variables against forecasters of political variables, since there are easy and hard variables within each, and many of them are one-off events that don't have natural reference classes.

Expand full comment

I agree that comparisons across types of predicted events is not addressed with forecast skill metrics. I think it may be impossible to compare an economist's skill at forecasting inflation with a political scientist's skill at forecasting approval rating.

Brier skill scores are also not comparable for different predictions even in the same category. Because the climate prediction is in the denominator, the values are heavily influenced by the baseline rarity of events: for rare events, the climate prediction is more accurate (as measured by Brier score) and it is much harder to show skill.

I'm also concerned about the sample sizes: meteorological forecasting metrics usually involve prediction of at least hundreds of events (say, temperature every day for a year). A single prediction, or small set of them, will be hard to evaluate for skill.

Expand full comment

My half finished prediction market awards extra points for being right when others were wrong and also being first to be right - that was my attempt to adjust for everything anyway.

In light of ever the above article - what do you (meaning fellow commenters) think of the relevance of those two criteria?

Expand full comment

A real-money prediction market like PredictIt accounts for those criteria implicitly. Why make a more complicated system that tries to replicate that effect?

Expand full comment

Letting the prediction market see the pundit is actually a fatal flaw. Those pundits that who the market accurately assesses the reliability of will make an efficient market better at predicting than they are by sharing their position and justification.

It’s one thing to be able to beat the market; it’s another thing entirely to be able to consistently beat the market while also sharing your reasoning and bets.

Maybe you could compare the prediction market before and after a post by a pundit who makes a prediction that differs from the market, to see which direction it updates, and then evaluate whether it more often updated in the direction that it finally evaluated, in conjunction with whether the market adjusted towards or away from the pundit’s estimate.

If the market updates away from the pundit consistently and improves accuracy that way, then they are providing useful analyses but are bad at prediction. If the market updates in a way that reduces accuracy, then the analyses are of negative value, even if the prediction is more accurate than the market.

But there is no way to hold all other things equal.

Expand full comment

Good point, maybe the steps for a conscientious pundit should be:

1. Make prediction.

2. Check to see if there are existing predictions like it. If there are none create a new bet on the prediction.

3. Compare your prediction with the public prediction. If its the same, no new info is really added so skip writing a post.

4. If it is different write up your post explaining why you think it is different.

5. At the end of the year tally up your differences from public market and personal predictions.

Expand full comment

You could publish a commitment to your predictions, too. For a simple one, take a random number of 256 bits, prepend it to a text file of your prediction, hash the whole thing with SHA256, and publish the hash. (If you want this to hide your prediction, you have to use a really random number, not just something you make up off the top of your head or get from a non-cryptographic RNG.)

Expand full comment

To open the commitment, reveal your random number and text file.

Expand full comment

Zeynep isn't wrong often. I'd love to see her vs. an aggregate punditry. Have you talked to the Good Judgment folks about running something like this for pundits that were willing to jump in? Have a parallel version for Superforecasters. Keep track of results.

Expand full comment

Most of what I read comes from the investing community. On that note, that's where my income comes from too.

Currently, my largest investment is in AT&T. The investment thesis here runs along the line of-

HBO Max will do well.

Even without HBO Max AT&T is fine.

What I want to point out here, is that the bet is two sided.

That is, I'm not saying [HBO good: 70%]. In fact, I have no specific number. It's more like [HBO good > Debt dangerous: >50%]. This same logic can be easily applied to politicians, media, and bureaucrats as well. So, Bush said Saddam had WMD, and that we should go to war to solve this.

The two sides of this prediction are:

Iraq war will prevent the nuclear annihilation of New York.

Iraq war will be quick and easy.

If the first half had been proven true, I would have given Bush lots of points for that, sure. But if only the second half had been proven true, I still would have given Bush lots of points. If Saddam did have WMD that he did plan to use on New York, but our military was crushed in a failed invasion (perhaps because of help from, China?) I would have penalized Bush by a ton.

Of course, both halves proved false, and I despise Bush, for that and other reasons, but, anyway, the point is, most policies have an implicit [potential cost]:[potential return]. In investing, it's not actually important to be right about anything. What matters is basically, to not be wrong. At least, the investors I tend to look to for advice, make lots of investments with the potential for profits, and little in the way of potential losses. I think the same can be asked of people like Trump, Fauci, and Tucker Carlson.

Expand full comment

https://metaculusextras.com/top_comments?start_date=2021-03-01&end_date=2021-03-08

Week before last's top comments:

- misha rounds up different forecasting platforms takes on the Olympics (https://www.metaculus.com/questions/5555/rescheduled-2020-olympics/?invite=tLxPdB#comment-56906)

- There were no terrorist attacks in OEDC founder states between the 3-Nov-20 and 1-Mar-21. (But Jgalt points out there were attacks on either side) (https://www.metaculus.com/questions/5441/terrorist-attack-in-oecd-founder-state/?invite=tLxPdB#comment-57003)

- F for SN10 (Jglat) (https://www.metaculus.com/questions/6498/will-sn10-land/?invite=tLxPdB#comment-57038)

- ege_erdil and I look at the distribution for bitcoin's peak in 2021 (https://www.metaculus.com/questions/6666/maximum-price-of-bitcoin-in-2021/?invite=tLxPdB#comment-56828, https://www.metaculus.com/questions/6666/maximum-price-of-bitcoin-in-2021/?invite=tLxPdB#comment-56829)

- ege_erdil calculates the base rate for resignations for politicians resigning after accusations of misconduct (https://www.metaculus.com/questions/6693/will-ny-governor-andrew-cuomo-resign-soon/?invite=tLxPdB#comment-57116)

https://metaculusextras.com/top_comments?start_date=2021-03-07&page=1

Last week's top comments:

- ThirdEyeOpen's application to be a moderator (https://www.metaculus.com/questions/6805/2021-spring-equinox-moderator-election/?invite=9I6hgw#comment-57728)

- Various meta-points on resolution (https://www.metaculus.com/questions/353/will-someone-born-before-2001-live-to-be-150/?invite=9I6hgw#comment-57513, https://www.metaculus.com/questions/6145/brent-crude-oil-to-exceed-70-in-2021/?invite=9I6hgw#comment-57508, https://www.metaculus.com/questions/5555/rescheduled-2020-olympics/?invite=9I6hgw#comment-57601)

- chudetz shares his docuseries on the BEAR question (https://www.metaculus.com/questions/6087/when-will-ai-understand-i-want-my-hat-back/?invite=9I6hgw#comment-57466)

- Cory points out US semi manufacturing might be on the rise (https://www.metaculus.com/questions/6249/november-2021-production-of-semiconductors/?invite=9I6hgw#comment-57456)

- EvanHarper identifies a source how other NY officials feel about Cuomo's position (https://www.metaculus.com/questions/6693/will-ny-governor-andrew-cuomo-resign-soon/?invite=9I6hgw#comment-57783)

Expand full comment

I caution against getting too far into the predictions game, because it draws attention away from substance -- as do the horse-race elements of election coverage. It's usually not important to predict stuff like Biden's end-of-year public approval rating. What counts is: (1) whether a pundit was on the mark in warning about an under-protected risk (Protect against pandemic, dummy); (2) whether a commentator was farsighted in revealing options that most others were ignorant about (e.g., Farrington Daniels' early book on solar power); and (3) whether an expert revealed hidden facets or complexities in ways that deepened readers'/viewers' thinking (perhaps "Think Like an Economist"). There no doubt are more such major categories, but not many.

Expand full comment

I agree whole-heartedly and one one more category: (4) pointing out that something which is currently attracting a great deal of Deep Concern is in fact a storm in a teacup.

A good pundit doesn't think for me, but directs my thinking towards what matters. This concept should make more than a little sense to a professional psychiatrist!

Expand full comment

I don't know if I view the primary job of a pundit as prediction. I see them as primarily there to provide context and perspective on issues, which prediction should be a part of, but how harshly should we judge bad predictions?

In the Iraqi WMDs example, pundits were making their assertions based on the available evidence that strongly indicated such weapons did exist - the US government, the US intel community, foreign governments/intel communities, Saddam Hussein himself... All widely agreed that Iraq did have WMDs. How often do we have events with such a large amount of misleading info and so little publicly available truth?

Expand full comment

Dishonesty seems rather common in politics. Here's Greg Cochran on how he knew the intelligence agencies were wrong about Iraq:

https://westhunt.wordpress.com/2019/02/20/beating-the-spooks/

Expand full comment

He's talking about nuclear weapons - at the time, the thrust of the argument was around chem/bio weapons and perhaps old Russian warheads. I was a sucker for the Iraq War lies, but I never thought Iraq was making nukes. (I definitely thought the Russians probably gave them some at the time.)

Expand full comment

https://westhunt.wordpress.com/2019/02/20/beating-the-spooks/#comment-126334

"Calling something like mustard gas, or even nerve gas, “WMD” is just a lie. Mind you, Iraq wasn’t producing either one in 2003."

From another comment there:

"There were gas shells buried in the mud or left in an abandoned bunker by mistake."

He discussed bio war in the main post:

"This was demonstrated again later, when the CIA and DIA concluded that we had found ‘mobile biological labs’ in Iraq. Which were actually vans with portable hydrogen generators – hydrogen for balloons intended to measure high-altitude winds, for increasing the accuracy of artillery. Pinheads."

Overall, he didn't think Iraq had the money or technical capacity to be much of a threat.

Expand full comment

Calling mustard gas a WMD is plain and literal truth, because as long as we've had the term "WMD", war gasses even of the crude WWI variety were one of the three central and definitive examples. The term "WMD" war gasses, biological weapons, nuclear weapons, and anything some future boffin invents that's as lethal and indiscriminate as those". If you want to talk about nuclear weapons because you don't think chemical and biological weapons are lethal enough to be in the same league, we've got a perfectly good term to talk about just nuclear weapons. We've got a term for "weapons", another term for "very bad indiscriminate weapons", and a third term for "the one type of really apocalyptically bad indiscriminate weapon we know of so far", and that's a pretty good set of linguistic tools.

Believing in 2003 that Iraq possessed WMD was reasonable, due to the Iraqi regime's deliberate attempt to convince everybody including its own army that the regime was totally badass because of the hidden bunkers full of Sarin and Anthrax. Believing in 2003 that Iraq possessed or soon would possess nuclear weapons, was foolish. Most of the people I was following at the time, seem to have been reasonable and not foolish.

Expressing one's reasonable belief in Iraqi weapons of mass destruction in terms likely to leave the lay public (or random congressman) believing that Iraq had or would soon have *nuclear* weapons, was malicious and deceptive. I saw a lot of that in 2003, even from people who were carefully couching their explicit claims in the more restrictive and technically accurate sense.

Expand full comment

Mustard gas wasn't even used in WW2. I wouldn't lump biological warfare in with it. The current pandemic has done far more damage than all chemical warfare put together. Infections can spread much further than gas while retaining potency (this also means they can come back to hit the country using them, so it matters how reckless a user is).

Expand full comment

Then you might want to use the somewhat cumbersome term, "nuclear and biological weapons". You will fail at communication if you use the term "WMD" to mean "only nuclear and biological weapons".

Expand full comment

Indeed. My recollection from back then was that every Iraqi commander we debriefed had no chemical weapons in his unit, but had been told by his superiors that such weapons existed and were held by certain elite units.

Eventually we figured out that those commanders had been lied to, and there's a good chance they were the source of the assurance from pretty much every European Intel at the time that "Saddam had Chemical weapons."

I don't recall ever hearing anyone credible assert that Saddam had a nuclear weapons program, but that's over twenty years ago now.

Expand full comment

Colin Powell's testimony before the UN claimed that Iraq had ordered aluminum tubes that could only be for building a centrifuge to refine uranium, IIRC. And the standard line I remember from various politicians was "Our first warning could be a mushroom cloud rising over an American city." Joseph Wilson (Valerie Plaime's husband) was sent off to find evidence that Iraq was buying uranium ores ("yellowcake") from someplace in Africa. And so on. The claim, and probably the belief, included nuclear weapons.

Expand full comment

To serious people, the question of whether Iraq was working on nuclear weapons was just about the only WMD question that mattered. Chemical weapons are World War I technology that Hitler got along without. Biological weapons are a good way to make yourself sick, not an offensive weapon of much use.

Nuclear weapons, however, can be big trouble.

In October 2002, Cochran explained why Iraq didn't have the money or brains anymore to develop nukes.

Expand full comment

There are perfectly effective offensive biological weapons. Tularemia and anthrax work fine. Smallpox would be pretty effective now too, since nobody is vaccinated against it anymore.

The commies were dropping tularemia on German units outside Stalingrad, trying to slow down advances. It worked.

And a really nasty biological weapon would be worse than a nuke. I remember reading some Soviet work on engineering human myelin protein into a fairly innocuous germ: you get sick, mild pneumonia, then a week or two after you clear the infection, major autoimmune response and you get induced multiple sclerosis.

Expand full comment

Ah . . forgot that piece of it, though I remembered l'affair Plaime and how silly it seemed at the time. Twenty years and all that.

Expand full comment

I've linked this above as well. In 2002, physicist Richard Muller, who had been involved in the post-Gulf War assessment of Iraqi nuclear programs, predicted that an invasion of Iraq was inevitable. He based that the evidence found previously, that entire secret enrichment programs using technology nobody in the West predicted Saddam would use, had been in operation under the nose of both intelligence and inspector surveillance.

https://www.technologyreview.com/2002/02/07/235255/springtime-taxes-and-the-attack-on-iraq/

Expand full comment

You can find Gregory Cochran's original October 14, 2002 explanation of why the Iraqis couldn't be progressing toward a lot of nuclear weapons here on Jerry Pournelle's blog:

https://www.jerrypournelle.com/archives2/archives2mail/mail227.html

Expand full comment

Wow, that article is shockingly racist. Calling the population "morons" (unlike the "smart" North Koreans), saying no one in the Muslim world has "invented or discovered anything in 700 years", etc.. This author is disgusting, and even if it should have been obvious that the Iraqis weren't making nuclear weapons it's still crazy how much bias he's bringing to his post.

Expand full comment

It's a blog post rather than an article published in some larger publication. The author, Greg Cochran (whose paper on Ashkenazi intelligence was cited in the SSC Manhattan Project post), could perhaps have a bias causing him to underestimate the number of inventions or scientific discoveries coming out of the Muslim world. But, not being an expert, I'm not aware of any. People with more expertise are invited to provide counter-examples.

Expand full comment

I suspect we have a slider bar we can manipulate w.r.t. our sources of information and commentary. On one end of the slider bar, we have very polite and never offensive commentary, but we lose a lot of insights that involve upsetting or offensive ideas or facts or claims. On the other end of the slider bar, we have impolite and often offensive commentary, but also more accurate information and thus better decisions.

In this particular case, the decision we made as a country led to something like a hundred thousand of people dying, a million or so displaced, and a several-year civil war along religious and ethnic lines that I imagine will leave its mark on Iraqi politics for decades to come. We also spent billions of dollars and got several thousand of our people killed, and tens of thousands of our people maimed.

I'm thinking we would have done a lot better to push that slider bar *all the way* toward impolite and offensive and right.

Expand full comment

In contrast with TGGP's Cochran example—which relies on technical expertise vis-a-vis nuclear weapons pathways—here's another blog post on how to outperform the mass consensus, from a very different angle.

https://blog.danieldavies.com/2004_05_23_d-squareddigest_archive.html

Expand full comment

I worry about over-reliance on prediction as a metric for assessing the quality of journalism. Even apart from the methodological challenges discussed in the post, I think that success in particularized factual forecasting—while an important skill—is often unrelated to the value of a piece of political writing or analysis. Few of the ideas I find most valuable are susceptible to short-term, isolated verification of falsification. Synthesizing disparate facts into an interesting theory, etc., can rarely be tested by a “wagerable” near-term market result, and I am skeptical that facility in the one is particularly relevant to the usefulness of the other.

Expand full comment

Perhaps prediction isn't the best metric, or doesn't capture everything, but I'm not sure that producing "an interesting theory" is any better. Maybe I'm just taking you too literally, and if so, oops, my bad, but, what is the value in such a theory? I see two possible ways it could be valuable : being informative, and being entertaining. Entertainment is all well and good, but I think it best to make the distinction clear. The value of something as being informative, is that it gives us an idea that might be true, and the value comes from the chance that it might be true. There may be truths that one with skill can glean from reading a situation, and which don't cleanly translate into a prediction about an outcome that can be uncontroversially measured/evaluated in the near future.

But, the beliefs should be useful for taking action in some way, right?

If the beliefs led to better choices, shouldn't it be possible to, like, look at how they influenced the decisions, and therefore---

well, I suppose one doesn't have access to what the outcomes would have been if one had taken other choices.

But, one can somewhat evaluate how well things turned out at least, and make some attempt at tracing how things turned out to what predictions were used.

Expand full comment

In my view, theories like that typically have actionable consequences and should lead to better choices. Unfortunately, I am skeptical that those empirical consequences are subject to definitive testing by using a simple declarative sentence that will be objectively provable with a period of several years.

Expand full comment