355 Comments
Mar 20·edited Mar 20

None of the venerable tests being accurate over 135 sounds like something that should be noticeable outside this community. Is this an accepted fact of IQ research, or if not how does that match with this? Is IQ just not normally distributed at the tails or what is going on?

Is there some other test that's used above 130?

Expand full comment
Mar 20·edited Mar 20

Another issue is that the official in-person Mensa test is apparently not a standard IQ test with standard deviation of 15, and most people taking it don't realise this.

This was discussed back on SSC at https://slatestarcodex.com/2018/01/14/ot93-giant-threadwood/#comment-588588 and https://slatestarcodex.com/2018/11/21/impending-survey-discussion-thread/#comment-693849.

My Mensa certificate just says on it "This gives a true IQ" and doesn't indicate which test (e.g. Wechsler, Stanford-Binet, Cattell, etc) it was. So for the first few years of Scott's survey I just wrote the number from that certificate, which does sound pretty high. I've since inferred from the linked SSC discussion that it was probably Cattell and that the corresponding Wechsler and S-B numbers are quite a bit lower. But given that the survey rubric doesn't specify which test or indicate that some so-called "IQ tests" (including the one used by Mensa) use a differently-calibrated scale, and given that even the Mensa certificate itself doesn't indicate which test it used, I am not surprised that some commenters are reporting too-high scores, in good faith.

Expand full comment

Some thoughts:

The people on the lowest end of the IQ distribution probably can't work as click workers due to being illiterate, unable to follow instructions, or similar reasons. So you'd expect the average to be above 100 in that sample.

People generally only take IQ tests if they expect the result will either be very low, or very high. The "very low" group will be underrepresented in the sample as noted above. So we're left with the people who took an IQ test because they expected to get a high number (for gifted classes, skipping a grade, MENSA etc). The people whose IQ is actually ~100 haven't taken a test.

As for the SAT, people probably remember the exact score if they tried to get into some college program where it mattered, or if it was flatteringly high. If your score was really mid, there's no reason to recall it.

Tl;dr very low IQ people don't take online surveys, and middling IQ people don't know their score, leaving the high scoring people to sample.

Expand full comment

> or just subtract 10-15 points from whatever you give me.

Ok, great, I'll just add 10-15 points to what I report next time around.

Expand full comment

I got about 138 in an IQ score 20 years ago - pretty official as it was in a company and I was locked in a room. That’s my highest result. I haven’t tested since and I’m probably not going to. So I’m probably reporting my highest ever result there (well I know I am but I’m not sure what the others were, just that they were lower).

Anyway that’s what I would answer if asked, “in my last test I got 138” - which is true but almost certainly high balling it.

Expand full comment

That last bit about 1/30 people finding internet statistics essays interesting is a bit of a reach, I think. Maybe only 3% of people are into that, but they aren't necessarily going to be the top 3%.

I can believe there's a pretty strong selection effect, but so strong that somebody with 114 IQ (like 85th percentile) is just as likely in the sample as somebody with 140+ IQ (like 99.9th percentile) is pretty strong.

Expand full comment

IQ test administrator here. The ones I use are WISC and WAIS usually. I’m here to add weight to the theory that above 135 scores are not as precise for the relevant groups, that is, adults and young adults reading this.

1) The WISC-V came out in the last seven years but even with the Flynn effect lagging as of late, it’s newer than the WAIS-IV, which has an update coming out this year. The SB is hopelessly old and I think its validity is suspect at this point. (Anecdotally, the final test item for WISC subtests “feel” harder than their adult equivalents.)

2) The WISC is used up to 17 years of age, and anecdotally speaking, that is, I’m not actually whipping out my norm conversion tables, the difference between a top possible score and a merely 130 score equivalent is sometimes a single item or the speed by which one answers a single item in a given subtest.

3) This ceiling effect is less true for younger ages, which makes sense because theoretically, even a six-year-old may be completing the same task as a 16-year-old.

4) This isn’t really a problem for the professionals typically using IQ tests as our main job is determining intellectual disabilities or specific learning disabilities. We’re rarely trying to tell super smart kids how smart they are. If you’re above 130, you qualify as gifted, and we don’t care anymore.

5) If you qualified for gifted (over 130), good for you. However, you probably qualified when you were super young and federal laws do not require that we retest you every three years like it does for disabilities. It’s quite possible that you may have had an intellectual growth spurt. We’d never know.

Expand full comment

I wonder about an obvious effect that wasn't mentioned: what happens when people take multiple IQ tests and report their highest score? For example, I've been accepted into Mensa, so scored about 135+ officially, but I guarantee that if I took 10 different IQ tests, in at least one of them (especially one involving language), I would score less than 125. I've even scored as low as 109 much earlier in my life.

Expand full comment

I got this right? a) If I claimed an IQ of 128: fine: most ACX readers should be (not sure, when reading some comments - but, hey, me probably 120+ and I AM much DUMBer than I wished)

b) If I claim 138 (I got as an early teen, due to age-bonus), then it gets loped off to 135 or 128 (138-10).

Why not just assume an IQ of 125 (or 128 - what does it matter, really)? And drop the question.

Or ask: 'What other blogs you read?' and adjust? SSC +5 /Zvi: +5 / Derek Lowe +8 / Hanania -10 / Pueyo -5 (yes, I pay subs. to Thomas P.)

Expand full comment

Curious about the assumption that readers will hate on Mensa. Why the hate?

I joined for a year ages ago when I was new to a town. The monthly game night was a blast.

I’ve never taken an official IQ test but have gone off the fact that Mensa accepted me based on GRE scores which I figure puts me top 2%. And based on the incredibly unreliable observation of how often I think people are smarter than me, I don’t think I’m much, if any, over top 1%. It occurs to me that anyone doing a similar gut check on a test result over 140 may have a hard time due to sample size.

Expand full comment

Unrelated: My headcanon is that every time Scott repeats a word, it's a deliberate easter egg, and there's some kind of secret prize for the person who finds them all. That said,

> So are people just taking terrible Internet IQ tests that inflate their score **about about** 20 points?

Expand full comment

When I was a psychology student, 30+ years ago, in a country far away from where I live now and from the US, we were told that an average IQ for *university students* then was about 120. This made rough sense considering the % of population that attended university there and then. I realize that it's not like that for the US now, but the idea that the audiences for those websites/your blog etc would be INCREDIBLY self selected by IQ feels plausible. Not 135+ on average obviously but certainly well above not just the population mean but even degree-level educated population.

I don't understand why you'd assume click workers would have population average IQs? To me it seems obvious that they'd be higher, because ANY group that engaged in any cognitive labour will have higher than average IQ. Not just because it would exclude those with the very lowest scores incapable of doing that labour but also because it'd exclude a lot of people with middling lowish to average scores who have less interest and inclination in such.

Also, self selection for even measuring IQ others mentioned and reporting the highest score: for example I'd likely report my 135+ Wechsler score from my student era (I was also subject to the one used by the US army historically, translated/normed of course but that was as a teenager and I don't trust that result) rather than 125+ Raven score (not just because it's higher but because I'm certain that the former measures "manifest" intelligence better). I'd not report middle aged English results (not because my vocabulary is worse, but because I've discovered that memory, especially digit memory, for me varies by 2 digits depending on language, both straight and backwards).

So I think selection bias in audience and reporting explains most of these results.

.

Expand full comment

>Most people don’t read long articles online about statistics. 1/30 sounds like as good an estimate as any other for the sort of person who would find that comprehensible and interesting.

My estimate for the % of English readers on the web who read long articles about stats on this type of topic, with any degree of regularity, is 0.1% - 0.3%. My wife and quite a few of my smart friends are never (ever) going to read 80%-90% the types of things this community proactively seeks out and reads on a monthly basis. It's not reasonable IMO to convert this percentage of the population to the percentile threshold for an IQ cutoff, but for grins and giggles if that were a reasonable proxy we'd be back to the 138-145 range.

Expand full comment

Intelligence changes with age. You shouldn't weight on just education, at least age needs to figure in as well. Testing age will necessarily be lower than in the survey, SATs are usually done when people are at peak intelligence, so tests in ClearerThinking are likely to be post-peak and past tests likely to be closer to peak. Plus: Flynn effect, norms change.

Will you/Spencer share the ClearerThinking data? Would be useful for our research on norming from convenience samples.

Expand full comment

About 2 months ago I read a headline: 5min (or some short) IQ tests work just as good as long IQ tests!

So I open Google and take the first test 5 min IQ test that popped up; I had to pay for my score when reached the end. I close that tab and search this time, very carefully- noting the word ‘free’ in the search results. Once again I get to the end of another free 5 min test, ‘We’ll send you an email with results’. The email I receive is a link to pay for results. I look through another 25 links, surely one must be free! I find a page “Free FREE FREE!” This is the one! I take the test, give me the results! 140? 155? It says I need to pay for results. but taking the /test/ was free…

I close all my tabs, and go for a walk.

Remembering my escapade 2 weeks prior (heaven knows I won’t tell a soul about *these* struggles) I doggedly find the headlined paper making the online buzz for 5min IQ tests. I download the software from the study -an old JavaScript program- which I have to debug because it’s ancient. I spend 20~ mins on this. Finally! I launch it. I take the test. I click ‘results’, and it launches me into a website where I can buy the full version of the software suite.

Expand full comment

There's an observable phenomenon where people think about IQ as if it averaged 110-115, not 100. A "really smart" person has an !IQ around 140, and a "really dumb" person has an !IQ of, like, 98. (At its extremes, this is the fictional genius whose reported IQ is in the 170s or higher.) I saw a particularly prominent example of this process in a discussion of personality that referred to how a similar personality would "look different in someone with an IQ of 140 to someone with an IQ of 90". Those aren't symmetrical, but if you think about IQ how people tend to, you end up saying something like that.

(This isn't a range-restriction thing, exactly -- everyone does it.)

I've always thought accordingly that you'd be about right to estimate the average IQs of people taking the SSC/ACX/LW surveys as in the lower-mid 120s. This is extremely high! It is, in fact, about as high as what people are really intuiting when they think about someone with "an IQ of 140" -- the range of way-way-above-average people in the general population. It's also a plausible average *full-scale IQ* for a group of people who tend to be both smart and...eccentric. Subscales are pretty well-correlated in the general population (the "g factor"), but the correlations are not groundbreakingly large and many unusual people have sizable gaps between subscales. There are whole portions of the WISC/WAIS that are just "oh yeah, these are the sections that routinely break and lead to people's FSIQs being markedly off from some of their subdivisions".

Now, it'd be a bit of an exaggeration to say you can't test above an IQ of 135, in the same sense it'd be a bit of an exaggeration to say you can't test below an IQ of 65. We can definitely test in the mild intellectual disability range. Below that...hahahaha good question, someday we'll stop giving autistic kids "adaptive functioning tests" that dock points for stimming. I suspect a lot of the invalidity on *this particular metric* is due to things other than "the tests don't work that high", like "people are lying" or "people are giving an unrepresentative result" or "people misunderstood the results somehow" (given in a different comment) or "people have unusual subtest divisions, and the questions in the surveys didn't account for this" (were they matrices?). Probably all of these across many different people.

Expand full comment

Don't IQ tests also have an upper bound? Like you sit with a cognitive therapist or neurologist or whoever and they do several hours of different types of tests and then they report the score.

Expand full comment

This way of doing the SAT to IQ conversion looks wrong for reasons that were explained on the subreddit.

https://www.reddit.com/r/slatestarcodex/comments/1avcld5/i_still_dont_think_slate_star_codex_readers_have/krccacl/

If all you know about a person is that they were +2 SD on the SAT then on average they'll be less than +2 SD on an IQ test, but if you select a group of people for some other reason and they're +2 SD on the SAT on average then then you shouldn't necessarily expect them to be worse at IQ. After all, you could run the same regression argument in reverse (if all you know is that someone is +2 SD on an IQ test then on average they'll be less than +2 SD on the SAT), and the group can't be worse at IQ than at the SAT and also worse at the SAT than at IQ.

I'm also pretty skeptical of Spencer's attempt to norm an IQ test which was given to a convenience sample rather than a representative sample. I wouldn't expect controlling for 1 variable to fix that.

Expand full comment

My mum was highly into IQ so I ended up taking an absolute tonne of IQ tests as a child, and I have fold memories of the one I got 225 on, but then on further investigation it turned up I'd added up my scores wrong so, you know, actually didn't. But if you count that one out, I remember getting scores between 105 and 175 from various tests, so, I mean, clearly they were not all super reliable. If I wanted, I could just take the highest (except the clearly erroneous 225) and walk around being like "yeah I tested at 175 IQ as a child" and it wouldn't be explicitly false but I'm not sure it would be honest. I do often suspect something like this is going on in these self-reports.

Expand full comment
founding

I honestly fail to see the mystery here.

> These are implausibly high. Only 1/200 people has an IQ of 138 or higher. 1/50 people have IQ 130, but the ClearerThinking survey used crowdworkers (eg Mechanical Turk) who should be totally average.

There's a huge availability bias in saying that those numbers are implausibly high. Scott, most people in his bubble, and probably a majority of the blog readers - they just don't interact with people below average IQ. Practically... never?

To quote Scott himself when talking about how well-selected his tribe is:

> And I don’t have a single one of those people in my social circle. It’s not because I’m deliberately avoiding them; I’m pretty live-and-let-live politically, I wouldn’t ostracize someone just for some weird beliefs. And yet, even though I probably know about a hundred fifty people, I am pretty confident that not one of them is creationist. Odds of this happening by chance? 1/2^150 = 1/10^45 = approximately the chance of picking a particular atom if you are randomly selecting among all the atoms on Earth.

Yeah, practically never. I don't know about ClearThinking, but the LessWrong numbers feel pretty much spot on. And even the bit about mturk workers being "totally average" - it's honestly a bit hilarious, and comes from never having to explain a below-average person concepts of equivalent complexity as mturk.

And once the large bias towards high IQ is explained, all that's left is disparity between various ways to measure it. Which yeah, is an interesting statistical exercise and I actually read and enjoyed the post. But, like I said, not a huge mystery.

Expand full comment

Curious as to why you found very high average IQ implausible for LessWrong readers? Personally, I'd be stunned if it **wasn't** an amalgam of the literal smartest people in the world.

Expand full comment

Thus we can only return to the status quo of using social status and citations as proxies for measuring extreme intelligence.

Expand full comment

Whenever I come across voluntary surveys of IQ, David Mitchell's rant about who tries to become a member of MENSA and who does not, comes to mind. It's about selection effects. Not particularly relevant in this context perhaps, but it is a fun rant, as are all of David Mitchells' rants:

https://www.youtube.com/watch?v=qPMKqyaXtHI

Expand full comment

I made one IQ test in my.life where I scored "below 130" (don't remember the exact number, since 130 was a cutoff for something) and one, later, with 135.

If I am asked for IQ number, I now report 130.

So for ppl who did multiple IQ tests, we have an upwards bias (even if asking for the most recent one, because people redo it more likely if they're unsatisfied with the last), whereas "doing live testing" gives us an unbiased snapshot, including things like "I slept bad and my coffee machine broke and my brother is dieing" unconcentrated for some)

I sometimes wonder: what would be the mean IQ if everybody was allowed to submit the best test result throughout their lives? 120?

Expand full comment

Who cares. I understand that you collect this information for your own uses and that a lot of people take tests offered by the internet because they want to know how smart they are but we don’t ask people what their IQ is and we suppress laughter when someone volunteers that information.

Nor does it predict success in life—life being more than professional attainment or wealth. Of course admission to elite universities requires greater facility with mathematics and language. And physicians and engineers are all pretty smart—but to me here is the key question. Two surgeons will have the same IQ but one will be outstanding and the other a disaster. A doctor with a lower IQ than their colleagues may be the better diagnostician of the bunch.

Let’s move from technical professions to business and law. The ability to relate to all sorts of people is a far more valuable attribute than one’s IQ. In some instances and to some extent it may be inversely proportional.

I love your brilliance, but I think this post was dumb.

Expand full comment

Some notes on Mensa:

Mensa advertises that 2/3 of people get in with “prior evidence” and don’t have to take the Mensa test. They also only accept SAT scores older than ‘86 IIRC.

I took the Mensa test about 15 years ago and they also administered the Wonderlic beforehand. Possibly for renorming. When I received my results it was 130+ (or maybe 132+) because that was the limit of the test. I would be suspicious of anyone claiming to get a very high score on the Mensa test.

Non Mensa related: I found a bootleg copy of Ravens and it also capped out in the low 130s. There’s a separate step 2 if you want to test higher.

Expand full comment

Given the community here I might get a hit on this and have always been fairly curious as to the experience of others: Anyone else get wildly different math and verbal scores? I was put in special education the first week of the first grade because I had a lot of difficulty making myself understood. I used to put that all on some panic stuff I had going on but now I’m less confident that was the biggest reason. Even after I eventually learned to translate my head universe into communicating with other people there was still a pretty significant gap. If I’m reaching back I think I got a 35 in the math section of the ACT and maybe 29 in verbal? I’m always able to look at something and figure out how to fix it much, much quicker than I’m able to figure out how to explain how I know to someone else.

Expand full comment

ACX/SSC-only people often won't get this, but in Lesswrong and AI safety-oriented culture, being merely one-in-300 is pretty terrible news; it means you were the smartest kid in almost every classroom growing up, but now you are median, and the people doing the lion's share of impact on Earth are the one-in-10,000 types.

Why would anyone read your post/paper at all, or hire you for their org, if they can spend that time reading/hiring a one-in-10,000 person instead?

Expand full comment
Mar 20·edited Mar 20

Is this not likely to be reporting bias/selection effects?

- If you think you are average will you bother getting an IQ test?

- If you did a test and got an average IQ are you likely to fill in a form about it or will you only bother if you scored high and want to boast a bit?

So it could be that only the people who suspect they are in the highest rungs will have had and IQ test, and of those the ones who scored highest are more likely to complete the surveys as a boast and you get very skewed selection effects

Expand full comment

I had a tested IQ of 138...when I was 10, on the WISC. This raises three obvious lines of questioning.

1. Does age-adjusted IQ at 10 correspond to an equally high adult IQ, and older adult IQ?

2. IQ declines over time, probably due to decades of substance (ab)use, head injuries, illness, and most importantly general aging. How does that skew the data?

3. Would someone with an IQ within a standard deviation of the mean be more or less likely to be sent to a fancy psychiatrist in the inner city to take the WISC/WAIS/etc.?

I think I'd expect to get 115 to 125 on the WAIS as an adult. My memory is shot, I'm slower than I used to be, and I struggle with some mental tasks I used to find easy. I remember getting somewhere around 120ish on the least-disreputable online IQ tests. It's not clear to me that "my IQ was X when it was tested" means "my IQ is X" and I think time is likely a bias.

It would also be interesting to see if the WISC was unusually biased compared to adult tests, and interesting to collect data on age of measurement. Maybe everyone has a higher tested IQ because most of them were either taking an age-adjusted test as a child, or in the 18-25 age range when their IQ should have been at its highest point.

Expand full comment

Part of the problem here is

Psychometricians noticed that scores on a lot of different types of cognitive test are correlated. The general intelligence factor, g, which IQ tests are intended to measure, is the correlated component of those scores. But it turns out that the relationship between scores on different types of cognitive tests is heteroskedatic— they become less correlated as you go further up the scale. This also means that if different assessments of g include different items, their correlations will drop at the high end. More profoundly, it implies that IQ-type metrics get less useful as predictors of performance on specific tasks after you get past a certain threshold. So, “reported IQs over 135 are less meaningful” is a good heuristic.

Expand full comment

Have you considered just asking your readers to take a well-normed IQ test that directly reports the results to you? Like, I’m pretty sure you can just pay Wonderlic to administer an IQ test to whoever you want (I don’t know how well normed they are, but it’s got to be better than the existing situation).

Expand full comment

SAT is a US thing. As an Australian I don't have one. I didn't read the article in detail but maybe that affected the scoring -- non-US-educated either not reporting or guestimating.

Expand full comment
founding

If you were to suggest that only the smartest 1/200 or even 1/10,000 people are the ones that read this blog/have taken Mensa tests/iq tests, I would absolutely believe you. The high level of reading / logical reasoning to both understand and enjoy this style of writing is such a strong filter.

I know we avoid complex language to make the blog more accessible to anyone, but to enjoy it is another story.

I’d assume based on my peer group that people with an average IQ, don’t read at all and mostly scroll tiktok. If they do read it is very simple books/novels.

Expand full comment

To what extent do these "IQ" tests remain correlated with wide-spectrum achievement at the 130+ level? The scholarly paper has 116 students, there's no way there are enough data to base a claim that these tests are measuring general intelligence at the tails. Not a test-repeatability question but a test-to-IQ question.

Having spent my life in places where a 1400 SAT would be considered kinda low, there is clearly not an expontential-tail distribution where the 1500 SAT people are so obviously less intelligent than the 1600 people, even on specific things. And the "true genius" types are less likely to have general intelligence that you would call g, and instead seem exceptional in a particular area.

The concept of Bell-curve IQ describing general intelligence at genius levels should go IMO. Taking with it all the implications for godlike AGI.

Expand full comment

My 1985 SAT scores - 710 math, 720 verbal - were good enough to get me into Mensa - 98th %ile - and Colloquy - 99.5th %ile - but not the triple 9 societies. I estimate my IQ as 3 SD up, ~140. A friend of my mom's, Stephanie Creamer, was getting her doctorate in industrial psych around 1980; consequently, I took A LOT of IQ tests, and usually hit around there, though my scores trended slightly upward as I kept practicing.

Expand full comment

>The Biggest SAT → IQ Conversion Site Is Wrong

Jensens point is very reasonable but Im not sure its right for your application. If you have genpop looking at an internet site to convert their SAT to IQ, you should use his calculation. If you have a sample thats selected from genpop for their SAT, then his calculation will also give their average IQ correctly. On the other hand, if you have a sample selcted for IQ, and then you measure their SAT, and then you apply his calculation, you will underestimate IQ.

So, if your survey sample is unusually smart, what you should do depends on why you think that is: If its selected mostly for SAT, use Jensens calculation. If its selected mostly for IQ, you need to do the opposite correction, and if its about in the middle, then the original percentile calculation is about right.

Expand full comment

"People who took the SAT but “don’t remember” their score have tested-IQ 104"

But...I don't remember my score. It was 45 years ago! I know it had to be above 1200 combined, because in those days California was legally required to let you into a campus of the University of California if your GPA was over 3.2 and your SAT was over 1200, and if you met several other requirements like X number of math, science, and foreign language classes. And I remember thinking "Great! UC is legally required to admit me, I can exhale." And indeed I did go to Cal.

But I don't remember my exact score.

Expand full comment

Only tangentially related

I can’t find the hard copy of the Al Franken book I’m going to reference right now so I’ll have to paraphrase from memory.

Apparently one of the first questions Ted Cruz likes to ask when he meets a person is “What’s your IQ?” If the person doesn’t know he asks “Well, what is your SAT score?”

Franken goes on to say “I like Ted Cruz more than anyone else in the Senate does, and I hate Ted Cruz.”

I really like Al, he’s a funny guy and was a great senator IMO and his being driven out of the Senate was uncalled for, again IMO.

I still get his emails from him with the greeting “Dear person who still opens my emails.”

He still works for Democratic candidates and sells merch on his web site to that end. One item is a coffee cup with a pretty good caricature, done by Al, of Cruz on one side and Al’s withering opinion on the other.

I have one. Ted Cruz rubs me the wrong way too.

Expand full comment

I'm curious how you think your analysis here works with your viewpoint here? https://www.astralcodexten.com/p/selection-bias-is-a-fact-of-life

Expand full comment

Given that IQ kind of -is- a rank order question, I don't think the problem #3 is the problem you are presenting it as.

Expand full comment
Mar 20·edited Mar 20

This was a question from the 2022 ACX Survey:

"SAT score verbal/reading

The SAT is a test Americans take before going to college. In the old days, it included a section called Verbal; now that section is called Reading. If you took the SAT, what was your score on this section?"

Several dozen respondents put in a value >1500.

Expand full comment

"even official IQ tests are gobbledygook over 135"

This.

How do you make a test for someone smarter than you are? At some point, you're running into the doomsday scenario for AI, where it runs rings around you.

I assume IQ test makers tend to be smart people. But how smart, and how many of them are there?

Expand full comment

I think that your method of considering the distributions could use refinement. If one just assumes that nobody below 100 IQ is involved in reading less wrong then the new mean for the half-normal distribution is 112 100+(sqrt(2)*sigma/sqrt(pi)). Essentially, eliminating a significant bottom portion of the distribution substantially skews the mean without the need to assume an inordinate number of high IQ individuals.

Expand full comment

This article is especially fascinating to me because there needs to be another possibly: testing bias(?)

I'm old enough that I've taken: the GRE, GMAT, SAT and ACT. My respective scores were 2200, 780, 1410 and 34. I also took the Wunderlic for a job and scored 49.

So this should mean I'm some sort of brianiac eh?

Nope. When my IQ was actually tested - in a rigid setting - it came out as 111. Not dumb but definitely just... bright normal, which I think is the actual term ascribed to me.

So what gives? Outliers exist but I am highly skeptical that I'm the only person out there who aced a bunch of standardized tests only to actually be told: hey, you're bright but not exactly Einstein

Expand full comment

Should crowdworkers come in at a little above 100 because people with really low IQs aren't likely to be crowdworkers?

Expand full comment

" the ClearerThinking survey used crowdworkers (eg Mechanical Turk) who should be totally average"

Why would you expect people who seek out online crowdwork to be "totally average?"

Expand full comment

> It looks like up to about 140, self-reported IQ and normed IQ rise together, and then the relationship breaks down. Sure enough, looking at the subset of self-reported IQ scores below 140, the correlation with tested IQ rises to .6, and looking at the subset above 140, the correlation is nonsignificant at -0.02. I don’t want to assert that the breakpoint is exactly 140, but I do think the test stops working somewhere in the 130 - 140 range.

Didn't ...you... already write a post about this being an illusion? Pointing out that the reason for a weaker correlation between IQ and salary in the subset of people with very high IQs than overall is simply that there is no other possibility, and that in fact in a model where we know the relationship between IQ and salary is constant, we observe exactly the same thing? Why is the same argument that's invalid as applied to IQ-vs-salary valid as applied to IQ-vs-IQ_in_the_past?

------

I think I've seen Emil Kirkegaard make the same point as in #1 - that you shouldn't convert SAT scores to IQs on a percentile-to-percentile basis, because the correlation between them is only the stylized 0.8 that's expected between two different IQ tests, and not 1.

But I don't really understand why this is an argument against doing the percentile-to-percentile conversion. The correlation between the Wechsler and the Stanford-Binet is also a stylized 0.8, because they're both IQ tests. But we don't go around saying that measured Stanford-Binet IQs are incorrect (they're correct by definition!) because they need to be regressed by the correlation to Wechsler, which is the True Arbiter of IQ, and we especially don't simultaneously say that, even though Stanford-Binet IQs are too high because in reality they need to be regressed by the correlation to Wechsler, Wechsler IQs are 𝗮𝗹𝘀𝗼 too high, because they need to be regressed by the correlation to Stanford-Binet!

We do say that if someone has a high measured IQ, their actual IQ is most likely lower, because of regression to the mean. This also applies to SAT scores, which are the same thing. But that argument suggests, to me, that percentile-to-percentile is the 𝗿𝗶𝗴𝗵𝘁 way to do concordance. What am I missing?

Expand full comment

IQ is a normalized distribution so for every person with an IQ of 140 there is a person with an IQ of 60. Someone with an IQ of 60 is not going to be taking these kinds of quizzes while someone with an IQ of 140 is very likely to. I’d even expect people with an IQ of 100 to be very unlikely to be interested in these kinds of surveys. If you cut off the left tail of the IQ distribution, of course your average will be higher than expected.

Expand full comment

I think that if people take IQ tests, they take them as kids because they're being tested for "are you really bright/is something going on as to why you're not doing well in school and/or socialising with your peers?"

And thus I think people as adults may be reporting child IQ test results, which I don't know if one is related to the other accurately. Yes you were very smart for your age when you were twelve, but are you still very smart for your age now you're thirty?

Anyway, I cling proudly to my Completely Reliable, Honestly! Online IQ Score result of 98 😁

I do think it's fairly accurate because it was based (they said) on Ravens Matrices and I am *hopeless* at pattern recognition, shape rotation, or maths. Given that the tests were all pattern recognition, shape rotation, and maths, this result seems legit to me.

I do wonder what it's like to be able to look at a sequence of numbers and *immediately* see that "The pattern is X, Y, Z so the next digit should be A". But I'll never know, no more than I'll ever know what it's like to be a bird or a whale or a sloth, happily clinging to a branch in a South American jungle as the lichen grows on my fur because I am so inactive.

This makes me think that we're moving to a new age of calculating "intelligence", where now we're in the Information Age of AI And Computers, it's all based on mathematical ability rather than verbal. We're moving between C. P. Snow's Two Cultures, where before to be considered 'smart' was judged on the basis of your cultural and literacy attainments, but today it's based on "can you even code, bro?":

https://en.wikipedia.org/wiki/The_Two_Cultures

" 'A good many times I have been present at gatherings of people who, by the standards of the traditional culture, are thought highly educated and who have with considerable gusto been expressing their incredulity at the illiteracy of scientists. Once or twice I have been provoked and have asked the company how many of them could describe the Second Law of Thermodynamics. The response was cold: it was also negative. Yet I was asking something which is the scientific equivalent of: Have you read a work of Shakespeare's? I now believe that if I had asked an even simpler question – such as, What do you mean by mass, or acceleration, which is the scientific equivalent of saying, Can you read? – not more than one in ten of the highly educated would have felt that I was speaking the same language. So the great edifice of modern physics goes up, and the majority of the cleverest people in the western world have about as much insight into it as their Neolithic ancestors would have had.'

...Snow's Rede Lecture condemned the British educational system as having, since the Victorian era, over-rewarded the humanities (especially Latin and Greek) at the expense of scientific and engineering education, despite such achievements having been so decisive in winning the Second World War for the Allies. This in practice deprived British elites (in politics, administration, and industry) of adequate preparation to manage the modern scientific world. By contrast, Snow said, German and American schools sought to prepare their citizens equally in the sciences and humanities, and better scientific teaching enabled these countries' rulers to compete more effectively in a scientific age."

I'm one of the dinosaurs of the Neolithic who is verbally able, but totally lost now in the era of you small nimble mammals with your maths manipulation abilities.

Expand full comment

I feel like income is a much better proxy for what we value in “intelligence”. Obviously Jeff Bezos isn’t necessarily smarter than Scott Aaronson but they’re definitely in the same league or else Bezos wouldn’t have been in the position he’s in today. This is perfectly captured in the “if you’re so smart why aren’t you so rich” quote.

So… isn’t it better to just ask for peoples incomes, stack rank them against people of their age and place of residence and call that “IQ”?

Expand full comment

I think there's a typo in your last paragraph - 1/30 people don't have an IQ above 128 (z-score of 0.9974). I think 1/30 is roughly the number for a score of 120 (z-score of 0.9772).

Expand full comment

Why even bother with some IQ test of adults who are gainfully employed? Of what value is it other than perhaps for self gratification for some who see via internet test/survey that they score high on an IQ test. Me: Could care less as having multiple degrees and financial and career success I find it all a bit silly. Yes, for the very young it might be of help for seeing those with very very high IQ get the proper degree of educational nourishment.

Expand full comment

What life circumstances lead a cognitively normal person to get an IQ test? This seems completely foreign to me, like something I only hear about on the internet. The only real-life person I know who has gotten an IQ test is my older brother, who is has an intellectual disability.

I guess maybe for a job? I had to take the Wonderlic once for a job. But that's not exactly an IQ test, I don't think.

Expand full comment

Why do people hate Mensa?

Expand full comment
founding

> The LessWrong average was 1446, corresponding to IQ 140.

I think you're using the pre-1994 SAT correspondence, but most of the people responding to the survey probably took the SAT after 1994. As stated on that link, they made a change then that dramatically compressed the range; as a result, I don't think the SAT is that useful for assessing IQs above ~120.

Expand full comment
Mar 20·edited Mar 20

Re. "the ClearerThinking survey used crowdworkers (eg Mechanical Turk) who should be totally average.": Why do you think that?

Mechanical turk workers have a lot of filters on them. First, they must have a computer and an Internet connection in their homes. In India, this is a big filter. They must be non-conformist enough to risk getting this weird new kind of job; that's another filter. Then they must apply for the job and pass. Then, once they're on the job, they compete for tasks, especially high-paying tasks. And your results might be enriched in mechanical turk workers who searched for that job because already knew about these surveys from being involved in LW or ClearerThinking.

Expand full comment

As a child I tested 144 and my SAT was 1380. I can name poker hands and explain the infield fly rule but I know next to nothing about pineapples or how to build a table. People are too obsessed with these measurements, and I have a theory that that is holding back human progress.

Expand full comment

I occurs to me that one way of establishing a norm could be another test a lot of people take: the ASVAB. It isn't hindered by the "smarter people take it more often" like the SAT. It probably is hindered a bit because it probably isn't taken by many people at the upper and lower bounds. The AFQT score from it is a literal percentile and the score most people remember.

As a data point , my AFQT was 99 and my SAT was 1400. According to the correlations listed above and the percentiles at iqcomparisonsite.com if AFQT is an accurate percentile, that apparently puts my IQ at somewhere between 135 and 137.

Expand full comment

The first point you make, about regression, should apply equally in both directions. If SAT and IQ are correlated at .8, then using either one to predict the other should result in regression to the mean. So if IQ scores are higher than you should expect by regression from SAT scores, because they are at the same percentile, then by the same token, SAT score are higher than you should expect by regression from IQ scores.

More likely, both of these average reported scores are being dragged up by some third factor (which might be one or more of the other things you mention) rather than one of them leading and the other being dragged.

Expand full comment

"and least likely of all to optimize their lies for getting IQ/SAT correspondences right."

That's comparing averages, not individual scores, so consistent with people getting the correspondence badly wrong but their errors roughly averaging out.

One possibility you didn't mention is people getting their IQ from a test done when they were children, with the score being literally mental age over physical age, not the norming of adult tests. The only IQ figure I have for myself was due to a fellow high school student copying the school's list of students and IQs and showing it around. I was at that school from kindergarten on, so the score could be me at ten and only weak evidence of my adult IQ.

Expand full comment
Mar 20·edited Mar 21

I have no problem with all the discussion around the inaccuracy of test scores collected in a uncontrolled manner.

But also: is it really so hard to believe that the audiences for LessWrong and ClearerThinking have a selectivity of at least 1/200?

Expand full comment

The only time I’ve ever bothered to get my IQ officially tested was back when I was 18 - I was very interested in IQ and IQ tests, I practiced on all the online tests and similar games (anyone remember Brain Age on Nintendo DS?), and I prided myself on being “smart” a lot more than I do now, so I gave it all the preparation (nicotine!) and effort I could.

If I took the test today, I bet I’d score 10 points lower. This effect surely doesn’t account for a whole standard deviation, but I bet there’s some factor where people test their IQs when they’re at their “smartest”, narrowly construed. The idealized IQ test may be impossible to prepare for, and remain constant over a person’s life, but in real life that’s definitely not the case.

Expand full comment

I don't really care. When I was well above fifty years old I got the result of an IQ test I did as a teenager and reported that here. After forty years of more or less moderate drinking, I should score significantly lower now.

Expand full comment

Just so you know, internet iq tests can be terrible in the other direction. Once I took an obviously sarcastic IQ test on a humor site that asked a bunch of ridiculous questions and trick questions, then reported that my IQ was 7.

Expand full comment

Gwern, I think Scott got the joke: my calling an essay on IQ, dumb. Besides, who am I for Scott to feel insulted by any opinion I offer. But to answer your critique of my comment, a few observations.

First. What I like about Scott’s articles is that he respects his audience and doesn’t dumb down his discussions. But his Substack is subscribed by numerous ordinary people like myself who find his analysis of ideas and issues cogent even if we can’t always follow his math. Unlike you I do not presume to know whether that was his intent when he created his blog, but I suspect that he knows a great deal about those who read his posts.

Second. My objection to his discussion is that he doesn’t relate his analysis to anything in the real world. The only thing he says is why he collects this information. If his post was directed only at specialists, he should have published it in an academic journal.

Finally. If there are counter arguments that undermine or refute my comments, educate me. I don’t have any problem with someone explaining why i am wrong.

Jon May

Expand full comment

Sebastian's corrections to the SAT are wrong in your context.

If I sampled a random person in the population and told you their SAT was +2std above average, then yes, Sebastian is right that this shouldn't correspond to IQ 130 due to the imperfect correlation.

But if I first fixed a secret target IQ level L (maybe 120 or 130, you don't know it, let's say I picked it from a prior that's uniform(110,140)), and then I sampled people who have IQ levels close to L (say, L plus some error) and told you the average SAT score, you should very much NOT be regressing that down.

The second situation is basically your situation: you want to estimate the average IQ of a fixed group like LessWrongers, and that fixed group is expected to have IQ distribution which is centered at L>>100. You should not be regressing the SAT scores like that!

Expand full comment
Mar 20·edited Mar 20

Also, I just realized that ClearerThinking's survey is the same one from which Spencer Greenberg posted this thread:

https://twitter.com/SpencrGreenberg/status/1767207372777308464

That thread became a meme due to how bad the IQ questions are. If you believe IQ is real and important, read that thread and see if you change your mind :)

(I do believe IQ is real and important, just less important than IQcels think. Even I had to do a double-take at those questions, though. It reads like parody. Check it out.)

Expand full comment
Mar 20·edited Mar 20

Speaking of people underestimating how dumb the average person is, there was this opinion piece on CNN ( https://www.cnn.com/2024/03/19/opinions/zone-of-interest-holocaust-movie-rutland/index.html ) complaining how the movie "The Zone of Interest" (a Holocaust film) was humanizing Nazis and ignoring the suffering of the Jews. That alone is deserving of mockery for blatantly missing the point of the film, but then there's this paragraph:

> “The Zone of Interest” is rather tedious as a film. It barely has a plot, and the conversations and daily routines are repetitious. Several scenes will leave viewers confused, such as the one where Höss finds a jawbone while fishing in the river and drags his kids out of the water. I would not have known what was happening except I had previously read in a review that there are supposedly human remains being dumped in the river.

<

...I still can't come to terms with the fact that a human being actually wrote those sentences. Like, first he had to be stupid enough to not realize the obvious implications of human remains being in the river in a fucking Holocaust movie, then he had to be stupid enough to actually admit that he didn't understand that obvious implication without realizing how absurdly stupid it made him look. And this isn't just some guy CNN found on the street, he's a COLLEGE PROFESSOR. (And according to the editor, "an expert in the politics of contemporary Russia".) ...All of this is legitimately making me want to kill myself. How the hell am I the same species as this person?

Expand full comment
Mar 21·edited Mar 21

The only formal IQ test results I have are from a test I took as a child. Such tests are notoriously unreliable, and that was a long time ago, but if that's what you ask for, that's what you're going to get. I assume many others are in the same boat.

Expand full comment

I am sure someone mentioned this, but I couldn’t find it in the comments: in 1995 the College Board did a major recentering of the SAT which moved Verbal scores massively. A 730V (looking in mirror) before 1995 was equivalent to 800 after recentering. 780 went to 800. So any mapping to IQ must specify which scales is used.

Expand full comment

Be careful that you don't look too deep Scott, you wouldn't want to discover the IQ score at which people start selecting *out* of our community. ;-)

Expand full comment

While I concede that IQ measurements are mildly interesting for social research, I for one can not fathom making a fetish out of one's IQ score. Given that getting a measurement of IQ is actual work, much more involved than measuring one's height or weight, I would argue that in most cases where people set out to measure their IQ there is some degree of fetish involved.

In my social circles nobody has ever brought up their IQ. If anyone did, I think it would be met similarly to them mentioning them the length of their penis -- incredulity at someone going through all the troubles of measuring that and then having the poor taste to mention it.

Personally, I don't feel that my intelligence is a bottleneck in most endeavors I might undertake. Real life is not like D&D where you can just win your battles by throwing INT-based fireballs. Take it from me, in real life, having dumpstats like Conscientiousness, Drive, non-Depression (same as Drive?), Communication and so on really limits what you can achieve, more like a few extra points of IQ would likely help. (Not that I would know, nobody ever cast Fox's Cunning on me.)

Expand full comment

Bryan Caplan's comment on X: "Very interesting piece on SAT --> IQ conversions. My own view is that the SAT *is* an IQ test. The fact that it only correlates at .85 with official IQ tests shows nothing, because that's roughly the correlation between different IQ tests, too."

Expand full comment
Mar 21·edited Mar 21

As someone in the category where I've been administered exactly one respectable IQ test in my life and reported it on your past surveys, I'd suggest asking for "age when tested". I was tested as a child and there may be interesting effects with regression to the mean in adulthood, cognitive decline over decades, Flynn effect, etc., that you could examine if you're doing a large survey and ask for "age when tested" as well as current age. (Also I for one would state "WISC-R" if asked, but I don't know if asking for the test type would yield good data in general.)

And I suspect I'm in a common category. Probably nearly everyone who takes a respectable IQ test is being tested either by a school psychologist or by Mensa.

(Finally, I echo various people's point that "only the smartest people know the result of a respectable IQ test performed on them" is probably even stronger than "only the smartest people remember their SAT score". In normal life, there's no reason to administer an IQ test without some existing suspicion that the subject is either unusually smart or unusually dumb, but then dumb people rarely end up taking the ACX survey.)

Expand full comment

There is no evidence, nor could there be any evidence, that IQ correlates to job performance if for no other reason than outcomes are determined by so many factors that such an evaluation could not be valid.

And what is the measure of performance? Certainly there are many occupations where a higher IQ is necessary. Anything involving the application of advance mathematics, or chemistry or physics. But performance? Just look at Boeing. When it was run by engineers and not by bean counters, doors weren’t falling off of planes. But I would bet that some of those bean counters have higher IQs than their engineer predecessors.

Selecting a doctor by their IQ would be more than reckless. I am sure you are aware that the best predictor of a successful outcome in surgery are the number and frequency of surgeries performed.

Consider an area I know something about. Expert witnesses. A prepared lawyer of lesser intelligence can destroy an expert of far greater intelligence in every field of expertise. Happens all the time. I’ve done it, and my wife can attest to the fact that I’m no genius.

How many truly smart people were taken in by Elizabeth Smart. In fact there is a real correlation between people of high intelligence and academic achievement falling victim to con artists. I know, I represent the con artists.

Show me the studies that support your contention and I will explain why they are flawed. Or maybe even made up as some very, very, very supposedly smart professors of medicine at the most elite institutions have been caught doing of late. So what is the measure of performance when people of high IQ achieve success by cheating. Pardon me if I hesitate to choose them over my University of Miami trained Internist.

Expand full comment

When I took a Mensa IQ test they told us two interesting things:

1. The test is most accurate around the 130 mark. It's what they're interested in and what it's optimized for. They said to not take low scores too seriously. (And it doesn't test above 145)

2. About 50% of test takers score above the cutoff.

I previously figured there was a selection effect, but I didn't expect it to be so large.

Expand full comment

```even official IQ tests are gobbledygook over 135.```

Of course they are. It would be outright shocking if there were commonly available (or even uncommonly available) tests which could distinguish IQ at that range.

Suppose you have a group of 50 people. 30 of them have IQs of 100. 15 have IQs of 115. 4 have IQs of 130. And exactly one has an IQ of 145.

Your job is to sort them all into their correct group. All of them are available to you at all times, and will takes as many tests as you want, to the best of their ability, under ideal conditions every time.

Outside of this group you have access to arbitrarily many people of IQs 100, 115, etc. whose IQs are known to you, so you can norm the tests to them.

The best way to write the test is to have three groups of questions. Questions which an IQ 115+ reliably can answer, but IQ <115 reliably can't, questions that IQ 130+ can answer, and you get the idea.

But herein lies the problem. If you're not at least IQ 145, you can't write, or even *conceive* of a question that an IQ 145 person can reliably answer, but a less intelligent person can't. If IQ means anything at all, it means that.

So if you're an IQ 130 person, the best you can do is sort out the 100s, and 115s, and then everyone else is in the 130+ bucket.

There are ways to figure out through interaction that someone is smarter than you. And you could even figure out which of two people, both of whom are smarter than you, is the smarter of the pair.

But you can't write test questions that will distinguish the two, even under ideal circumstances.

Expand full comment

Has anyone mentioned the possibility that SAT scores may stop correlating with IQ above 135 not because the IQ scores are gobbledegook, but because teenagers with IQ above 135 are using their brains pretty differently than their peers, leading to all sorts of confounding academic and social issues, and this might lead to a wider range of SAT outcomes, including testing worse than a peer with an IQ of 125?

Expand full comment

"It looks like up to about 140, self-reported IQ and normed IQ rise together, and then the relationship breaks down. Sure enough, looking at the subset of self-reported IQ scores below 140, the correlation with tested IQ rises to .6, and looking at the subset above 140, the correlation is nonsignificant at -0.02. I don’t want to assert that the breakpoint is exactly 140, but I do think the test stops working somewhere in the 130 - 140 range."

There seems to be an inferential step here saying that in general, if you reapply the correlation calculation after taking a cutoff, a decrease is indicative of the test failing to measure something past that point. This is not in general true, as even in the case of applying a cutoff to randomly generated data for at some fixed correlation, this will reduce the correlation coefficient of the remaining data (on average). This is especially true when the cutoff removes more of the original datapoints or the data is sparse past the cutoff, which just from eyeballing the graph, this seems to do to a great deal. It may be the case that the test fails around this area, but this calculation provides little evidence in that direction.

Expand full comment

Note to self, add an additional 10-15 points when giving my IQ to Scott. And if he leaves a comment saying he will subtract an additional 15 points in my case, I will need to add 15 more. In fact he may read this and say nothing, so it's best to add another 15 just in case...

Expand full comment

People today are such dummies. They can't even start fires in the wilderness from scratch, navigate by the stars, grow their own food, or memorize and recite epic poetry like Beowulf or The Odyssey. All this newfangled technology, like this "writing" thing, is making morons of us all.

Expand full comment

What's wrong with assuming averages of 130s or even 140s? Have you ever been on the forums of Mensa, Triple nine, or any other high-IQ socities? If you read the kind of thoughts people have on there, you will probably be disappointed. The comment section here is way better. I don't think we're overestimaing IQ scores, I think we're overestimating how intelligent people with, say, 130 IQ actually are.

You're too humble. How many people have IQs one or two standard deviations above you? If this is the middle of the pyramid, then where the hell is the top? Can anyone name a single book written by somebody "truly intelligent"?

Expand full comment

Another point re: SAT score / IQ conversions.

It's pretty common practice to take the SAT multiple times. In fact, not only that, but colleges allow you to submit your best score from each section, even if those sections are across multiple attempts.

Let's say Bob took the SAT twice. On his first attempt he got verbal 600 and math 700. On his second attempt he got verbal 550 and math 720. He would apply to schools with a 600/720. What score do you think he's likely to give when he knows someone is using it to estimate his IQ?

Expand full comment
Mar 27·edited Mar 27

I actually think the percentile-matching thing is potentially reasonable!

On the one hand, if you trust the SAT scores, think your readers are otherwise typical, and want an estimate of the IQ scores, the regression thing is correct. But ACX readers are probably not filtered strongly by SAT, with no additional selection for IQ.

On the other hand, if you want to test the hypothesis "ACX readers are as high-IQ as they claim and otherwise typical", in a sort of Bayesian way, by computing implied SAT scores and comparing to reported scores, you should actually do the regression the other way, which (I think) produces implied SAT scores *lower* than reported, and makes the IQ reports seem easily believable or even an underestimate.

And if you want to test the hypothesis "ACX readers are selected for both high IQ and high SAT", with similar amount of selection for both, matching percentiles is at least halfway reasonable. (One version of this hypothesis: readers are selected for high g, and IQ and SAT are two different noisy measures of g with similar noise. But in that case, I'd guess SAT is noisier and we should assume ACX readers are higher-percentile on IQ.)

I'm not really sure what you should actually do here, though, except note that "ACX readers are selected for some factor-of-intelligence which is independently correlated with both IQ and SAT" is consistent with the numbers.

(At the very least, if a population reports being comparably high-percentile on two different tests, you probably shouldn't immediately assume they're lying about one of them!)

On the gripping hand: the main use case for that website -- "I know my SAT score, how high is my probable IQ?" -- has exactly the flaw you describe, with exactly the solution you describe.

[disclaimer: haven't read the rest of the post yet]

Expand full comment

In my Big Kink Survey I ask if people have ever had an officially-administered IQ test; if they say yes, they get a question where I ask what their score was. The average reported IQ for this is around 128. I forget my sample for this exactly (would have to load up the giant data again which takes forever) but iirc it was around 30k people.

The people taking this are like... US-based young very-online people, mostly coming from tiktok. Not intellectual culture at all.

Expand full comment

"Third, it has a ceiling effect: you can’t get more education than being a PhD, so it will underestimate the intelligence of a sample where a significant percent of subjects have PhDs or other advanced degrees."

This is wrong. Professional degree holders tend to be smarter than holders of adjacent PhDs. It's a bit complicated because physics PhDs are obviously smarter than lawyers, but MD>hardsci PhD>Lawyer>humanities/socsci PhD. So you could education norm to higher levels by separating out professional degrees and/or types of PhD.

Expand full comment