My friend returned from a dinner party he had been to a couple of nights previously looking pretty pleased with himself. He revealed to me that he had taken an online test of intelligence which came back with a score of 147 (out of 200 or 150 or something he said). He even said he was going to add this to his LinkedIn profile.
This test took my friend 8 minutes to complete. He did a couple of patterns to see where the next shape would go. He was given a list of words and had to pick the best synonym for a particular word. I have only included a couple of examples of the kinds of questions that he was given to complete in order to give you an idea of what he had to do in order to receive his score.
There are a couple of key points here. First, the most important thing that you need to know about any type of intelligence test is whether or not that test is reliable. The way that psychologists define reliability is consistency. So, a test would be reliable if you took it on a Tuesday, and you got a certain score, and then you took it on a Thursday, and you got a very similar score. It would be reliable if you took it with one test administrator, and you got a similar score to what you would get from another test administrator. And it would be reliable if you were a very rested person, and you took the test, and then you took it again when you were a little bit tired, and you
What reliability actually means (and why it’s not obvious)
It follows from the above that reliability in this context refers to the consistency of results in relation to a given set of test questions. Thus a test would need to be as accurate on a Tuesday as it would be on a Thursday. It would need to be administered by a variety of examiners and the results be as consistent as possible. Above all, a test of this nature would need to be picking up a constant within you as opposed to picking up on your current mood or past experience with tests of similar nature.
A good test (of anything) is analogous to a good scale. It gives you a reading that’s roughly the same if you take it 3 times (as opposed to one time). A bad test is like flaky scales that can go up and down wildly until you realize that they’re completely inaccurate and thus utterly worthless. And please – don’t try to tell me that a high reading is more valuable than a low reading. The number of either is simply noise dressed up as signal.
Unreliable test scores are flattery. The free, casual online IQ tests have poor reliability. The tests have not been validated on large populations, the test-taker has not been tested on other occasions by other examiners, and therefore the test scores obtained from these free online tests are not reliable. The test scores are mostly flattery to keep the user engaged to take more tests, and to obtain more scores to display on the website. The incentive structure of these websites is against accurate test scores.
Validity is the other half of the problem
In short, reliability of a test is not sufficient on its own to establish a test’s utility. A test can be very reliable (i.e. its results are consistent) but it can also fail to measure what it purports to measure. In addition to assessing a test’s reliability, one must also assess the test’s validity.
The test also has to be valid, i.e. it has to measure the things it claims to measure. In order to establish the validity of a test, it has to be compared with other tests that are known to measure the same constructs. In other words, the results from the test have to correlate with the results from other tests that also measure the same things. For instance, if a test is supposed to measure fluid intelligence, then the test results have to correlate with the results from other tests that are also known to measure fluid intelligence. This means that the constructs that a test measures have to correspond to real aspects of human cognition as studied by real cognitive scientists. The construct of intelligence is particularly difficult to pin down, as even the most knowledgeable and educated people have vastly different perspectives on the matter, and some even argue that it cannot be measured at all.
The bigger problem is that there are so many online tests that are not even reliable and therefore not valid. And so the score that you get from these tests is essentially just for decoration. It does not give you any reliable indication of your abilities.
What standardized scoring actually tells you
The score that you receive from a well-constructed test of cognitive ability is reported in a distribution of scores. This distribution of scores has been used to create a normed test. The normed test has been used to compare the scores of the test-taker to the scores of a large sample of individuals from a wide variety of demographics. There are individuals of all different ages and from all levels of education in the sample. Therefore, when someone receives a score of 115 on a test of cognitive ability, this individual has scored above average on the test. The individual scored above average on the test based on the performance of thousands of individuals in the sample of individuals used to create the normed test of cognitive ability.
These numbers are meant to be compared to other scores, typically from a large group of test-takers within
| Feature | Unreliable online test | Standardized cognitive assessment |
| Consistency across attempts | Low, often varies wildly | High, statistically verified |
| Normed against population | Rarely | Yes, across large samples |
| Measures validated constructs | Unlikely | Core requirement |
| Score you can actually use | Not really | Yes, for meaningful comparison |
But does any of this matter if you’re just curious?
I get that people mostly take these online tests as some form of amusement while they’re browsing around online. The curiosity about intelligence and insight into yourself and how you are are great things to explore while browsing online. However, don’t be mistaken and confuse your online scores with your real intelligence.
The biggest problem with an unreliable test is that we trust the results to define how we see ourselves. For example, someone who takes an online test receives a score of 147. This person believes that they are “so smart” because of their IQ score of 147. Unbeknownst to this person, the test they took was completely unreliable. The test could have yielded a score of 120 or lower had they taken the test under different circumstances. However, because the test produced a high score, the person believes that they are capable only of tasks that a person with an IQ of 147 can accomplish. This person will then limit their academic and professional pursuits to the scope of their perceived intelligence. Had the test produced a lower score, this person would have challenged themselves more.
So what should you actually look for?
You can tell an accurate IQ test by what they tell you about how the test was administered to others, who the test was normed on, and what your score on the test would mean to you in terms of the test’s distribution. An example of a test that you can trust is found here: accurate IQ test.
- Ask whether the test has been independently validated
- Check if results are normed against a real, documented population sample
- Look for transparency about what’s being measured and why
- Be skeptical of any test that takes under 15 minutes and returns an elite score
Follow this, it’s not that hard to do.
The number is the least interesting part
When I use the example of a bathroom scale before, I described how a really reliable test would report a score of 115. This score is above average and indicates that the person who took the test scored better than most other people who took the test. What is so special about this score, however, is that it is based on a large distribution of scores from a huge sample of the population. The test-takers that this score was compared against were of all different demographics, of all different ages, of all different amounts of formal education and so on. This test has been refined over decades of the most intense psychometric study of human cognition ever. In short, a score of 115 is above average in a meaningful way.
My friend got a 147 on one of these test, and I was too afraid to tell him that these test are basically un-useful. Recently, I suggested to him that he take another test that was a lot more reliable, valid, and meaningful. He took it, and got a score lower than his 147. But he seemed a lot happier with this test than he was with his 147! I think the reason for this is that he felt he got the real scoop on his intelligence for the first time, as opposed to being flattered by a test that was programmed to give him high numbers.
This is the important part: the score you receive from a reliable test may not be the highest, but it will be a true score, rather than a score that has been manipulated for your benefit in order to draw in customers. And, to reiterate, true is very, very important.
