There's a common perception that an article packed full of stats is more credible than one without, but statistics can be just as subjective as anything else. We've put together a guide to help you ask the big questions about those numbers.
There's a common perception that an article packed full of statistics is more accurate, and therefore more credible, than one without. The assumption is that more numbers equals better research.
On the surface this makes a lot of sense—statistics usually exist in the realms of science and mathematics, both fields that emphasise the importance of objective thought. However look past the numbers and you'll find that the world of stats can be far more subjective and nuanced than first thought.
So how do you make sure that you're not being misled by the numbers? There are some important questions you can ask to verify the accuracy and relevance of the statistics that you consume.
Question one: which statistic do you want to verify?
This question sounds obvious, but it's important to think carefully about what it is that you want to verify. Just because something is a number does not mean that it is a statistic.
Think of a statistic like a snapshot—it's just one fact, or little piece, that has been taken from a larger sample of numerical data. When we talk about fact checking a statistic, we're really talking about fact checking a claim that has been made, using a statistic to back it up.
Question two: where does the statistic come from?
Try and find the primary source of the statistic—that is, the set of data in which it originally appeared. Check that the statistic from the claim matches the one in the dataset that you've identified. If it doesn't then the claim is already inaccurate.
If possible, try to check this source against other sources to see how it compares. For example, if you're trying to fact check a statistic from a private security company which claims that crime rates in Delhi have increased over the past ten years, you could try to find similar datasets from the police service, the government, NGOs or even institutions like the United Nations.
Question three: how is the statistic being used?
Now that you've identified the source of the statistic, it's time to think about how the way this source compares to the statistic being used in your claim. It's important to understand what the is data actually saying. What conclusions can we really draw from this data? Again, this seems obvious, but it's common to find statistics being referenced out of context, or being used to draw misleading inferences.
Pay attention to any conclusions being drawn in the claim that you're checking. Do they make sense in the context of the broader dataset? For example, if the article asserts that "the UK population has hit a shocking ten-year low," using a statistic taken from national census data, how does this figure compare to trends in the same dataset over the previous twenty, thirty, or forty years? Is it as unprecedented as the article implied, or is it actually in line with long-term population trends?
Know how to calculate your mean, median and mode. This can help you to understand how your single statistic fits into the bigger picture of the dataset. Being aware of what's behind key terms is also important—if the claim talks about distribution, base rates or averages, try to find out what exactly it is that's being referred to.
Question four: how reliable is the source?
So you've identified the source of the statistic, and compared this to the claim that's being made. The final step in your fact checking process is understanding the reliability of the dataset itself.
Ask yourself who collected the data and why. Was it a national-level survey done by a global institution, or a small-scale inquiry conducted by a private company? Was the survey conducted for purely research purposes, to inform policy or business strategy, or a mixture of these?
How was it collected—what kind of methodology was used? If the data was collected from a sample, was the sample size large enough to give an accurate picture of the whole population? Does data showing an increase in the instances of infant mortality account for general population growth?
Understanding the data will help you to decide how credible a statistic is, and how relevant it is in the context of the claim that you're trying to check.
Some things to remember
With stats, context is everything. Understanding where your statistic came from and how it is being used will help you start to build a clearer picture of the claim that you're trying to check. From there, you can start to assess its credibility as you would for any other source.
While fact checking a statistic can sometimes seem like a messy and overwhelming task, there a few things to remember to help make the process easier.
- Try to anticipate the questions that you want to ask about the statistic before you go through the original data, this will help to make your inquiry more targeted,
- Learn and practice some basic mathematical processes, then apply them to the data yourself and see what you find. Calculating the mean, mode and average of a set of numbers is a great place to start,
- When in doubt, try to ask yourself if there is another possible explanation for the data behind the statistic. Remember that the claim that you're trying to check is using numbers to try and tell a story. Which other stories could explain the numbers that your seeing?
This article's header image is by Stephen Dawson from Unsplash.