Data must be interpreted in order to add meaning.
We can interpret data by assuming a specific structure our outcome and use statistical methods to confirm or reject the assumption. The assumption is called a hypothesis and the statistical tests used for this purpose are called statistical hypothesis tests.
Whenever we want to make claims about the distribution of data or whether one set of results are different from another set of results in applied machine learning, we must rely on statistical hypothesis tests.
In this tutorial, you will discover statistical hypothesis testing and how to interpret and carefully state the results from statistical tests.
After completing this tutorial, you will know:
 Statistical hypothesis tests are important for quantifying answers to questions about samples of data.
 The interpretation of a statistical hypothesis test requires a correct understanding of pvalues and critical values.
 Regardless of the significance level, the finding of hypothesis tests may still contain errors.
Let’s get started.
Tutorial Overview
This tutorial is divided into 3 parts; they are:
 Statistical Hypothesis Testing
 Statistical Test Interpretation
 Errors in Statistical Tests
Statistical Hypothesis Testing
Data alone is not interesting. It is the interpretation of the data that we are really interested in.
In statistics, when we wish to start asking questions about the data and interpret the results, we use statistical methods that provide a confidence or likelihood about the answers. In general, this class of methods is called statistical hypothesis testing, or significance tests.
The term “hypothesis” may make you think about science, where we investigate a hypothesis. This is along the right track.
In statistics, a hypothesis test calculates some quantity under a given assumption. The result of the test allows us to interpret whether the assumption holds or whether the assumption has been violated.
Two concrete examples that we will use a lot in machine learning are:
 A test that assumes that data has a normal distribution.
 A test that assumes that two samples were drawn from the same underlying population distribution.
The assumption of a statistical test is called the null hypothesis, or hypothesis 0 (H0 for short). It is often called the default assumption, or the assumption that nothing has changed.
A violation of the test’s assumption is often called the first hypothesis, hypothesis 1 or H1 for short. H1 is really a short hand for “some other hypothesis,” as all we know is that the evidence suggests that the H0 can be rejected.
 Hypothesis 0 (H0): Assumption of the test holds and is accepted at some level of significance.
 Hypothesis 1 (H1): Assumption of the test does not hold and is rejected at some level of significance.
Before we can accept or reject the null hypothesis, we must interpret the result of the test.
Statistical Test Interpretation
The results of a statistical hypothesis test must be interpreted for us to start making claims.
This is a point that may cause a lot of confusion for beginners and experienced practitioners alike.
There are two common forms that a result from a statistical hypothesis test may take, and they must be interpreted in different ways. They are the pvalue and critical values.
Interpret the pvalue
We describe a finding as statistically significant by interpreting the pvalue.
For example, we may find perform a normality test on a data sample and find that it is unlikely that sample of data deviates from a Gaussian distribution, effectively accepting the null hypothesis. The result of the test is accepted at a chosen level of statistical significance that we can use when describing the finding.
A statistical hypothesis test may return a value called p or the pvalue. This is a quantity that we can use to interpret or quantify the result of the test and either accept or reject the null hypothesis. This is done by comparing the pvalue to a threshold value chosen beforehand called the significance level.
The significance level is often referred to by the Greek lower case letter alpha.
A common value used for alpha is 5% or 0.05. A smaller alpha value suggests a more robust acceptance of the null hypothesis, such as 1% or 0.1%.
The pvalue is compared to the prechosen alpha value. A result is statistically significant when the pvalue is less than alpha. This signifies a change was detected: that the default hypothesis can be rejected.
 If pvalue > alpha: Accept the null hypothesis.
 If pvalue < alpha: Do not accept the null hypothesis (i.e. reject the null hypothesis).
For example, if we were performing a test of whether a data sample was normal and we calculated a pvalue of .07, we could state something like:
The test found that the data sample was normal, accepting the null hypothesis at a 5% significance level.
The significance level can be inverted by subtracting it from 1 to give a confidence level of the hypothesis given the observed sample data.

confidence level = 1 – significance level 
Therefore, statements such as the following can also be made:
The test found that the data was normal, accepting the null hypothesis at a 95% confidence level.
Common pvalue Misinterpretations
This section highlights some common misinterpretations of the pvalue in the results of statistical tests.
True or False Null Hypothesis
The interpretation of the pvalue does not mean that the null hypothesis is true or false.
It does mean that we have chosen to accept or reject the null hypothesis at a specific statistical significance level based on empirical evidence and the chosen statistical test.
You are limited to making probabilistic claims, not crisp binary or true/false claims about the result.
pvalue as Probability
A common misunderstanding is that the pvalue is a probability of the null hypothesis being true or false given the data.
In probability, this would be written as follows:
This is incorrect.
Instead, the pvalue can be thought of as the probability of the data given the prespecified assumption embedded in the statistical test.
Again, using probability notation, this would be written as:
It allows us to reason about whether or not the data fits the hypothesis. Not the other way around.
The pvalue is a measure of how likely the data sample would be observed if the null hypothesis were true.
PostHoc Tuning
It does not mean that you can resample your domain or tune your data sample and rerun the statistical test until you achieve a desired result.
It also does not mean that you can choose your pvalue after you run the test.
This is called phacking or hill climbing and will mean that the result you present will be fragile and not representative. In science, this is at best unethical, and at worst fraud.
Interpret Critical Values
Some tests do not return a pvalue.
Instead, they might return a list of critical values and their associated significance levels, as well as a test statistic.
These are usually nonparametric or distributionfree statistical hypothesis tests.
The choice of returning a pvalue or a list of critical values is really an implementation choice.
The results are interpreted in a similar way. Instead of comparing a single pvalue to a prespecified significance level, the test statistic is compared to the critical value at a chosen significance level.
 If test statistic < critical value: Accept the null hypothesis.
 If test statistic >= critical value: Reject the null hypothesis.
Again, the meaning of the result is similar in that the chosen significance level is a probabilistic decision on acceptance or rejection of the base assumption of the test given the data.
Results are presented in the same way as with a pvalue, as either significance level or confidence level. For example, if a normality test was calculated and the test statistic was compared to the critical value at the 5% significance level, results could be stated as:
The test found that the data sample was normal, accepting the null hypothesis at a 5% significance level.
Or:
The test found that the data was normal, accepting the null hypothesis at a 95% confidence level.
Errors in Statistical Tests
The interpretation of a statistical hypothesis test is probabilistic.
That means that the evidence of the test may suggest an outcome and be mistaken.
For example, if alpha was 5%, it suggests that (at most) 1 time in 20 that the null hypothesis would be mistakenly accepted or rejected because of the statistical noise in the data sample.
Given a high pvalue, it may mean that the null hypothesis is true (we got it right) or that the null hypothesis is false and some unlikely event occurred (we made a mistake). If this type of error is made, it is called a false positive. We falsely believe the null hypothesis or assumption of the statistical test.
Alternately, a low pvalue either means that the null hypothesis false (we got it right) or it is true and some rare and unlikely event has been observed (we made a mistake). If this type of error is made, it is called a false negative. We falsely believe the rejection of the null hypothesis.
Each of these two types of error has a specific name.
 Type I Error: The incorrect rejection of a true null hypothesis or a false positive.
 Type II Error: The incorrect acceptance of a false null hypothesis or a false negative.
All statistical hypothesis tests have a chance of making either of these types of errors. False findings or false disoveries are more than possible; they are probable.
Ideally, we want to choose a significance level that minimizes the likelihood of one of these errors. E.g. a very small significance level. Although significance levels such as 0.05 and 0.01 are common in many fields of science, harder sciences, such as physics, are more aggressive.
It is common to use a significance level of (3 * 10)^7 or 0.0000003, often referred to as 5sigma. This means that the finding was due to chance with a probability of 1 in 3.5 million independent repeats of the experiments. To use a threshold like this may require a much large data sample.
Nevertheless, these types of errors are always present and must be kept in mind when presenting and interpreting the results of statistical tests. It is also a reason why it is important to have findings independently verified.
Extensions
This section lists some ideas for extending the tutorial that you may wish to explore.
 Find an example of a research paper that does not present results using pvalues.
 Find an example of a research paper that presents results with statistical significance, but makes one of the common misinterpretations of pvalues.
 Find an example of a research paper that presents results with statistical significance and correctly interprets and presents the pvalue and findings.
If you explore any of these extensions, I’d love to know.
Further Reading
This section provides more resources on the topic if you are looking to go deeper.
Articles
Summary
In this tutorial, you discovered statistical hypothesis testing and how to interpret and carefully state the results from statistical tests.
Specifically, you learned:
 Statistical hypothesis tests are important for quantifying answers to questions about samples of data.
 The interpretation of a statistical hypothesis test requires a correct understanding of pvalues.
 Regardless of the significance level, the finding of hypothesis tests may still contain errors.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.