Hypothesis Test Basics

Hypothesis testing is the keystone of many statistical applications. All hypothesis tests share a simple logical structure. A null hypothesis and an alternative hypothesis are put forward about a sample or a set of samples. The test consists in calculating the probability that the null hypothesis is true given the data. The null hypothesis is rejected if the probability is less than a predetermined significance level. The probability is calculated from the distribution of the test statistic.

This logical structure is encapsulated by the HypothesisTest class.

Some tests use the data for one sample. These classes inherit from the OneSampleTest class. Other tests use data for two or more samples. These inherit from TwoSampleTest and MultiSampleTest<T>, respectively. Some tests require all sample data to calculate the test statistic. The goodness-of-fit tests fall in this category. Other tests only require summary data such as the mean and the standard deviation. Often both options are available.

Hypothesis tests work by calculating a test statistic from the given data. The test statistic follows a distribution that may depend on some properties of the data. The value of the test statistic can be obtained using the Statistic property. The corresponding probability (often called the p-value) is available through the PValue property.

Whether the null hypothesis is rejected depends on a number of factors. The significance level (SignificanceLevel property) specifies the smallest permissible probability that the null hypothesis is true. This value is always between 0 and 1. 0.05 is a common value. If the probability or p-value is less than the predetermined significance level, the null hypothesis is rejected. Different null hypotheses give rise to different ways of calculating the probability. There are three possibilities, enumerated by the HypothesisType enumeration:

For some tests, the hypothesis type is fixed. For others, it may be supplied by the user. In this case, different alternative hypothesis will give rise to different hypothesis types. The hypothesis type corresponds to the HypothesisType property.

The Reject method tests the actual hypothesis. If the p-value is less than the significance level, the null hypothesis is rejected and this method returns true. Otherwise, the null hypothesis cannot be rejected, and the method returns false. This method is overloaded. One overload takes no arguments and uses the test's current significance level as a reference. The other overload lets you specify the significance level to test against.

Many tests have a null hypothesis that states that a parameter has a specific value. For example, the z test is used to test if the mean of a sample is equal to a specific value. In these cases, a confidence interval can be calculated. The confidence interval for a specified confidence level is the interval that contains the actual value of the parameter with a probability equal to the confidence level.

Confidence intervals can be calculated using the GetConfidenceInterval method, which has two overloads. If no parameter is specified, the confidence level corresponding to the test's current significance level is used. If the significance level is 5%, the confidence level is 95%. The second overload takes the confidence level as its only argument. The confidence level is between 0 and 1. For example, a confidence level of 95% corresponds to a value of 0.95.