Testing Goodness-Of-Fit

It is often necessary to verify whether the distribution of a variable fits a certain theoretical distribution. Goodness-of-fit tests can be used to perform this verification. Goodness of fit tests require all sample values. They can't be performed using only the summary statistics.

The Chi-Square Test for Goodness-of-Fit

The chi-square ( $χ^{2}$ ) goodness-of-fit test compares observed cell frequencies from a sample with the cell frequencies expected from the proposed underlying distribution. It is used to determine if a sample comes from a population with a specific distribution.

Definition

The null hypothesis ( $H_{0}$ ) is that the observed cell frequencies ( $O_{i}$ ) are equal to the expected frequencies ( $E_{i}$ ) for all cells. Formally, $H_{0} : O_{i} = E_{i}$ for all $i$ .

The test statistic ( $χ^{2}$ ) is calculated as:

χ^{2} = \sum_{i = 1}^{k} \frac{(O_{i} - E_{i})^{2}}{E_{i}}

where $k$ is the number of cells, $O_{i}$ is the observed frequency for cell $i$ , and $E_{i}$ is the expected frequency for cell $i$ .

The distribution of the test statistic $χ^{2}$ is approximated by the chi-square distribution with $k - 1$ degrees of freedom, where $k$ is the number of cells.

Assumptions

The chi-square test assumes that the variable is categorical in nature. When the variable is continuous, the chi-square test cannot be used directly. It is possible to group the data into cells and use the categorized data in the test. The sample must be randomly selected from the population, and the expected frequency of each cell should be large enough, typically at least 5.

Applications

The chi-square goodness-of-fit test is used in various fields such as genetics, marketing, and quality control to determine if a sample comes from a population with a specific distribution.

The ChiSquareGoodnessOfFitTest class

The chi-square goodness-of-fit test is implemented by the ChiSquareGoodnessOfFitTest class. This class provides multiple constructors to accommodate different types of input data.

Samples can be provided as a vector, a categorical vector, or as histogram data. The expected distribution can be provided as a histogram or a distribution. In the case of a distribution, you can supply the number of estimated parameters, which affects the degrees of freedom of the test.

For instance, you can use the constructor that takes a categorical vector and an expected vector to perform the test on categorical data. Alternatively, you can use the constructor that takes a sample vector and a distribution to test if the sample comes from a specified continuous or discrete distribution. These techniques are illustrated below.

Example 1 - Fitting a Discrete Distribution

In a gambling game, the payout is directly proportional to the number of sixes that are thrown. A very successful customer has the following results: She threw 3 sixes twice, 2 sixes eleven times, 1 six thirty-five times, and no sixes fifty-two times.

The casino management suspects that the customer may be using weighted dice. The significance level for this test is 0.01.

The number of sixes thrown follows a binomial distribution with $p = \frac{1}{6}$ . The expected values can be calculated easily using the GetExpectedHistogram method of the BinomialDistribution. We then compare the results to the actual:

var sixesDistribution = new BinomialDistribution(3, 1 / 6.0);
var expected = sixesDistribution.GetExpectedHistogram(100);
var actual = Vector.Create<double>(51, 35, 12, 2);
var chiSquare = new ChiSquareGoodnessOfFitTest(actual, expected);
chiSquare.SignificanceLevel = 0.01;
Console.WriteLine("Test statistic: {0:F4}", chiSquare.Statistic);
Console.WriteLine("P-value:        {0:F4}", chiSquare.PValue);
Console.WriteLine("Reject null hypothesis? {0}",
    chiSquare.Reject() ? "yes" : "no");

Visual Basic

Dim sixesDistribution = New BinomialDistribution(3, 1 / 6.0)
Dim expected = sixesDistribution.GetExpectedHistogram(100)
Dim actual = Vector.Create(Of Double)(51, 35, 12, 2)
Dim chiSquare = New ChiSquareGoodnessOfFitTest(actual, expected)
chiSquare.SignificanceLevel = 0.01
Console.WriteLine("Test statistic: {0:F4}", chiSquare.Statistic)
Console.WriteLine("P-value:        {0:F4}", chiSquare.PValue)
Console.WriteLine("Reject null hypothesis? {0}",
    IIf(chiSquare.Reject(), "yes", "no"))

Visual Basic

No code example is currently available or this language may not be supported.

let sixesDistribution = BinomialDistribution(3, 1.0 / 6.0)
let expected = sixesDistribution.GetExpectedHistogram(100.0)
let actual = Vector.Create(51., 35., 12., 2.)
let chiSquare = ChiSquareGoodnessOfFitTest(actual, expected)
chiSquare.SignificanceLevel <- 0.01
printfn "Test statistic: %.4f" chiSquare.Statistic
printfn "P-value:        %.4f" chiSquare.PValue
printfn "Reject null hypothesis? %s"
    (if chiSquare.Reject() then "yes" else "no")

The value of the chi-square statistic is 9.6013 giving a p-value of 0.0223. As a result, the hypothesis that the dice are weighted is rejected at the 0.01 level.

Example 2 - Fitting a Continuous Distribution

A store manager wants to check if the waiting times at the store follow a gamma distribution. The manager collected waiting times (in minutes) from a sample of 100 customers.

We put all the observations in an array and fit the Gamma distribution using Maximum Likelihood Estimation (MLE). We do this by calling the GammaDistribution constructor with our sample, and print the parameters of the fitted distribution.

We then compare the observed frequencies to the expected frequencies using the chi-square test. We don't have to worry about binning the data. All we need to do is supply the distribution and specify that we estimated 2 distribution parameters using the appropriate ChiSquareGoodnessOfFitTest constructor. We finally print the test statistic and p-value.

The complete code looks as follows:

double[] waitingTimes = new double[]
{
    12.9, 5.3, 17.6, 19.0, 5.7, 20.5, 23.7, 20.0, 16.5, 27.9,
    4.1, 18.4, 8.0, 8.6, 9.4, 11.6, 19.8, 40.6, 0.3, 7.3, 13.2,
    14.0, 5.6, 24.7, 32.0, 3.2, 27.6, 6.0, 6.7, 19.2, 12.0,
    16.5, 3.5, 8.8, 9.8, 11.0, 14.5, 6.4, 21.9, 16.2, 8.7,
    14.3, 5.8, 19.9, 13.0, 3.8, 5.5, 13.4, 21.5, 21.3, 8.4,
    4.8, 9.4, 12.7, 14.0, 5.0, 18.5, 10.1, 5.8, 11.0, 4.7,
    17.7, 2.4, 12.8, 7.5, 18.3, 7.0, 16.9, 19.8, 10.3, 7.3,
    19.6, 0.1, 15.4, 9.6, 4.3, 9.6, 6.1, 17.2, 14.3, 3.7, 2.3,
    21.7, 22.4, 4.1, 7.5, 5.2, 17.3, 7.1, 8.3, 23.6, 19.0,
    20.4, 8.6, 15.3, 12.1, 6.3, 10.1, 14.2, 4.8
};
// The manager suspects that the waiting times follow a gamma distribution.
// We fit the distribution using Maximum Likelihood Estimation:
var gamma = new GammaDistribution(waitingTimes);
Console.WriteLine("Estimated gamma distribution:");
Console.WriteLine($"  Shape: {gamma.ShapeParameter:F3}");
Console.WriteLine($"  Scale: {gamma.ScaleParameter:F3}");

// Perform the chi-square goodness-of-fit test. We specify that
// we estimated two parameters of the distribution.
var x2Test = new ChiSquareGoodnessOfFitTest(waitingTimes, gamma, 2);

Console.WriteLine($"Chi-Square Statistic: {x2Test.Statistic:F3}");
Console.WriteLine($"P-Value: {x2Test.PValue:F4}");
Console.WriteLine($"Reject? {(x2Test.Reject(0.05) ? "yes" : "no")}");

Visual Basic

Dim waitingTimes = {
    12.9, 5.3, 17.6, 19.0, 5.7, 20.5, 23.7, 20.0, 16.5, 27.9,
    4.1, 18.4, 8.0, 8.6, 9.4, 11.6, 19.8, 40.6, 0.3, 7.3, 13.2,
    14.0, 5.6, 24.7, 32.0, 3.2, 27.6, 6.0, 6.7, 19.2, 12.0,
    16.5, 3.5, 8.8, 9.8, 11.0, 14.5, 6.4, 21.9, 16.2, 8.7,
    14.3, 5.8, 19.9, 13.0, 3.8, 5.5, 13.4, 21.5, 21.3, 8.4,
    4.8, 9.4, 12.7, 14.0, 5.0, 18.5, 10.1, 5.8, 11.0, 4.7,
    17.7, 2.4, 12.8, 7.5, 18.3, 7.0, 16.9, 19.8, 10.3, 7.3,
    19.6, 0.1, 15.4, 9.6, 4.3, 9.6, 6.1, 17.2, 14.3, 3.7, 2.3,
    21.7, 22.4, 4.1, 7.5, 5.2, 17.3, 7.1, 8.3, 23.6, 19.0,
    20.4, 8.6, 15.3, 12.1, 6.3, 10.1, 14.2, 4.8 }
' The manager suspects that the waiting times follow a gamma distribution.
' We fit the distribution using Maximum Likelihood Estimation:
Dim gamma = new GammaDistribution(waitingTimes)
Console.WriteLine("Estimated gamma distribution:")
Console.WriteLine($"  Shape: {gamma.ShapeParameter:F3}")
Console.WriteLine($"  Scale: {gamma.ScaleParameter:F3}")

' Perform the chi-square goodness-of-fit test. We specify that
' we estimated two parameters of the distribution.
Dim x2Test = New ChiSquareGoodnessOfFitTest(waitingTimes, gamma, 2)

Console.WriteLine($"Chi-Square Statistic: {x2Test.Statistic:F3}")
Console.WriteLine($"P-Value: {x2Test.PValue:F4}")
Console.WriteLine($"Reject? {Iif(x2Test.Reject(0.05), "yes", "no")}")

Visual Basic

No code example is currently available or this language may not be supported.

//#region Gof1b
    let waitingTimes = [|
        12.9; 5.3; 17.6; 19.0; 5.7; 20.5; 23.7; 20.0; 16.5; 27.9;
        4.1; 18.4; 8.0; 8.6; 9.4; 11.6; 19.8; 40.6; 0.3; 7.3; 13.2;
        14.0; 5.6; 24.7; 32.0; 3.2; 27.6; 6.0; 6.7; 19.2; 12.0;
        16.5; 3.5; 8.8; 9.8; 11.0; 14.5; 6.4; 21.9; 16.2; 8.7;
        14.3; 5.8; 19.9; 13.0; 3.8; 5.5; 13.4; 21.5; 21.3; 8.4;
        4.8; 9.4; 12.7; 14.0; 5.0; 18.5; 10.1; 5.8; 11.0; 4.7;
        17.7; 2.4; 12.8; 7.5; 18.3; 7.0; 16.9; 19.8; 10.3; 7.3;
        19.6; 0.1; 15.4; 9.6; 4.3; 9.6; 6.1; 17.2; 14.3; 3.7; 2.3;
        21.7; 22.4; 4.1; 7.5; 5.2; 17.3; 7.1; 8.3; 23.6; 19.0;
        20.4; 8.6; 15.3; 12.1; 6.3; 10.1; 14.2; 4.8 |]
    // The manager suspects that the waiting times follow a gamma distribution.
    // We fit the distribution using Maximum Likelihood Estimation:
    let gamma = GammaDistribution(waitingTimes)
    printfn "Estimated gamma distribution:"
    printfn "  Shape: %.3f" gamma.ShapeParameter
    printfn "  Scale: %.3f" gamma.ScaleParameter

    // Perform the chi-square goodness-of-fit test. We specify that
    // we estimated two parameters of the distribution.
    let x2Test = ChiSquareGoodnessOfFitTest(waitingTimes, gamma, 2)
    printfn "Chi-Square Statistic: %.3f" x2Test.Statistic
    printfn "P-Value: %.4f" x2Test.PValue
    printfn "Reject? %s" (if x2Test.Reject(0.05) then "yes" else "no")
    //#region Gof2
    let weibull = WeibullDistribution(2.0, 1.0)
    let logNormal = LognormalDistribution(0.0, 1.0)
    let logNormalSample = logNormal.Sample(25)
    let ksTest = OneSampleKolmogorovSmirnovTest(logNormalSample, weibull)
    printfn "Test statistic: %.4f" ksTest.Statistic
    printfn "P-value:        %.4f" ksTest.PValue
    printfn "Reject null hypothesis? %s"
        (if ksTest.Reject() then "yes" else "no")
    //#endregion
    //#region Gof3
    let weibullSample = weibull.Sample(25)
    let ksTest2 = TwoSampleKolmogorovSmirnovTest(logNormalSample, weibullSample)
    printfn "Test statistic: %.4f" ksTest2.Statistic
    printfn "P-value:        %.4f" ksTest2.PValue
    printfn "Reject null hypothesis? %s"
        (if ksTest2.Reject() then "yes" else "no")
    //#endregion
    //#region Gof4
    let strength = Vector.Create(
                    18.830, 20.800, 21.657, 23.030, 23.230, 24.050,
                    24.321, 25.500, 25.520, 25.800, 26.690, 26.770,
                    26.780, 27.050, 27.670, 29.900, 31.110, 33.200,
                    33.730, 33.760, 33.890, 34.760, 35.750, 35.910,
                    36.980, 37.080, 37.090, 39.580, 44.045, 45.290,
                    45.381)
    let adTest = AndersonDarlingTest(strength, 30.81, 7.38)
    printfn "Test statistic: %.4f" adTest.Statistic
    printfn "P-value:        %.4f" adTest.PValue
    printfn "Reject null hypothesis? %s"
        (if adTest.Reject() then "yes" else "no")
    //#endregion
    //#region Gof5
    let swTest = ShapiroWilkTest(strength)
    printfn "Test statistic: %.4f" swTest.Statistic
    printfn "P-value:        %.4f" swTest.PValue
    printfn "Reject null hypothesis? %s"
        (if swTest.Reject() then "yes" else "no")
    //#endregion

let Normality =
    //#region Normality1
    let sample = Vector.Create(
                    18.830, 20.800, 21.657, 23.030, 23.230, 24.050,
                    24.321, 25.500, 25.520, 25.800, 26.690, 26.770,
                    26.780, 27.050, 27.670, 29.900, 31.110, 33.200,
                    33.730, 33.760, 33.890, 34.760, 35.750, 35.910,
                    36.980, 37.080, 37.090, 39.580, 44.045, 45.290,
                    45.381)
    let shapiroWilk = ShapiroWilkTest(sample)
    let andersonDarling = AndersonDarlingTest(sample)
    let jarqueBera = JarqueBeraTest(sample)
    let dagostinoPearson = DAgostinoPearsonOmnibusTest(sample)
    //#endregion

    //#region Normality2
    let sample2 = Vector.Create(
                    18.830, 20.800, 21.657, 23.030, 23.230, 24.050,
                    24.321, 25.500, 25.520, 25.800, 26.690, 26.770,
                    26.780, 27.050, 27.670, 29.900, 31.110, 33.200,
                    33.730, 33.760, 33.890, 34.760, 35.750, 35.910,
                    36.980, 37.080, 37.090, 39.580, 44.045, 45.290,
                    45.381)
    let tests : HypothesisTest[] =
        [|
            ShapiroWilkTest(sample2) :> HypothesisTest
            AndersonDarlingTest(sample2) :> HypothesisTest
            JarqueBeraTest(sample2) :> HypothesisTest
            DAgostinoPearsonOmnibusTest(sample2) :> HypothesisTest
        |]
    for test in tests do
        printfn "%s" test.Name
        printfn "  Statistic: %.4f" test.Statistic
        printfn "  P-value:   %.4f" test.PValue
        printfn "  Reject?    %s" (if test.Reject() then "yes" else "no")
    //#endregion

    //#region Normality3
    let sample3 = Vector.Create(
                    18.830, 20.800, 21.657, 23.030, 23.230, 24.050,
                    24.321, 25.500, 25.520, 25.800, 26.690, 26.770,
                    26.780, 27.050, 27.670, 29.900, 31.110, 33.200,
                    33.730, 33.760, 33.890, 34.760, 35.750, 35.910,
                    36.980, 37.080, 37.090, 39.580, 44.045, 45.290,
                    45.381)
    let jb = JarqueBeraTest(sample3)
    let dp = DAgostinoPearsonOmnibusTest(sample3)
    printfn "Jarque-Bera skewness:        %.4f" jb.Skewness
    printfn "Jarque-Bera kurtosis:        %.4f" jb.Kurtosis
    printfn "D'Agostino skewness z:       %.4f" dp.SkewnessZScore
    printfn "D'Agostino kurtosis z:       %.4f" dp.KurtosisZScore
    //#endregion

let Heteroscedasticity =
    //#region Bartlett1
    let batch = Vector.FromFunction(100, fun i -> 1 + i / 10).AsCategorical()
    let diameter = 
        Vector.Create(
            1.006, 0.996, 0.998, 1.000, 0.992, 0.993, 1.002, 0.999, 0.994, 1.000,
            0.998, 1.006, 1.000, 1.002, 0.997, 0.998, 0.996, 1.000, 1.006, 0.988,
            0.991, 0.987, 0.997, 0.999, 0.995, 0.994, 1.000, 0.999, 0.996, 0.996,
            1.005, 1.002, 0.994, 1.000, 0.995, 0.994, 0.998, 0.996, 1.002, 0.996,
            0.998, 0.998, 0.982, 0.990, 1.002, 0.984, 0.996, 0.993, 0.980, 0.996,
            1.009, 1.013, 1.009, 0.997, 0.988, 1.002, 0.995, 0.998, 0.981, 0.996,
            0.990, 1.004, 0.996, 1.001, 0.998, 1.000, 1.018, 1.010, 0.996, 1.002,
            0.998, 1.000, 1.006, 1.000, 1.002, 0.996, 0.998, 0.996, 1.002, 1.006,
            1.002, 0.998, 0.996, 0.995, 0.996, 1.004, 1.004, 0.998, 0.999, 0.991,
            0.991, 0.995, 0.984, 0.994, 0.997, 0.997, 0.991, 0.998, 1.004, 0.997)
    let bartlett = BartlettTest(diameter, batch)
    let variables = diameter.SplitBy(batch).ToArray()
    let bartlett2 = BartlettTest(variables)
    //#endregion
    //#region Bartlett2
    printfn "Test statistic: %.4f" bartlett.Statistic
    printfn "P-value:        %.4f" bartlett.PValue
    printfn "Reject null hypothesis? %s"
        (if bartlett.Reject() then "yes" else "no")
    //#endregion
    //#region Bartlett3
    printfn "Critical value: %.4f at 95%%"
        (bartlett.GetUpperCriticalValue(0.05))
    printfn "Critical value: %.4f at 99%%"
        (bartlett.GetUpperCriticalValue(0.01))
    //#endregion

    //#region Levene1
    let levene = LeveneTest(diameter, batch)
    printfn "Test statistic: %.4f" levene.Statistic
    printfn "P-value:        %.4f" levene.PValue
    printfn "Reject null hypothesis? %s"
        (if levene.Reject() then "yes" else "no")
    //#endregion
    //#region Levene2
    printfn "Critical value: %.4f at 95%%"
        (levene.GetUpperCriticalValue(0.05))
    printfn "Critical value: %.4f at 90%%"
        (levene.GetUpperCriticalValue(0.1))
    //#endregion

    //#region Fligner1
    let fligner = FlignerKilleenTest(diameter, batch)
    printfn "Test statistic: %.4f" fligner.Statistic
    printfn "P-value:        %.4f" fligner.PValue
    printfn "Reject null hypothesis? %s"
        (if fligner.Reject() then "yes" else "no")
    //#endregion
    //#region Fligner2
    printfn "Critical value: %.4f at 95%%"
        (fligner.GetUpperCriticalValue(0.05))
    printfn "Critical value: %.4f at 90%%"
        (fligner.GetUpperCriticalValue(0.1))
    //#endregion
    0

let NonParametric =
    //#region MW1
    let males = Vector.Create(19, 22, 16, 29, 24)
    let females = Vector.Create(20, 11, 17, 12)
    let mwTest = MannWhitneyTest<int>(males, females)
    printfn "Test statistic: %.4f" mwTest.Statistic
    printfn "P-value:        %.4f" mwTest.PValue
    printfn "Reject null hypothesis? %s"
        (if mwTest.Reject() then "yes" else "no")
    //#endregion
    //#region WilcoxonSignedRank1
    let sample = Vector.Create(1.2, 0.7, -0.3, 1.8, 2.1, 0.4, -0.1)
    let signedRank1 = WilcoxonSignedRankTest(sample, 0.0)
    printfn "Statistic: %.4f" signedRank1.Statistic
    printfn "P-value:   %.4f" signedRank1.PValue
    printfn "Reject?    %s" (if signedRank1.Reject() then "yes" else "no")
    //#endregion
    //#region WilcoxonSignedRank2
    let before = Vector.Create(82.0, 91.0, 75.0, 88.0, 79.0)
    let after = Vector.Create(85.0, 93.0, 78.0, 87.0, 84.0)
    let signedRank2 = WilcoxonSignedRankTest(before, after, HypothesisType.OneTailedLower)
    printfn "W:       %.4f" signedRank2.Statistic
    printfn "W+:      %.4f" signedRank2.PositiveRankSum
    printfn "W-:      %.4f" signedRank2.NegativeRankSum
    printfn "P-value: %.4f" signedRank2.PValue
    //#endregion
    //#region KW1
    let method1 = Vector.Create(94., 87., 90., 74., 86., 97.)
    let method2 = Vector.Create(82., 85., 79., 84., 61., 72., 80.)
    let method3 = Vector.Create(89., 68., 72., 76., 69., 65.)
    let kwTest = KruskalWallisTest(method1, method2, method3)
    printfn "Test statistic: %.4f" kwTest.Statistic
    printfn "P-value:        %.4f" kwTest.PValue
    printfn "Reject null hypothesis? %s"
        (if kwTest.Reject() then "yes" else "no")
    //#endregion
    //#region Friedman1
    let data =
        Matrix.Create(5, 4,
            [|
                7.0; 9.0; 9.0; 8.0
                6.0; 8.0; 7.0; 9.0
                8.0; 9.0; 10.0; 9.0
                5.0; 7.0; 6.0; 8.0
                6.0; 8.0; 8.0; 9.0
            |],
            MatrixElementOrder.RowMajor)

    let friedman = FriedmanTest(data)

    printfn "Statistic: %.4f" friedman.Statistic
    printfn "P-value:   %.4f" friedman.PValue
    printfn "Kendall W: %.4f" friedman.KendallW
    //#endregion
    //#region Friedman2
    for treatment = 0 to friedman.TreatmentCount - 1 do
        printfn "Treatment %d: rank sum = %.4f, mean rank = %.4f"
            treatment
            (friedman.GetSumOfRanks(treatment))
            (friedman.GetMeanRank(treatment))

    printfn "Iman-Davenport F: %.4f" friedman.ImanDavenportStatistic
    printfn "Iman-Davenport p: %.4f" friedman.ImanDavenportPValue
    //#endregion
    printfn "\nRuns Test\n"
    //#region Runs1
    let densities = 
        Vector.Create(
            5.36, 5.29, 5.58, 5.65, 5.57, 5.53, 5.62, 5.29, 
            5.44, 5.34, 5.79, 5.10, 5.27, 5.39, 5.42, 5.47, 
            5.63, 5.34, 5.46, 5.30, 5.75, 5.68, 5.85)
    let rt = RunsTest<double>(densities, densities.Median())
    printfn "Test statistic: %.4f" rt.Statistic
    printfn "P-value:        %.4f" rt.PValue
    printfn "Reject null hypothesis? %s" 
        (if rt.Reject() then "yes" else "no")
    //#endregion

let Outliers =
    //#region Grubbs
    let grubbsSample = Vector.Create([| 199.31; 199.53; 200.19; 
                        200.82; 201.92; 201.95; 202.18; 245.57 |])
    let grubbs = new GrubbsTest(grubbsSample, HypothesisType.OneTailedUpper)
    printfn "Grubbs G:     %.5f" grubbs.Statistic
    printfn "       Crit.: %.5f" (grubbs.GetUpperCriticalValue())
    printfn "       Reject: %A" (grubbs.Reject())
    //#endregion
    //#region gESD1
    let sample = Vector.Create(
                    [|
                     -0.25; 0.68; 0.94; 1.15; 1.20; 1.26; 1.26; 1.34;
                     1.38; 1.43; 1.49; 1.49; 1.55; 1.56; 1.58; 1.65;
                     1.69; 1.70; 1.76; 1.77; 1.81; 1.91; 1.94; 1.96;
                     1.99; 2.06; 2.09; 2.10; 2.14; 2.15; 2.23; 2.24;
                     2.26; 2.35; 2.37; 2.40; 2.47; 2.54; 2.62; 2.64;
                     2.90; 2.92; 2.92; 2.93; 3.21; 3.26; 3.30; 3.59;
                     3.68; 4.30; 4.64; 5.34; 5.42; 6.01 
                    |])
    let test = new GeneralizedEsdTest(sample, 10, HypothesisType.TwoTailed)
    //#endregion
    //#region gESD2
    printfn "Number of outliers: %d" test.NumberOfOutliers
    let outliers = sample[test.GetOutlierIndexes()]
    printfn "Outliers: %O" outliers
    //#endregion
    //#region gESD3
    let test4 = test.GetTest(4)
    printfn "%s" (test4.Summarize())
    //#endregion
    0

let Contingencies =
    //#region CT1
    // Create categorical vectors for rows and columns  
    let rowVariable1 = Vector.CreateCategorical([| "A"; "B"; "A"; "B" |])
    let columnVariable1 = Vector.CreateCategorical([| "X"; "X"; "Y"; "Y" |])

    // Create a contingency table  
    let table1 = new ContingencyTable(rowVariable1, columnVariable1);
    //#endregion

    //#region CT2
    // Create categorical vectors for rows and columns  
    let rowVariable2 = Vector.CreateCategorical([| "A"; "B"; "A"; "B" |])
    let columnVariable2 = Vector.CreateCategorical([| "X"; "X"; "Y"; "Y" |])

    // Create a count vector  
    let countVariable = Vector.Create([| 1.0; 2.0; 3.0; 4.0 |])

    // Create a contingency table  
    let table2 = new ContingencyTable(rowVariable2, columnVariable2, countVariable)
    //#endregion

    //#region CT3
    // Create a matrix for counts  
    let counts1 = Matrix.CopyFrom(array2D [ [ 1.0; 2.0 ]; [ 3.0; 4.0 ] ])

    // Create row and column scales  
    let rowIndex = Numerics.NET.DataAnalysis.Index.Create([| "A"; "B" |])
    let columnIndex = Numerics.NET.DataAnalysis.Index.Create([| "X"; "Y" |])

    // Create a contingency table  
    let table3 = new ContingencyTable(counts1, rowIndex, columnIndex);
    //#endregion

    //#region CTProps
    // Access properties
    let  chiSquare = table2.ChiSquare
    let  phi = table2.Phi
    let  coefficientOfContingency = table2.CoefficientOfContingency
    let  cramerV = table2.CramerV

    // Display results
    printfn "Chi-Square: %f" chiSquare
    printfn "Phi: %f" phi
    printfn "Coefficient of Contingency: %f" coefficientOfContingency
    printfn "Cramer's V: %f" cramerV
    //#endregion

    //#region CTIndex
    // Using numerical index:
    let cell01 = table2[0, 1]
    // Using category names:
    let cell10 = table2["A", "X"]
    // Total for row 0:
    let cell0_ = table2[0, table2.ColumnCount]
    // Total for column 1:
    let cell_1 = table2[table2.RowCount, 1]
    // Total for the table:
    let cell__ = table2[table2.RowCount, table2.ColumnCount]

    // Display results

    // Display results
    printfn "Cell (0, 1):"
    printfn "  Expected: %f" cell01.ExpectedCount
    printfn "  Actual:   %f" cell01.Count
    printfn "  Relative frequency:"
    printfn "    In row:   %f" cell01.RelativeFrequencyInRow
    printfn "    In column:%f" cell01.RelativeFrequencyInColumn
    printfn "    In table: %f" cell01.RelativeFrequency
    //#endregion

    //#region CTTests
    let chiSquareTest = table2.GetChiSquareTest()
    printfn "Chi-Square Test: %s" (chiSquareTest.Summarize())

    let yatesCorrectedChiSquareTest = table2.GetYatesCorrectedChiSquareTest()
    printfn "Yates Corrected Chi-Square Test: %s" (yatesCorrectedChiSquareTest.Summarize())

    let likelihoodRatioTest = table2.GetLikelihoodRatioTest()
    printfn "Likelihood Ratio Test: %s" (likelihoodRatioTest.Summarize())

    let mantelHaenszelTest = table2.GetMantelHaenszelTest()
    printfn "Mantel-Haenszel Test: %s" (mantelHaenszelTest.Summarize())

    let fisherExactProbability = table2.GetFisherExactProbability()
    printfn "Fisher Exact Probability: %f" fisherExactProbability

The fitted gamma distribution has shape parameter 2.582 and scale parameter 3.705. Running the Chi square test gives a test statistic of 7.414 with a corresponding p-value of 0.1917. The manager concludes that he can't reject the hypothesis that the waiting times follow a gamma distribution at the 0.05 level.

The One Sample Kolmogorov-Smirnov Test

The one sample Kolmogorov-Smirnov test (KS test) is used to test the hypothesis that a given sample was taken from a proposed continuous distribution.

Definition

The null hypothesis ( $H_{0}$ ) is that the sample comes from the proposed continuous distribution. Formally, $H_{0} : F (x) = F_{0} (x)$ for all $x$ , where $F (x)$ is the cumulative distribution function (CDF) of the sample and $F_{0} (x)$ is the CDF of the proposed distribution.

The test statistic ( $D$ ) is defined as the maximum absolute difference between the empirical distribution function (EDF) of the sample and the CDF of the proposed distribution:

D = sup_{x} | F_{n} (x) - F_{0} (x) |

where $F_{n} (x)$ is the EDF of the sample, $F_{0} (x)$ is the CDF of the proposed distribution, and $sup$ denotes the supremum.

The distribution of the test statistic $D$ under the null hypothesis is known as the Kolmogorov distribution. The critical values for the test are obtained from this distribution.

Assumptions

The KS test can be applied to any continuous distribution. However, it can't be applied to discrete distributions and is more sensitive near the center of the distribution than at the tails. The distribution must be completely specified. If one or more of the distribution's parameters is estimated, the distribution of the test statistic is different from the Kolmogorov-Smirnov distribution.

Applications

The KS test is used in various fields such as finance, biology, and engineering to test if a sample comes from a specific continuous distribution.

The OneSampleKolmogorovSmirnovTest class

The one sample Kolmogorov-Smirnov test is implemented by the OneSampleKolmogorovSmirnovTest class. It has three constructors. The first constructor takes no arguments. All test parameters must be specified through properties of the test object. The second constructor takes two arguments. The first is a Vector<T> object that specifies the sample. The second is a Func<T, TResult> delegate, which specifies the cumulative distribution function of the distribution being tested. The third constructor also takes two arguments. The first argument is once again vector. The second argument must be of a type derived from ContinuousDistribution.

Example

In this example, we take samples of a lognormal distribution, and test whether it could come from a similar looking Weibull distribution.

var weibull = new WeibullDistribution(2, 1);
var logNormal = new LognormalDistribution(0, 1);
var logNormalSample = logNormal.Sample(25);
var ksTest = new OneSampleKolmogorovSmirnovTest(logNormalSample, weibull);
Console.WriteLine("Test statistic: {0:F4}", ksTest.Statistic);
Console.WriteLine("P-value:        {0:F4}", ksTest.PValue);
Console.WriteLine("Reject null hypothesis? {0}",
    ksTest.Reject() ? "yes" : "no");

Visual Basic

Dim weibull = New WeibullDistribution(2, 1)
Dim logNormal = New LognormalDistribution(0, 1)
Dim logNormalSample = logNormal.Sample(25)
Dim ksTest = New OneSampleKolmogorovSmirnovTest(logNormalSample, weibull)
Console.WriteLine("Test statistic: {0:F4}", ksTest.Statistic)
Console.WriteLine("P-value:        {0:F4}", ksTest.PValue)
Console.WriteLine("Reject null hypothesis? {0}",
    IIf(ksTest.Reject(), "yes", "no"))

Visual Basic

No code example is currently available or this language may not be supported.

let weibull = WeibullDistribution(2.0, 1.0)
let logNormal = LognormalDistribution(0.0, 1.0)
let logNormalSample = logNormal.Sample(25)
let ksTest = OneSampleKolmogorovSmirnovTest(logNormalSample, weibull)
printfn "Test statistic: %.4f" ksTest.Statistic
printfn "P-value:        %.4f" ksTest.PValue
printfn "Reject null hypothesis? %s"
    (if ksTest.Reject() then "yes" else "no")

First we create a Weibull and a lognormal distribution. We then get 25 random samples from the lognormal distribution using its Sample(Int32) method.

Because we use random samples, the results of the test are different on each run. The trend is that the p-value is anywhere from 0.03 to 0.3. We can conclude from this that it is not possible to distinguish a lognormal distribution from a Weibull distribution using only 25 sample points.

The Two Sample Kolmogorov-Smirnov Test

The two sample Kolmogorov-Smirnov test is used to test the hypothesis that two samples come from a population with the same, unknown distribution.

Definition

The null hypothesis ( $H_{0}$ ) is that the two samples come from the same underlying distribution. Formally, $H_{0} : F_{1} (x) = F_{2} (x)$ for all $x$ , where $F_{1} (x)$ and $F_{2} (x)$ are the cumulative distribution functions (CDFs) of the two samples.

The test statistic ( $D$ ) is defined as the maximum absolute difference between the empirical distribution functions (EDFs) of the two samples:

D = sup_{x} | F_{n 1} (x) - F_{n 2} (x) |

where $F_{n 1} (x)$ and $F_{n 2} (x)$ are the EDFs of the two samples, and $sup$ denotes the supremum.

The distribution of the test statistic $D$ under the null hypothesis is known as the Kolmogorov distribution. The critical values for the test are obtained from this distribution.

Assumptions

The two sample KS test assumes that the samples are independent and that the distributions are continuous.

Applications

The two sample KS test is used in various fields such as medicine, economics, and environmental science to compare two samples and determine if they come from the same distribution.

The TwoSampleKolmogorovSmirnovTest class

The two sample Kolmogorov-Smirnov test is implemented by the TwoSampleKolmogorovSmirnovTest class. It has two constructors. The first constructor takes no arguments. The second constructor takes two arguments. Both are vectors that specify the two samples that are being compared.

Example

We investigate whether we can distinguish a sample taken from a lognormal distribution from a sample taken from a similar looking Weibull distribution. We use the lognormal samples we created in the previous section.

var weibullSample = weibull.Sample(25);
var ksTest2 = new TwoSampleKolmogorovSmirnovTest(logNormalSample, weibullSample);
Console.WriteLine("Test statistic: {0:F4}", ksTest2.Statistic);
Console.WriteLine("P-value:        {0:F4}", ksTest2.PValue);
Console.WriteLine("Reject null hypothesis? {0}",
    ksTest2.Reject() ? "yes" : "no");

Visual Basic

Dim weibullSample = weibull.Sample(25)
Dim ksTest2 = New TwoSampleKolmogorovSmirnovTest(logNormalSample, weibullSample)
Console.WriteLine("Test statistic: {0:F4}", ksTest2.Statistic)
Console.WriteLine("P-value:        {0:F4}", ksTest2.PValue)
Console.WriteLine("Reject null hypothesis? {0}",
    IIf(ksTest2.Reject(), "yes", "no"))

Visual Basic

No code example is currently available or this language may not be supported.

let weibullSample = weibull.Sample(25)
let ksTest2 = TwoSampleKolmogorovSmirnovTest(logNormalSample, weibullSample)
printfn "Test statistic: %.4f" ksTest2.Statistic
printfn "P-value:        %.4f" ksTest2.PValue
printfn "Reject null hypothesis? %s"
    (if ksTest2.Reject() then "yes" else "no")

Normality tests

Normality testing is covered separately in Testing Normality. That page discusses the ShapiroWilkTest, AndersonDarlingTest, JarqueBeraTest, and DAgostinoPearsonOmnibusTest classes in detail.

This page remains focused on general goodness-of-fit procedures, including chi-square goodness-of-fit tests and one-sample and two-sample Kolmogorov-Smirnov tests.