Principal Component Analysis
Principal component analysis (PCA) is a data reduction technique that expresses a data set in terms of components or combinations of variables that contribute most to the variation in the data. As a result, the total number of variables used to describe the data is reduced, at the cost of losing some of the fine-grained information in the data.
Defining PCA models
All classes related to Principal Component Analysis reside in the Numerics.NET.Statistics.Multivariate namespace. The main type is PrincipalComponentAnalysis, which represents a PCA analysis.
The PrincipalComponentAnalysis class has three constructors. The first constructor takes one argument: a Matrix<T> whose columns contain the data to be analyzed. The second constructor also takes one argument: an array of Vector<T> objects.
var matrix = Matrix.Random(100, 10);
var pca1 = new PrincipalComponentAnalysis(matrix);
var vectors = matrix.Columns.ToArray();
var pca2 = new PrincipalComponentAnalysis(vectors);The third constructor takes two arguments. The first is a IDataFrame (a DataFrame<R, C> or Matrix<T>) that contains the variables that may be used in the analysis. The second argument is an array of strings that contains the names of the variables from the collection that should be included in the analysis. The example creates a data frame from a matrix and then constructs a PCA object using a subset of columns:
var rowIndex = Index.Default(matrix.RowCount);
var allNames = new string[] { "x1", "x2", "x3",
"x4", "x5", "x6", "x7", "x8", "x9", "x10" };
var columnIndex = Index.Create(allNames);
var dataFrame = matrix.ToDataFrame(rowIndex, columnIndex);
var names = new string[] { "x1", "x2", "x3", "x8", "x9", "x10" };
var pca3 = new PrincipalComponentAnalysis(dataFrame, names);Performing the analysis
When the variables in a PCA analysis use very different scales, the principal components will give more weight to the variable with the larger values. To put all variables on an equal footing, the variables are often scaled. The ScalingMethod property determines if and how this transformation is performed. This value is of type ScalingMethod which can take on the following values:
Value | Description |
|---|---|
None | No scaling is performed. |
UnitVariance | The columns are scaled to have unit variance. This is the default. |
VectorNorm | The columns are scaled to have unit norm. |
Pareto | The columns are scaled by the square root of the standard deviation. |
Range | The training columns are scaled to [0, 1], or [-1, 1] when centered about the fitted midpoint. New observations use the same affine transformation and may fall outside this nominal interval; values are not clipped. |
Level | The columns are scaled by the column mean. |
Training columns are scaled by their maximum absolute value, or their maximum absolute deviation from the fitted mean when centered. New observations with larger magnitudes may transform to absolute values greater than 1; values are not clipped. |
The Fit method performs the actual calculations. The code below sets the scaling method for the PCA object created earlier and runs the analysis:
pca3.ScalingMethod = ScalingMethod.VectorNorm;
pca3.Fit();Results of the Analysis
Once the computations are complete, a number of properties and methods give access to the results in detail. The Components property provides access to a collection of PrincipalComponent objects that provide details about each of the principal components. The components are sorted in order of their contribution to the variance in the data, in descending order.
The VarianceProportions and CumulativeVarianceProportions properties summarize the contribution of the components. The GetVarianceThreshold method calculates how many components are needed to explain a certain proportion of the total variation in the data.
PrincipalComponent objects provide more detailed information. The Eigenvalue property returns the eigenvalue corresponding to the component. This is an absolute measure for the size of the contribution. The EigenvalueDifference property returns the difference between the eigenvalues of the component and the next most significant component. This gives another indication of the signficance of a component. The greater the difference, the more important the component is compared to the remaining components. The ProportionOfVariance and CumulativeProportionOfVariance properties give the contribution of the component to the variation in the data in relative terms. Finally, the Value property returns the component as a Vector<T>. The code below illustrates these properties:
Console.WriteLine(" # Eigenvalue Difference Contribution Contrib. %");
for (int i = 0; i < 5; i++)
{
var component = pca3.Components[i];
Console.WriteLine("{0,2}{1,12:F4}{2,11:F4}{3,14:F3}%{4,10:F3}%",
i, component.Eigenvalue, component.EigenvalueDifference,
100 * component.ProportionOfVariance,
100 * component.CumulativeProportionOfVariance);
}The ComponentMatrix property returns the components as the columns of a matrix. The ScoreMatrix property expresses the observations in terms of the components. The GetPredictions method reconstructs the model's training observations using the specified number of components, in the original feature units. For new observations, use the transformation and reconstruction workflow described below. The sample code below shows how to get the predictions for the components that explain 90% of the variation in the data:
int count = pca3.GetVarianceThreshold(0.9);
Console.WriteLine("Components needed to explain 90% of variation: {0}", count);
var prediction = pca3.GetPredictions(count);
Console.WriteLine("Predictions using {0} components:", count);
Console.WriteLine(" Pr. 1 Act. 1 Pr. 2 Act. 2 Pr. 3 Act. 3 Pr. 4 Act. 4");
for (int i = 0; i < 10; i++)
Console.WriteLine(
"{0,8:F4}{1,8:F4}{2,8:F4}{3,8:F4}"
+ "{4,8:F4}{5,8:F4}{6,8:F4}{7,8:F4}",
prediction[i, 0], matrix[i, 0],
prediction[i, 1], matrix[i, 1],
prediction[i, 2], matrix[i, 2],
prediction[i, 3], matrix[i, 3]);Transforming and reconstructing new observations
After fitting, use Transform to project an arbitrary new observation into score space. Supply the same number of features, in the same order, as the training data. The model applies the fitted centering and scaling parameters automatically; you do not need to retrieve or reproduce them. These fitted parameters apply unchanged to future observations, including values outside the original training range.
Pass these scores to InverseTransform to reconstruct the observation in its original feature units. The NumberOfComponents property selects the retained subspace for scoring and Q-residual calculation. Reconstructing the resulting scores uses those retained components. With fewer components, reconstruction generally approximates the original observation.
The Q-residual, also called squared prediction error (SPE), measures the part of an observation not represented by the retained PCA subspace. It is the squared Euclidean norm of the difference between the preprocessed observation and its reconstruction in that same preprocessing space. Use GetQResidual(Vector<Double>) for one observation, or GetQResiduals(Matrix<Double>) for a matrix with one observation per row. The latter returns one value per row and preserves the row index.
When scaling is enabled, including the default unit-variance scaling, Q is generally not the squared reconstruction error in the original physical units. Larger Q means more of the observation lies outside the retained PCA subspace; this diagnostic does not supply an automatic statistical threshold or classify an observation as an outlier.
In this example, pca is the fitted six-feature model from the preceding example. Two components are retained, and the new observations follow its feature order.
// Use the fitted six-feature model from the preceding example.
pca.NumberOfComponents = 2;
// Feature order: x1, x2, x3, x8, x9, x10.
Vector<double> observation = Vector.Create(0.2, 0.4, 0.1, 0.8, 0.5, 0.3);
Vector<double> scores = pca.Transform(observation);
Vector<double> reconstructed = pca.InverseTransform(scores);
double q = pca.GetQResidual(observation);
Console.WriteLine("Reconstruction in original units: {0}", reconstructed);
Console.WriteLine("Q (squared prediction error): {0}", q);
Matrix<double> observations = Matrix.CopyFrom(new double[,]
{
{ 0.2, 0.4, 0.1, 0.8, 0.5, 0.3 },
{ 0.6, 0.1, 0.7, 0.2, 0.4, 0.9 }
});
Vector<double> qResiduals = pca.GetQResiduals(observations);The fitted preprocessing and retained component state are preserved by Deploy() and by JSON serialization with ToJson() and FromJson(String). Deployed and deserialized models support this same new-observation scoring, reconstruction, and Q-residual workflow without retaining the training observations.