Reductions and Transformations

Reductions are operations that reduce a tensor to a scalar or a tensor of lower rank. Transformations are operations that transform a tensor along one or more axes into another tensor of the same rank that may have a different size along the transformed dimensions.

Examples of reductions are: sum, product, minimum, maximum, mean, etc. Reductions don't need to be of the same type as the tensor elements. For example, counting the number of elements that are not missing is a reduction.

In its simplest form, a reduction operation turns a 1D tensor into a scalar. More generally, reductions are commonly performed along an axis where each set of elements along the axis is reduced to a single element. As a result, the rank of the tensor is reduced by one. For example, summing a matrix along the rows results in a vector. Reductions may also be performed on the entire tensor at once.

Most reductions are implemented as static extension methods on the Tensor class. They come in two variants: one that returns a scalar and one that returns a tensor. In what follows, we'll use the sum operation as an example.

Sum is implemented by the Sum method. One overload computes the sum of all elements in the tensor. It takes up to three arguments. The first is the tensor that is to be reduced. The second is a boolean tensor that specifies the elements that should be included in the sum. The third argument is a boolean that specifies whether missing values should be skipped. The method returns a scalar of the same type as the tensor elements.

The second overload computes the sum of the elements along one or more axes. It takes up to 7 arguments, all but two of which are optional. The first argument is the tensor that is to be reduced. The second argument is the axes along which the sum is to be computed.

The remaining arguments are optional. They are often specified by name. The third argument is a tensor that is to hold the result. If omitted or null, a new tensor is created. The element order of the new tensor can be specified by the order argument.

The fourth argument, keepDimensions, is a boolean that specifies whether the dimensions of the result should be kept. This is useful in a situation where the reduction is followed by another operation that involves broadcasting the result. Setting keepDimensions to true ensures that the result has the correct shape for broadcasting with the original tensor.

For example, let's say we want to center the rows of a 2D tensor around their mean. We can do this by subtracting the mean from each row. Computing the means is easy using the Mean method. We could proceed as follows:

var a = Tensor.CreateRandom((10, 5));
var means = Tensor.Mean(a, axis: 1);
// means.Shape -> (10)
// This won't work:
// var centered = a - means;

Dim a = Tensor.CreateRandom((10, 5))
Dim means = Tensor.Mean(a, axis:=1)
' means.Shape -> (10)
' This won't work:
' var centered = a - means;

No code example is currently available or this language may not be supported.

let a = Tensor.CreateRandom(TensorShape(10, 5))
let means = Tensor.Mean(a, axis=1)
// means.Shape -> (10)
// This won't work:
// let centered = a - means;
//

Ideally, we would like to use broadcasting to subtract the mean from each row. However, the vector of means has the wrong shape. Its shape is (10), but we need it to have shape (10,1) to broadcast properly. We can achieve this by setting the keepDimensions argument to true:

var means = Tensor.Mean(a, axis: 1, keepDimensions: true);
// means.Shape -> (10, 1)
var centered = a - means;

Dim means = Tensor.Mean(a, axis:=1, keepDimensions:=True)
' means.Shape -> (10, 1)
Dim centered = a - means

No code example is currently available or this language may not be supported.

let means = Tensor.Mean(a, axis=1, keepDimensions=true)
// means.Shape -> (10, 1)
let centered = a - means
//

The fifth argument, mask, is a boolean tensor that specifies the elements that should be included in the reduction. This tensor must be broadcastable with the input tensor.

The sixth argument, skipMissingValues is a boolean that specifies whether missing values should be skipped. The default is false.

The seventh argument, order, specifies the order of the elements in a newly created the result. The default is to keep the order of the input tensor.

Some reduction operations require additional arguments. For example, the Quantile method requires the quantile probability. In such a case, the additional arguments are listed immediately after the tensor arguments.

The table below lists the reduction operations.

Binary Reductions

The reduction operations listed above are all unary operations. They take one tensor as input and produce a scalar or a tensor of lower rank. There are also binary reduction operations that take two tensors as input. The reduction acts on corresponding elements of the two tensors, so the two tensors must have the same shape or must be jointly broadcastable. Examples are: dot product, covariance, correlation, etc.

The simplest binary reduction takes two 1D tensors as input and produces a scalar. More generally, each corresponding pair of sets of elements in the two input tensors is reduced to a single element. As a result, the rank of the output tensor is reduced by one.

Most binary reductions are implemented as static methods on the Tensor class. They come in two variants: one that returns a scalar and one that returns a tensor. In what follows, we'll use the dot product as an example.

The dot product is implemented by the Dot``1(Tensor<UMP>, Tensor<UMP>, Tensor<Boolean>, Boolean) method. One overload computes the sum of all elements in the tensor. It takes up to four arguments. The first two are the tensor that are being reduced. The third is a boolean tensor that specifies the elements that should be included in the sum. The fourth argument is a boolean that specifies whether missing values should be skipped. The method returns a scalar of the same type as the tensor elements.

The second overload computes the dot product of the elements along one or more axes. It takes up to 8 arguments, all but three of which are optional. The first two arguments are the tensors that are to be reduced. The third argument is the axes along which the sum is to be computed.

The remaining arguments are optional. They are often specified by name. The fourth argument is a tensor that is to hold the result. If omitted or null, a new tensor is created. The element order of the new tensor can be specified by the order argument.

The fifth argument, keepDimensions, is a boolean that specifies whether the dimensions of the result should be kept. As discussed earlier, this is useful in a situation where the reduction is followed by another operation that involves broadcasting the result. Setting keepDimensions to true ensures that the result has the correct shape for broadcasting with the original tensor.

The sixth argument, mask, is a boolean tensor that specifies the elements that should be included in the reduction. This tensor must be broadcastable with the input tensor.

The seventh argument, skipMissingValues is a boolean that specifies whether missing values should be skipped. The default is false.

The eighth argument, order, specifies the order of the elements in a newly created the result.

The table below lists the binary reduction operations.

Transformations are operations that transform a tensor along one or more axes and produce a new tensor of the same rank. The size of the transformed dimensions may be different from the original tensor. Examples of transformations that preserve the shape of the tensor are: cumulative sum and product, Fourier transforms, and sorting. Examples of transformations that change the size of the tensor along the transformed dimensions are: computing multiple quantiles and grouped reductions. These can be thought of as reductions with multiple outputs.

Most transformations are implemented as static methods in the Tensor class. In what follows, we'll use the cumulative sum operation as an example.

The cumulative sum is implemented by the CumulativeSum``1(Tensor<UMP>, Int32, Tensor<UMP>, Tensor<Boolean>, Boolean, TensorElementOrder) method. It takes up to six arguments which are mostly similar to the arguments of reduction methods. Once again, the first two arguments are required. The first is the tensor that is to be transformed. The second argument is the axis along which the sum is to be computed. Unlike reductions, transformations can only be performed along a single axis.

In its simplest form, computing the cumulative sum looks like this:.

var a = Tensor.CreateRange(5.0);
// a -> [ 0, 1, 2, 3, 4 ]
var b = Tensor.CumulativeSum(a, 0);
// b -> [ 0, 1, 3, 6, 10 ]

Dim a = Tensor.CreateRange(5.0)
' a -> [ 0, 1, 2, 3, 4 ]
Dim b = Tensor.CumulativeSum(a, 0)
' b -> [ 0, 1, 3, 6, 10 ]

No code example is currently available or this language may not be supported.

let a = Tensor.CreateRange(5.0)
// a -> [ 0, 1, 2, 3, 4 ]
let b = Tensor.CumulativeSum(a, 0)
// b -> [ 0, 1, 3, 6, 10 ]
//

The remaining four arguments are optional. They are often specified by name. The third argument, result, is a tensor that is to hold the result. If omitted or null, a new tensor is created. The element order of the new tensor can be specified by the order argument.

The fourth argument, mask, is a boolean tensor that specifies the elements that should be included in the reduction. This tensor must be broadcastable with the input tensor.

The fifth argument, skipMissingValues is a boolean that specifies whether missing values should be skipped. The default is false.

When true, the effect is identical to passing a mask argument that provides the locations where the element is not missing. Building on the earlier example, here is what happens when we set one of the elements to NaN:

a.SetValue(double.NaN, 2);
// a -> [ 0, 1, NaN, 3, 4 ]
var c = Tensor.CumulativeSum(a, 0);
// c -> [ 0, 1, NaN, NaN, NaN ]
var d = Tensor.CumulativeSum(a, 0, skipMissingValues: true);
// d -> [ 0, 1, NaN, 4, 8 ]

a.SetValue(double.NaN, 2)
' a -> [ 0, 1, NaN, 3, 4 ]
Dim c = Tensor.CumulativeSum(a, 0)
' c -> [ 0, 1, NaN, NaN, NaN ]
Dim d = Tensor.CumulativeSum(a, 0, skipMissingValues:=True)
' d -> [ 0, 1, NaN, 4, 8 ]

No code example is currently available or this language may not be supported.

a.SetValue(nan, 2)
// a -> [ 0, 1, NaN, 3, 4 ]
let c = Tensor.CumulativeSum(a, 0)
// c -> [ 0, 1, NaN, NaN, NaN ]
let d = Tensor.CumulativeSum(a, 0, skipMissingValues=true)
// d -> [ 0, 1, NaN, 4, 8 ]
//

Missing values propagate through calculations. Any operation that involves a missing value results in a missing value. In the first call, the missing value is included in the sum. Once a NaN (i.e. missing value) is encountered, the cumulative sum will be NaN as well. All elements in the result from that point on are NaN.

In the second call, the missing value is skipped. The computation of the cumulative sum is continued with the next element that is not missing. The missing value is preserved in the output.

The sixth argument, order, specifies the order of the elements in a newly created the result. The default is to keep the order of the input tensor.

The table below lists the binary reduction operations.