Manipulating Tensors
Tensors in Numerics.NET are versatile objects. They can represent scalars, vectors, matrices, and higher-dimensional arrays. They can be used to perform a wide range of operations, from simple arithmetic to complex linear algebra.
The previous section discussed at length how to access and modify the elements of a tensor. This section covers structural operations that can be performed on tensors. These operations include broadcasting, reshaping, transposing, and copying elements.
Broadcasting
Broadcasting refers to the process of expanding a tensor by duplicating (broadcasting) along one or more singleton dimensions. Broadcasting is a powerful technique that allows many calculations to be expressed simply and performed efficiently.
Broadcasting enables you to perform element-wise operations (like addition, multiplication, etc. discussed in the next section) on tensors of different shapes. It’s like stretching or aligning tensors so that they become compatible for these operations.
Broadcasting operations can take several forms. The simplest form is broadcasting a single tensor to match the shape of another tensor. It is also possible to broadcast two or more tensors to a common shape.
Broadcasting A Single Tensor
Broadcasting a single tensor means expanding it along one or more singleton dimensions so that its shape matches a desired shape.
Broadcasting takes place in two steps. First, the tensor is made to have the desired rank. This is done by inserting singleton (size 1) dimensions at the beginning of the tensor shape. This doesn't change the elements of the tensor. Accessing values in the tensor will be the same as before, except that some 0 indices may need to be inserted to match the new rank.
The second step is to expand or stretch any singleton dimensions (including, possibly, the ones just inserted) to match the size of the dimension in the target shape. This is done by repeating the same value along the axis. At the end of the broadcasting operation, the tensor has the desired shape.
For example, let's say we have a tensor with shape (3). We want to broadcast it to shape (2, 3). The tensor is first expanded to have shape (1, 3). Then, the first axis is expanded to have size 2.
[ 1, 2, 3 ] -> [ [ 1, 2, 3 ] ]
-> [ [ 1, 2, 3 ],
[ 1, 2, 3 ] ]Broadcast Conditions
Certain conditions must be met for a tensor to be broadcast to another shape:
The rank of the tensor must be less than or equal to the rank of the target shape.
When aligned from the last dimension to the first, the size of the tensor's dimensions must be either 1 or equal to the size of the target dimension.
When a tensor's shape meets these conditions, it is set to be broadcastable to the target shape.
Some examples: Both (1, 3) and (4, 1) are broadcastable to (4, 3), but (1, 4), (3, 1), and (5, 3) are not. (3) (a one-dimensional shape) is not broadcastable to (3, 1) because dimensions are aligned from the end.
These conditions can be interpreted as defining a partial order on the set of tensor shapes. A tensor can be broadcast to another shape if the tensor's shape is less than or equal to the target shape.
This then allows us to define the concept of compatible shapes. Two or more shapes are compatible if they can be broadcast to a common shape. Their common shape is the smallest shape they can all be broadcast to. By extension, two or more tensors are compatible under broadcasting when their shapes are compatible.
To determine if two shapes (or tensors) are compatible, the shapes are compared element-wise from the last (trailing) dimension to the first. Two dimensions are compatible when they are equal or when one of them is 1. A set of shapes (or tensors) is compatible if they are all compatible with each other.
Obviously when two shapes are equal, they are also compatible. A scalar is compatible with all other non-empty shapes.
Some more examples: The shapes (3, 1) and (1, 4) are compatible because they can both be broadcast to (3, 4). The shapes (3, 1) and (4, 1) are not compatible because the size of the first dimension is different.
Moving on to sets of three: (1, 3, 4), (3, 1, 4), and (3, 1) are compatible because they can all be broadcast to (3, 3, 4). (1, 3, 4), (3, 1, 4), and (3, 2, 4) are not compatible because the size of the second dimension is different.
Broadcasting in Operations
Broadcasting is performed automatically for all element-wise operations. In general, the shapes of the tensors that take part in the operation must satisfy broadcasting rules.
When the result of an operation is specified, then all operands must be broadcastable to the shape of the result.
This is true in particular for setting part of a tensor using an indexer, as discussed in the section on accessing tensor components. The following example sets the second column of a tensor to the value 99. The indexer produces a shape (2, 1) while the right-hand side is a scalar with a zero-dimensional shape, which is broadcast to match the shape of the left-hand side:
var a = Tensor.CreateFromArray(new[,] { { 1, 2, 3 }, { 4, 5, 6 } });
// a = [[ 1, 2, 3 ]
// [ 4, 5, 6 ]]
a[.., 1] = Tensor.CreateScalar(99);
// a = [[ 1, 99, 3 ]
// [ 4, 99, 6 ]]When the result is not specified, then all operands must be compatible, and the shape of the result is the common shape of all the operands. The following example adds a 1x3 row vector to a 2x1 column vector to produce a 2x3 tensor:
var x = Tensor.CreateFromArray(new[,] { { 10 }, { 20 } });
var y = Tensor.CreateFromArray(new[] { 1, 2, 3 });
var z = x + y;
// z = [[ 11, 12, 13 ]
// [ 21, 22, 32 ]]Some operations, like matrix multiplication, have their own specific compatibility rules. Reductions, where an operation is performed along one or more axes, also have their own rules.
Reshaping
Reshaping a tensor means changing the number of dimensions and the size of each dimension while keeping the total number of elements the same. One common use of reshaping is to convert a tensor to a different rank by inserting singleton dimensions.
The Reshape method reshapes a tensor to a specified shape. The shape can be specified either as an array of integers or as a TensorShape. As mentioned, the new shape must have the same number of elements as the original. For example, a tensor with shape (2, 3) can be reshaped to (3, 2) or (6) but not to (2, 2).
In the general case, a reshaped tensor will return a copy of the elements. Only when the elements are contiguous in memory will the reshaped tensor be a view that shares the same elements as the original tensor.
In the first example below, we create a 1D tensor with values for 0 to 11. The total number of elements is 12, so we can reshape it to a 3x4 or 4x3 tensor. In this case, the reshaped tensor is a view of the original tensor.
var a = Tensor.CreateRange(12);
// a -> [ [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 ]
var b = a.Reshape(3, 4);
// b -> [ [ 0, 1, 2, 3 ],
// [ 4, 5, 6, 7 ],
// [ 8, 9, 10, 11 ]
var c = a.Reshape(4, 3);
// c -> [ [ 0, 1, 2 ],
// [ 3, 4, 5 ],
// [ 6, 7. 8 ],
// [ 9, 10, 11 ] ]When specifying the new shape, you can use a size of -1 to indicate that the size of that dimension should be inferred from the total number of elements and the other dimensions. Continuing with the previous example, if we specify that the second dimension has 6 elements, the method will infer that the first dimension will have 2 elements:
var d = a.Reshape(-1, 6);
// d -> [ [ 0, 1, 2, 3, 4, 5 ],
// [ 6, 7. 8, 9, 10, 11 ] ]A common use of this feature is to 'flatten' a tensor to a 1D shape.
var e = b[.., 0..3];
// e -> [ [ 0, 1, 2 ],
// [ 4, 5, 6 ],
// [ 8, 9, 10 ] ]
var f = e.Reshape(-1);
// f -> [ 0, 1, 2, 4, 5, 6, 8, 9, 10 ]
var g = e.Reshape(-1, TensorElementOrder.FortranStyle);
// f -> [ 0, 4, 8, 1, 5, 9, 2, 6, 10 ]Transposing and Moving Axes
Transposing a matrix means exchanging its row and column dimensions. With only two dimensions to work with, there is only one way to transpose a matrix. Tensors can have more than two dimensions, so there are many ways to move axes around.
Moving axes does not change the elements of the tensor. It only changes the way they are accessed. Methods that move axes around always return a view of the tensor. Axes are specified by their index, starting at 0. A negative value indicates that the axis should be counted from the end.
There are four such methods. The most general is MoveAxes, which is overloaded. It takes two sets of integers as input. The first set identifies the axes that should be moved. The second set specifies the new positions of these axes. Any axes not in the first set are moved if needed to accommodate the new positions. A second overload takes a single set of integers that specifies the new positions of all axes.
In the code below, we move the first axis (with index 0) and the last one, and move them to the last and first position, respectively. The change is reflected in the shapes of the tensors:
var a = Tensor.CreateRange(3 * 4 * 5).Reshape(3, 4, 5);
// a.Shape -> (3, 4, 5)
var b = a.MoveAxes([0, -1], [-1, 0]);
// b.Shape -> (5, 4, 3)The MoveAxis method moves a single axis to a new position. It takes two arguments: the index of the axis to move and its new position.
The SwapAxes method swaps two axes. It takes the indices of the axes to swap as arguments. Other axes are not affected. Below we perform the same operation as earlier, exchanging the first and the last axes:
var c = a.SwapAxes(0, 2);
// c.Shape -> (5, 4, 3)Finally, the Transpose method swaps the last two axes of the tensor. It is equivalent to calling SwapAxes(-1, -2). It treats the tensor as a stack of matrices and transposes them. The following code shows how this works. A 2x3x4 tensor is transposed to a 2x4x3 tensor:
var d = a.Transpose();
// d.Shape -> (3, 5, 4)