Tensor Basics

This section introduces the different types of tensors in Numerics.NET, and the basic properties shared by all tensors.

The Tensor<T> class is an abstract base class that exposes the mathematical methods and properties of tensors of arbitrary types. It also contains a default implementation for many of these methods and properties. A series of derived classes implement specific types of tensors. These are listed in the table below.

The tensor library serves a purpose that is distinctly different from the existing vector and matrix library.

A matrix and a vector are mathematical objects from the field of linear algebra, with a rich history going back centuries. In mathematical terms, a vector is a point in a vector space, and a matrix is a linear transformation between two vector spaces. The special form of matrix multiplication follows naturally as the composition of linear transformations. This mathematical formalism has given rise to many powerful algorithms and techniques across the technological spectrum.

A tensor, in general, does not have the same grounding in linear algebra. A tensor is, first and foremost, a raw multi-dimensional array of (usually) numeric values. Actually, it can be useful to think of a tensor as a stack of vectors or matrices. The main differences are summarized in the table below:

Which is most appropriate depends on the application.

There are a few properties that are shared by all tensors.

Rank

The number of dimensions of the array is called its rank. A tensor of rank 0 is a scalar. A tensor of rank 1 is a vector. A tensor of rank 2 is a matrix. You can get the rank of a tensor from its Rank property.

The dimensions of a tensor are often referred to as axes.

Shape

The shape of an array is a list of non-negative integers that specify the number of elements along each axis. The number of entries in the list equals the rank of the tensor.

By convention, the last two axes are the rows and columns, respectively. Therefore a rank 1 tensor with shape (m) is a column vector with m elements. An mxn matrix has shape (m, n). A row vector with n elements has shape (1, n).

When the size of a dimension equals one, it is called a singleton dimension.

The shape of a tensor can be accessed through its Shape property. The type of the shape is TensorShape, a read-only value type.

The Rank property returns the number of dimensions. The ElementCount property returns the total number of elements of a tensor with the given shape. The AsSpan method returns the list of dimensions as a read-only span. There is also an indexer property that directly returns the size along the specified dimension or axis.

Shapes can be created by calling its constructor with a list of sizes for each dimension. For added convenience in many situations, shapes can be implicitly converted from tuples of integers.

A number of static members return common shapes. The Scalar property returns the shape of a scalar. There are three methods to create vector shapes. The Vector returns the shape of a 1-dimensional tensor of the given size. The RowVector and ColumnVector both return rank 2 shapes for row and column vectors, respectively. The CreateFromArray method returns the shape equivalent to a given .NET array.

Shapes can also be constructed by transforming existing shapes. For example, the Append method returns a new shape with one extra dimension of the specified size. The Insert method returns a new shape with one extra dimension of the specified size inserted at the specified position. The RemoveAt method returns a new shape with one fewer dimension by removing the axis at the specified position. The SwapAxes method returns a new shape with the sizes of two dimensions exchanged. Finally, the WithDimension method returns a new shape with the dimensions of the specified axis or axes replaced with the specified size.

There are also some methods related to broadcasting which are covered in the section on that topic.

Layout

The layout of a tensor refers to the layout of the elements of a tensor in memory. For the most part this is an implementation detail that is relied on heavily internally but is important only in a limited number of scenarios that use the public API. In can have an effect on performance which should be taken into account when performance is critical.

Tensors are generally stored in a linear array. Each axis has a corresponding stride that specifies the distance between successive elements in the underlying storage array. A stride of 1 is desirable for many reasons, but (in general) only one axis can have a stride of 1.

There are two principal ways of laying out the elements of a multidimensional tensor or array in a linear fashion, called C style and Fortran style.

C style layout is the default. As the name suggests, it corresponds to the way elements of multidimensional arrays are stored in C style languages, including C#. In this layout, the last index changes the fastest, so the last axis has stride 1. For a 2x2 tensor a in C style order, the elements are stored as: a[0,0], a[0,1], a[1,0], a[1,1]. For 2-dimensional arrays, this is also called row major order because rows are stored as contiguous blocks.

Fortran style layout corresponds to the way elements of arrays are stored in Fortran. In this layout, the first index changes the fastest, so the last axis has stride 1. For a 2x2 tensor a in Fortran order, the elements are stored as: a[0,0], a[1,0], a[0,1], a[1,1]. For 2-dimensional arrays, this is also called column major order because columns are stored as contiguous blocks.

Fortran has been around for more than six decades and has been used heavily for numerical computing. As a result, a whole lot of code, even code ported to C style languages, uses column major order. It is the default layout for matrices in Numerics.NET.

Other layouts are possible, even for two-dimensional tensors. For example, it is not necessary that one of the strides is 1. Strides can even be negative!

Tensors have several properties that give more information about their memory layout. The Layout property returns an object that encapsulates much of the information about the shape of the tensor.

Several properties return a boolean value that indicate if the tensor has a certain property: IsCStyle, IsFortranStyleIsContiguous.

IsCompressed indicates whether one of the axes is compressed by only storing non-zero elements. If so, then the tensor is sparse. The CompressedAxis then returns the axis that is compressed. There can be only one such axis.