Prediction Function and Universal Prediction Theory

From the standpoint of machine learning (ML), an important element in creating effective predictive models is the hypothesis that defines the prediction function. This function establishes the relationship between feature variables and learning parameters. Its choice must be justified. The main task of the prediction function is to predict any linear function in the feature space with high accuracy by using a specific set of learning parameters. It is important that the prediction function is assumed to be a complete linear function of the feature variables; this is a fundamental hypothesis in this area.

A complete linear prediction function performs affine transformations of data in an affine space. Such transformations preserve affinity: Euclidean distances and line parallelism remain unchanged, although angles and distances between points may vary. This provides a unique and continuous transformation of data and prevents different points from merging or one point from splitting into two when the learning parameters are changed.

An affine transformation is a combination of a linear transformation and a shift controlled by the learning parameters. It is capable of reproducing any function up to first order in the feature space, which is critical for the predictive capacity of ML models.

In this chapter, we will examine the formulation and predictability of prediction functions in detail, moving on to the analysis of affine transformations, their properties, and their influence on model behavior. We will also provide Python examples. All code presented in this textbook, and in this chapter in particular, can be found at https://sohoware.ru/SohoBook/. The main focus will be on two aspects:

The ability of a model to predict functions in the feature space.

The characteristics of affine transformations in affine space, including their unique properties and applications in different machine-learning scenarios.

Next, we introduce the concept of an affine transformation unit (ATU), or linear prediction-function unit. Simple neural networks capable of performing affine transformations will be presented, with emphasis on their behavior and properties. We will also consider how learning parameters encode features and what makes these encodings unique, emphasizing the importance of data-to-parameter transformation.

We will discuss the extension of the ATU concept to an affine transformation assembly (ATA), and how this can lead to the development of an activation function wrapped around an ATA to form multilayer perceptrons (MLPs) or deep neural networks. This helps explain how deep networks can achieve predictability for high-order nonlinear functions.

The chapter concludes with an introduction to Universal Prediction Theory, which provides a fundamental basis for explaining the effectiveness of deep networks in prediction. This theory helps explain why deep networks can be used so effectively for predictive purposes in different areas of machine learning.

5.1 Linear Prediction Function and Affine Transformation

In artificial neural networks (ANNs), we use an affine transformation between data and learning parameters in a neuron in order, to some extent, to imitate the process of information transformation in a neurotransmitter.

An affine transformation in an ANN is a combination of a linear transformation and a shift controlled through learning parameters. Such a transformation makes it possible to model complex dependencies in data while preserving geometric properties in the feature space.

5.1.1 Linear Prediction Function: The Main Hypothesis

In machine-learning models, the hypothesis assumes that the prediction function z is defined by the equation:

Here,z(\widehat{w}; x)is interpreted as “the functionzof\widehat{w}for a givenx,” and the vectors are defined as follows:

Here:

x_i(i = 1, 2, \ldots, p)are the feature variables used in linear basis functions.

The weightsw_i(i = 1, 2, \ldots, p) \in \mathbb{R}and the biasb(also denoted asw_0) are model parameters that form vectors in the corresponding spaces. The hat over\widehat{w}indicates an extended vector that includes the biasb, thereby forming a new vector in the hypothesis space\mathbb{W}^{p+1}.

The weights and bias can be adjusted to predict any given linear function in the feature space exactly. Special attention is paid to the use of transposition for learning parameters, which indicates their representation as a matrix, in this case a matrix with one column. Features are represented as row vectors, so the matrix of learning parameters acts on the feature vector from the right.

We have deliberately combined the most commonly used formulas for the prediction functionz(\widehat{w}; x)into a single unified form. This makes it possible to clearly illustrate the relationship among all these variables. Readers are encouraged to study this formulation carefully in order to better understand the subsequent formulations in the text.

When the formulationz=xw+bis used, we call it thexw+bformulation. In the casez=\overline{x}\widehat{w}, where the biasbis integrated into\widehat{w}, we call it the\overline{x}\widehat{w}formulation. Both formulations are used interchangeably in this book because, in essence, they express the same concept. Thexw+bformulation makes it possible to separate the roles of weights and bias explicitly during analysis, whereas the\overline{x}\widehat{w}formulation is more concise in derivations and makes affine transformations explicit.

5.1.2 Predictability for Constants: The Role of Bias

When constants are predicted, the biasbis a useful tool that properly aligns the model with the target values. It compensates for systematic errors and ensures the stability and accuracy of predictions. If we setb=0and rely exclusively on the weightsw, our model loses the ability to predict simple functions such as a constant.

Consider the functiony(x)=c, wherecis a constant independent ofx. Ifx=0, then obviouslyy(x=0)=c. Now, if we want to use the equationxw+bto predictc, it is necessary thatz(w,b; x=0)=c. However, if we excludebfrom the equation, then regardless of the choice ofw, the hypothesis always predictsz=0 \cdot w=0.

This shows that without the biasb, the hypothesis can never predict the value of the constantc. Thus, a simple linear transformation using only weightswis insufficient for adequate prediction because it cannot predict basic constant values.

On the other hand, if the biasbis present in the model, we can simply setb=c. Thus, the hypothesis can successfully predict the constantc. This also indicates that the variablezmust be part of the affine space\mathbb{R}^{p+1}, which is an extended feature space. In such a space, adding a bias allows the model to account not only for the relationship between features and weights, but also for the absolute value of the target variable.

5.1.3 Predictability for Linear Functions: The Role of Weights

Understanding the role of the weightswand the biasbin linear prediction models opens up the possibility of accurately predicting a wide range of linear functions. Let us examine how this works in more detail.

Consider the linear functiony(x)=xk+c, where:

cis a constant,

kis a vector in the vector space\mathbb{W}^{p}.

Althoughkmay belong to the space\mathbb{R}^{p}, for vector operations it must be considered within the space\mathbb{W}^{p}.

The key point here is the correct choice of model parameters. If we choosew^{*}=kandb^{*}=c, we can reproduce the functiony(x)exactly with our model. This is expressed as follows:

This choice of the parametersw^{*}andb^{*}allows the model to predict the given linear function exactly. This means that any arbitrary linear function depending on variablesx \in \mathbb{X}^{p}can be predicted using the hypothesis given in the equationz(\widehat{w}; x)=\overline{x}\widehat{w}.

Together, the weightswand the biasbdetermine how the input data, that is, the variablesx, are transformed into the predicted value. The weights determine the influence of each feature on the final prediction, whereas the bias accounts for the constant component that does not depend on the input data in situations where the data are not centered, that is, where the mean value of the features is not zero. In such cases, the bias helps adjust the overall level of the predictions.

5.1.4 Predicting Linear Functions: The Machine-Learning Procedure

Using the equationz(\widehat{w}; x)=\overline{x}\widehat{w}to predict a linear function is a relatively simple process. This is because the model parameters, or learning parameters, can be chosen on the basis of observations. In a linear model, these parameters are usually weights, or coefficients, that determine how each input feature influences the predicted value. However, for more complex tasks in which the relationships between the data and the predicted values are not so obvious, or in which the data have a more complex structure, simply “choosing” parameters on the basis of observations is no longer sufficient. In such cases, a more systematic approach is required.

To find optimal learning parameters in more complex scenarios, a minimization process can be used. This process involves finding such values of the model parameters that minimize the error between predicted and true values. This time, let us use the\overline{x}\widehat{w}formulation. We rewrite the equationy(x)=xk+cas

Step 1: Defining the loss function. In machine learning, a loss function measures how well the model predicts the correct values. The loss function evaluates the difference between model predictions, represented by the hypothesis equation\overline{x}\widehat{w}, and the true values, represented by the label functiony(x)=\overline{x}\widehat{k}. The goal is to minimize this difference, making model predictions as accurate as possible.

One of the most commonly used loss functions is the squared error, known as the L2 error function. It is defined as follows:

Here:

z(\widehat{w};\overline{x})is the model prediction,

y(x)is the true value.

This function squares the difference between the predicted and true values, which makes the error always positive and increases the influence of large errors. The expanded form of this error function,

where\overline{x}\widehat{w}is the model prediction and\overline{x}\widehat{k}is the target value, or label, is obtained by multiplying the difference between predictions and true values by its transposed vector, resulting in a scalar value. Next, the equation is transformed into the quadratic form

where\widehat{w}^T-\widehat{k}^Tand(\widehat{w}-\widehat{k})are vectors of differences between model parameters and target values, and\overline{x}^T\overline{x}is a matrix composed of products of the input data\overline{x}.

The L2 error function is widely used because it strongly penalizes large errors, since the error is squared, and because it is differentiable, which is important for optimization algorithms such as gradient descent. This means that the model will seek a precise match with the data by assigning greater weight to more significant errors.

It is important to note that the choice of loss function depends on the specific task and the nature of the data. The L2 error function is one of the most common, but it is not the only possible loss function in machine learning.

Step 2: Minimizing the loss function. The goal of this step is to find the optimal values of the model parameters\widehat{w}that minimize the loss function\mathcal{L}\bigl(z(\widehat{w})\bigr). These optimal parameters provide the best fit of the model to the data.

A stationary point of a function is a point at which all partial derivatives of the function are equal to zero. In the context of a machine-learning loss function, this means that small changes in the model parameters(\widehat{w})will not increase or decrease the value of the loss function. The gradient of the loss function is the vector of all its partial derivatives. It indicates the direction of greatest increase of the function. Mathematically, the gradient of the loss functionL(\widehat{w})with respect to the parameter\widehat{w}is written as

A stationary point is reached when the gradient of the loss function is zero:

This condition means that, at this point, there is no direction in which one can move to reduce the value of the loss function. In other words, either a local minimum of the loss function has been reached or the point is a saddle point.

In the equation, the symmetry of the matrix\overline{x}^T\overline{x}is taken into account, which simplifies the calculations. Symmetry of a matrix means that transposition does not change the matrix. The result is the equation

This equation says that the optimal parameters\widehat{w}must be equal to the parameters\widehat{k}in order to minimize the loss function. However, because the matrix\overline{x}^{T}\overline{x}has rank 1, the solution is not unique. This means that there may be other values of\widehat{w}that also satisfy the equation but differ from\widehat{k}. These alternative solutions lie in the so-called null space of the matrix\overline{x}^{T}\overline{x}. Because multiple solutions are possible, attention must be paid to the quality and quantity of data. Too few data points or low-quality data may lead to a nonempty null space of the matrix, meaning that many solutions may be possible. A unique solution requires a sufficient amount of high-quality data to eliminate these ambiguities.

Combining the formulas, we obtain

Thus, the second step in the machine-learning process, minimization of the loss function, is necessary to ensure model effectiveness. It involves finding a stationary point of the loss function. It is also important to take into account the potential ambiguity of solutions and to strive to collect a sufficient amount of high-quality data in order to ensure model reliability.

5.1.5 Affine Transformation

On the other hand, the equationz(\widehat{w}; x)=xw+bcan be used to perform an affine transformation, where the weightsw_i(i=1,2,\ldots,p)are responsible for the linear transformation. This means that each element of the input vector\overline{x}is multiplied by the corresponding weight, which changes the scale and orientation of the original data. The biasbis responsible for translation, or shifting, of the transformed data.

To show explicitly how this equation is used to perform an affine transformation, we carry out the following maneuver in matrix formulations. First, using each\widehat{w}_i(i=1,2,\ldots,k)and the equationz(\widehat{w}; x)=\overline{x}\widehat{w}, we obtain

This equation shows how each element of the input vector\overline{x}is transformed by multiplication by the corresponding weights\widehat{w}_i. The result of this multiplication is the vectorz_i, which represents transformed data.

Now form the following vector:

The equation begins with the formation of the vectorz, which is the result of the affine transformation of the input vector\overline{x}. The vectorzis composed of the elementsz_1,z_2,\ldots,z_k, each of which is obtained by transforming the corresponding input component. Each elementz_iis computed as\overline{x}multiplied by a combination of the weightw_iand the biasb_i. Next, the weights and biases are combined into a single matrix. The bias vectorbconsists ofb_1,b_2,\ldots,b_k, andW_0=[b_1,b_2,\ldots,b_k]. Thus,\widehat{W}is an extended weight matrix that includes the biases.

At the output layer of a neural network, an additional affine transformation may not be required if the data have already been transformed in the previous layers. In this case, using\widehat{W}, which already includes the bias, allows the model to proceed directly to output computation without introducing a separate bias operation.

We can construct the following matrix operation:

or, equivalently,

In short,

This matrix equation states that the vector\overline{z}, that is, the transformed data, is obtained by multiplying the extended vector\overline{x}, that is, the input data with an added leading one, by the extended weight matrix\overline{W}. This extended weight matrix consists of the bias vectorband the weight matrixW, as well as a one and a vector of zeros, which are added to account for the affine shift.

The extended input vector\overline{x}is represented as

wherex^Tis the transposed input vector, and1is an additional dimension added to include the bias in the transformation.

The extended weight matrix\overline{W}is represented as

wherebis the bias vector,W^Tis the transposed weight matrix,0^Tis the transposed zero vector, and1is a scalar corresponding to the added dimension in the input vector.

From this follows the simplified equation\overline{z}=\overline{x}\overline{W}. This equation shows how the input vector\overline{x}is transformed into the output vector\overline{z}by an affine transformation that includes both a linear transformation, using the weight matrixW, and translation, using the bias vectorb.

In neural networks, the matrix\widehat{W}that we derived is used to perform affine transformations in hidden layers. This allows the model to perform not only linear transformations but also more complex data transformations, which are needed to identify complex nonlinear relationships in data. Neural-network layers use such transformations to ensure the correct connections between layers.

Consider several special cases:

If we set all learning parameters to zero,\widehat{W}=0, then we obtain[1\ z]=[1\ 0]. This means that any data point[1\ x]in affine space collapses to the same point[1\ 0]in another affine space.

If we setb=0andW=I, whereIis the identity matrix, then we obtain[1\ z]=[1\ x]. This means that any original point in affine space remains unchanged, that is, no transformation is performed.

If we setb=c, wherecis a constant vector, andW=I, then we obtain[1\ z]=[1\ c+x]. This means that any original point in affine space is translated byc.

If we setb=[c,0,\ldots,0]andW=[k,e_2,\ldots,e_p], wheree_iis a basis vector of\mathbb{W}^{p}, with all entries zero except for 1 in thei-th position, then we obtainz_i=x_i(i=2,\ldots,p)andz_1=xk+c.

This is the equation considered earlier,z(w^{*},b^{*};x)=xk+c=y(x). It means that prediction of a linear function in the feature space can be regarded as an affine transformation in affine space. In this representation, the vectorkinWis responsible for rotation.

5.2 Typical Data Structures

5.2.1 Networks with p Input Features and 1 Output Neuron

The equationz(\widehat{w};x)can be written in matrix form with a clearly defined dimension as follows:

Now the prediction functionz \in \overline{\mathbb{X}}^{p}is clearly defined as a function ofwandbfor anyx \in \mathbb{X}^{p}. For thei-th data pointx_i, we have

Note thatz(\widehat{w};x)is a scalar for a single data point. This means that when the prediction functionzis applied to one specific data pointx, it produces a single numerical value. For example, if a network is used to predict a price based on different features of a house, such as area, number of rooms, and so on, then for each input set describing one house, the network will produce one numerical value, the predicted price.

Also note that no further transformation is necessary forzin single-layer networks. In more complex multilayer networks, the output of one layer is often fed to the next layer, where it undergoes further transformations. However, in single-layer networks such as the one considered here, the output of the functionzdoes not need additional transformations. This simplifies the network architecture because the output signal of each neuron is directly the final output of the network. Thus, in single-layer networks, as soon as the scalar value from the functionzis obtained, it immediately becomes the final result of the network.

5.2.2 Networks with p Input Features and k Output Neurons

In hyperspace cases, we may have many, sayk, neurons at the output of the current layer. Each neuron independently performs an affine transformation based on the same dataset. Therefore, the output must be an array withkelements. The data can be structured in matrix form:

The matrix above can be written in compact matrix form with the dimension explicitly indicated:

It is worth noting that the number of neurons in each layer can be arbitrarily large because the weights assigned to each individual neuron are not tied to the weights of other neurons in the same layer. This ability to add an arbitrary number of neurons is important because it supports the realization of the universal approximation theory.

5.2.3 Networks with p Input Features, k Output Neurons, and m Data Points

For a dataset withmpoints, the data can be structured as the matrixX_{m\times p}by vertically stackingx_i. In this case,mcorresponding predictions can be made, and the matrix formulation becomes

Here, the vector or matrixBcontains the same bias vectorbfor all records. The matrix above can be written in compact matrix form with the dimension explicitly indicated:

Note that, in practice, we do not actually form the matrixZexplicitly when constructing the loss function, because the loss is usually represented as a summation overmdata points.

5.3 Demonstration Examples of Affine Transformation

We now present several examples of affine transformations. This is done as follows. For a given geometric pattern defined by a set of multiple data points\overline{X}\in\overline{\mathbb{X}}^{2}, that is, by an affine space, itsi-th row

is computed using the equationz(\widehat{w};x):

where\widehat{w}_x=[b_x,w_x]^T, in whichb_xandw_xare a given set of learning parameters in the hypothesis space\mathbb{W}^{3}, and

where\widehat{w}_y=[b_y,w_y]^T, in whichb_yandw_yare a modified set of learning parameters in\mathbb{W}^{3}.

This leads to the transformed data point\overline{Z}=[1,z_x,z_y]\in\overline{\mathbb{X}}^{2}. The procedure above, which uses an affine transformation on the original dataset\overline{X}\in\overline{\mathbb{X}}^{2}by changing\widehat{w}twice, produces the transformed dataset\overline{Z}, which lies in the same affine space\overline{\mathbb{X}}^{2}; this is an automorphism.

Now we will write code to demonstrate the similarity of the transformation described above. Because\overline{\mathbb{X}}^{2}is also a two-dimensional plane, we can conveniently plot both the original and transformed patterns together in the space\mathbb{R}^{2}, using onlyz_xandz_yfor visualization and analysis.

5.3.1 Affine Transformation of an Edge and a Rectangle

import numpy as np
import matplotlib.pyplot as plt


def affine_transformation(data, weights, bias):
    """
    Performs an affine transformation of data.

    data: input data
    weights: weight vector
    bias: bias

    Returns the transformed data.
    """
    return np.dot(data, weights) + bias


def rectangular_pattern(min_x, max_x, min_y, max_y, step, rotation_angle):
    """
    Creates a rectangular pattern in 2D space.

    min_x, max_x, min_y, max_y: rectangle boundaries
    step: spacing between points
    rotation_angle: pattern rotation angle

    Returns arrays x, y and the combined point array X.
    """
    x = np.arange(min_x, max_x + step, step)
    ymin, ymax = np.full(x.shape, min_y), np.full(x.shape, max_y)

    y = np.arange(min_y, max_y + step, step)
    xmin, xmax = np.full(y.shape, min_x), np.full(y.shape, max_x)

    x1 = np.concatenate([x, xmax, np.flip(x), xmin])
    x2 = np.concatenate([ymin, y, ymax, np.flip(y)])

    x1 = np.append(x1, (max_x + min_x) / 2)
    x2 = np.append(x2, (max_y + min_y) / 2)

    X1 = x1 * np.cos(rotation_angle) + x2 * np.sin(rotation_angle)
    X2 = x2 * np.cos(rotation_angle) - x1 * np.sin(rotation_angle)
    X = np.stack((X1, X2), axis=-1)
    return x, y, X


# Define weights and biases for different transformations.
weights_0, weights_0_1, bias_0 = [0.0, 1.0], [1.0, 0.0], 0  # Original figure.
bias_1 = 1  # Change the bias.
weights_2, weights_2_1 = [0.0, 0.5], [0.5, 0.0]  # Reduce the weights.
weights_3, weights_3_1 = [0.5, 0.3], [1.2, 0.1]  # Change the weights.

# Generate a rectangular pattern.
x, y, X = rectangular_pattern(-1.0, 1.0, 1.0, 3.0, 0.1, np.pi / 4)

# Create the plot.
plt.figure(figsize=(4.0, 4.0), dpi=90)
plt.scatter(
    affine_transformation(X, weights_0, bias_0),
    affine_transformation(X, weights_0_1, bias_0),
    label="Original",
    s=5,
    c="orange",
)
plt.scatter(
    affine_transformation(X, weights_0, bias_1),
    affine_transformation(X, weights_0_1, bias_1),
    label="Bias changed",
    s=5,
    c="blue",
)
plt.scatter(
    affine_transformation(X, weights_2, bias_0),
    affine_transformation(X, weights_2_1, bias_0),
    label="Weights reduced",
    s=5,
    c="red",
)
plt.scatter(
    affine_transformation(X, weights_3, bias_0),
    affine_transformation(X, weights_3_1, bias_0),
    label="Weights changed",
    s=5,
    c="green",
)

# Display the legend.
plt.legend(loc="center left", bbox_to_anchor=(1, 0.5))
plt.axis("scaled")  # Preserve scale ratios.
plt.show()

In this section, we consider the process of affine transformation applied to a rectangle. To begin, we create a rectangular pattern represented by a set of orange points. Then we apply affine transformations to these patterns, defined by the equationz(\widehat{w};x).

Next, we visualize the patterns before and after affine transformations. Such a representation allows us to clearly see the effects of applying different types of transformations.

Special attention should be paid to the rectangle because it consists of straight lines. This makes it an ideal object for demonstrating the affinity of a transformation, because changes in its shape and orientation are easy to notice against the original structure.

From the figure, the following observations can be made:

After an affine transformation, the original, orange, rectangular pattern is only rotated, scaled, shifted, and translated into a new position. The weightsware responsible for the results of the linear transformation, namely scaling and rotation, whilebis responsible for translation. The transformation maps a point to a point, an edge to an edge, and a quadrilateral to a quadrilateral.

The transformation preserves the ratio of lengths of parallel line segments. For example, the ratio of the two longer sides of the orange rectangle is the same as the ratio of the two longer sides of the green quadrilateral.

Parallel line segments remain parallel after an affine transformation.

The transformation does not preserve distances between points. It preserves only the ratios of distances between points lying on a straight line.

An affine transformation does not preserve angles between lines. This simple demonstration helps show how an affinely transformed pattern covers the same space by changingwandb. A purely linear transformation by itself does not change the origin and therefore has much more limited coverage.

5.3.2 A Circle under an Affine Transformation

Now let us consider an affine transformation of a circle.

import numpy as np
import matplotlib.pyplot as plt


def affine_transformation(data, weights, bias):
    """
    Performs an affine transformation of data.

    data: input data
    weights: weight vector
    bias: bias

    Returns the transformed data.
    """
    return np.dot(data, weights) + bias


def circular_pattern(radius, angle_step):
    """
    Creates a circular pattern in 2D space.

    radius: circle radius
    angle_step: angular step in radians

    Returns the x and y coordinates and the point array X.
    """
    angles = np.arange(0.0, 2 * np.pi, angle_step)
    x = radius * np.cos(angles)
    y = radius * np.sin(angles)
    X = np.column_stack((x, y))
    return x, y, X


# Define weights and biases for different transformations.
weights_0, weights_0_1, bias_0 = [0.0, 1.0], [1.0, 0.0], 0  # Original figure.
bias_1 = 1  # Change the bias.
weights_2, weights_2_1 = [0.0, 0.5], [0.5, 0.0]  # Reduce the weights.
weights_3, weights_3_1 = [0.5, 0.3], [1.2, 0.1]  # Change the weights.

# Generate a circular pattern.
x, y, X = circular_pattern(1, 0.1)

# Create the plot.
plt.figure(figsize=(4.0, 4.0), dpi=90)
plt.scatter(
    affine_transformation(X, weights_0, bias_0),
    affine_transformation(X, weights_0_1, bias_0),
    label="Original",
    s=5,
    c="orange",
)
plt.scatter(
    affine_transformation(X, weights_0, bias_1),
    affine_transformation(X, weights_0_1, bias_1),
    label="Bias changed",
    s=5,
    c="blue",
)
plt.scatter(
    affine_transformation(X, weights_2, bias_0),
    affine_transformation(X, weights_2_1, bias_0),
    label="Weights reduced",
    s=5,
    c="red",
)
plt.scatter(
    affine_transformation(X, weights_3, bias_0),
    affine_transformation(X, weights_3_1, bias_0),
    label="Weights changed",
    s=5,
    c="green",
)

# Display the legend.
plt.legend(loc="center left", bbox_to_anchor=(1, 0.5))
plt.axis("scaled")  # Preserve scale ratios.
plt.show()

As is clearly visible from the figure, after transformation, the original, orange, circle is rotated, scaled, shifted, and transformed into an ellipse. The observations we made for rectangles remain valid.

5.3.3 Koch Fractal under an Affine Transformation

Let us give one more example of an affine transformation of a complex drawing. This image is the Koch fractal.

The Koch fractal, or Koch snowflake, is an affine transformation of another one through a combination of rotation, scaling, reflection, and translation. For example, the red snowflake is an affine transformation of the dark-blue snowflake. The code for the affine transformation of the Koch fractal is given below.

import matplotlib.pyplot as plt
import numpy as np


def koch_snowflake(order, scale=10):
    """Recursive function for creating the Koch fractal."""
    if order == 0:
        return np.array([[0, 0], [1, 0], [0.5, np.sqrt(3) / 2], [0, 0]])

    points = koch_snowflake(order - 1, scale)
    new_points = []

    for i in range(len(points) - 1):
        start, end = points[i], points[i + 1]
        s = start + (end - start) / 3
        e = start + 2 * (end - start) / 3
        angle = np.pi / 3
        m = np.array(
            [
                [np.cos(angle), -np.sin(angle)],
                [np.sin(angle), np.cos(angle)],
            ]
        )
        t = s + np.dot(m, (e - s))
        new_points.extend([start, s, t, e])

    new_points.append(points[-1])
    return np.array(new_points)


def affine_transform(points, A, b):
    """Apply an affine transformation to points."""
    return np.dot(points, A.T) + b


# Create the Koch fractal.
order = 4
original_points = koch_snowflake(order)

# Apply an affine transformation.
A = np.array([[0.2, 0.1], [-0.3, 0.5]])  # Transformation matrix.
b = np.array([0.1, 0.5])  # Bias vector.
transformed_points = affine_transform(original_points, A, b)

# Draw the original fractal and its affine transformation on the same plot.
plt.figure(figsize=(8, 8))

# Original fractal.
plt.plot(original_points[:, 0], original_points[:, 1], "b-", label="Original Koch snowflake")

# Affinely transformed snowflake.
plt.plot(
    transformed_points[:, 0],
    transformed_points[:, 1],
    "r-",
    label="Affinely transformed Koch snowflake",
)

plt.title("Koch Snowflake: Original and Affinely Transformed")
plt.legend()
plt.show()

5.3.4 On the Linear Prediction Function with an Affine Transformation

The linear prediction function should not be confused with an affine transformation. In essence, they are the same hypothesis, but considered from different points of view. The linear prediction function refers to the ability of a machine-learning model to predict results on the basis of a linear combination of input features. This is important for understanding how a model generates predictions in the feature space, or, in other words, how it interprets and uses data. On the other hand, an affine transformation is a mathematical transformation that includes scaling, translation, or shifting, and rotation of objects in space. In the context of machine learning, an affine transformation can be used to transform data or features before they are used by a model.

Prediction of a linear function in the feature space can be regarded as an affine transformation in affine space. This is why we use these two terms interchangeably, while keeping this subtle distinction in mind.

5.3.5 Affine Transformation Combined with an Activation Function

When an affine transformation defined asz(w,b;x)is combined with a nonlinear activation function, the output value\varphi(z)is restricted to the range of that activation function, which leads to the loss of affine properties. However, this combination gives the system the ability to predict nonlinear dependencies because the activation functions used in machine learning are usually continuous, smooth, at least differentiable, and monotonic with respect toz.

For example, when an affinely transformed pattern is combined with a sigmoid activation function, the result\varphi(z)will be restricted to the interval(0,1). The following code can be used to demonstrate the application of affine transformations in machine learning:

import numpy as np
import matplotlib.pyplot as plt


def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))


def affine_transformation(data, weights, bias):
    return np.dot(data, weights) + bias


def rectangular_pattern(min_x, max_x, min_y, max_y, step, rotation_angle):
    x = np.arange(min_x, max_x + step, step)
    ymin, ymax = np.full(x.shape, min_y), np.full(x.shape, max_y)

    y = np.arange(min_y, max_y + step, step)
    xmin, xmax = np.full(y.shape, min_x), np.full(y.shape, max_x)

    x1 = np.concatenate([x, xmax, np.flip(x), xmin])
    x2 = np.concatenate([ymin, y, ymax, np.flip(y)])

    x1 = np.append(x1, (max_x + min_x) / 2)
    x2 = np.append(x2, (max_y + min_y) / 2)

    X1 = x1 * np.cos(rotation_angle) + x2 * np.sin(rotation_angle)
    X2 = x2 * np.cos(rotation_angle) - x1 * np.sin(rotation_angle)
    X = np.stack((X1, X2), axis=-1)
    return x, y, X


# Define weights and biases for different transformations.
weights_0, weights_0_1, bias_0 = [0.0, 1.0], [1.0, 0.0], 0  # Original figure.
bias_1 = 1  # Change the bias.
weights_2, weights_2_1 = [0.0, 0.5], [0.5, 0.0]  # Reduce the weights.
weights_3, weights_3_1 = [0.5, 0.3], [1.2, 0.1]  # Change the weights.

# Generate a rectangular pattern.
x, y, X = rectangular_pattern(-1.0, 1.0, 1.0, 3.0, 0.1, np.pi / 4)

# Create the plot.
plt.figure(figsize=(4.0, 4.0), dpi=90)
plt.scatter(
    sigmoid(affine_transformation(X, weights_0, bias_0)),
    sigmoid(affine_transformation(X, weights_0_1, bias_0)),
    label="Original",
    s=5,
    c="orange",
)
plt.scatter(
    sigmoid(affine_transformation(X, weights_0, bias_1)),
    sigmoid(affine_transformation(X, weights_0_1, bias_1)),
    label="Bias changed",
    s=5,
    c="blue",
)
plt.scatter(
    sigmoid(affine_transformation(X, weights_2, bias_0)),
    sigmoid(affine_transformation(X, weights_2_1, bias_0)),
    label="Weights reduced",
    s=5,
    c="red",
)
plt.scatter(
    sigmoid(affine_transformation(X, weights_3, bias_0)),
    sigmoid(affine_transformation(X, weights_3_1, bias_0)),
    label="Weights changed",
    s=5,
    c="green",
)
plt.title("Affine Transformation with a Sigmoid")

# Display the legend.
plt.legend(loc="center left", bbox_to_anchor=(1, 0.5))
plt.axis("scaled")  # Preserve scale ratios.
plt.show()

After analyzing the code and the figures, we can observe the following:

An affine transformation, even when it is combined with a sigmoid function, preserves the property of uniquely transforming each point and each edge. This means that each element of the input data, whether a point or an edge, corresponds to a single element in the output data. This uniqueness is preserved because the activation functions used, in this case the sigmoid, are continuous and smooth and also change monotonically.

When the sigmoid function is applied, the proportions of distances between points lying on a straight line change. This means that the linear relationships, or affinity, among the points are broken. Also, not all parallel segments remain parallel after applying the sigmoid, which indicates the introduction of nonlinearity into the transformation.

However, thanks to the nonlinear activation function, the output value\varphi(z(\widehat{w};x))becomes nonlinearly dependent on the input featuresx. This allows such a transformation to be used, for example, in logistic regression for classifying labels specified as 0 or 1 by learning the weights\widehat{w}.

Because\varphi(z(\widehat{w};x))is linearly independent of the input featuresx, the network can apply additional affine transformations to the output data of this layer when passing them to the next layer.

The function\varphi(z(\widehat{w};x))is also independent of the weights\widehat{w}used in this layer. This is important for building multilayer neural networks, or deep networks, because new weights independent of the previous layers can be used for each new layer.

We see serious distortions due to the nonlinearity of the sigmoid activation function. The point-to-point transformation is still observed, but whenzis close to 0.0 or 1.0, the original points and the transformed points are compressed closer together due to the action of the sigmoid, where its curve becomes flatter. In regions where the gradient of the sigmoid function approaches zero, changing the weightswand biasesbin a neural network during training becomes less effective. This is because small gradients lead to small changes in weights and biases during training, making it more difficult to adjust the model for accurate prediction or classification of data, especially in these boundary regions.

Thus, including a sigmoid function in an affine transformation adds nonlinearity to the model, making it more flexible and capable of handling complex relationships in data, while at the same time destroying the linear relationship between input and output data.

5.4 Parameter Encoding and the Main Learning Mechanism

5.4.1 Encoding from x to ŵ: The Data-to-Parameter Transformation Block

From the equationz(\widehat{w};x)=xw+b, we see how a dataset(x,z)can be represented or “encoded” as learning parameters\widehat{w}in the hypothesis space. Let us consider this using a concrete example. Imagine that we have straight lines on a plot, each corresponding to a point in the hypothesis space denoted by\mathbb{W}^{2}. For example, the red line corresponds to the point with coordinatesb=1andw_1=1in this space.

In machine learning, one task is to create a model that can accurately represent or predict data. This process begins with determining the optimal set of points that reflect the characteristics of our dataset. These points are model parameters, denoted byw_i. Imagine that we have data, for example, different shapes of lines or patterns, and our task is to train a model to reproduce or predict these shapes accurately.

To achieve this goal, we use the dataset to create a series of points in the hypothesis space. Each point in this space corresponds to a particular set of model parameters(w_i)that determine how the model interprets the data. For example, if we are working with linear equations, each point in the hypothesis space will represent different parameters, such as slope and intercept, that define a line.

Then, on the basis of these points, that is, the parameters, the model can reproduce or predict the behavior of the data, for example by drawing lines or patterns corresponding to the original data. This transformation of data into model parameters allows us to better understand and predict data behavior. It is important to note that the size and quality of the original dataset directly affect the model’s ability to represent these data accurately in affine space.

In addition, adjusting these parameters(w_i)makes it possible to create different prediction functions. By changing the model parameters, we can tune it so that its predictions correspond to the given labels in the dataset. This process of tuning parameters is the essence of learning in machine learning. Although real machine-learning models may be much more complex, the mechanism presented here underlies most learning methods.

Thus, in machine learning, we do not simply analyze data; we transform them into model parameters, which are then used for prediction or interpretation. This transformation of data into parameters and the subsequent tuning of these parameters form the basis of the learning process in machine learning.

5.4.2 Uniqueness of Encoding

We state that encoding a line in the\overline{X}-zspace into a point in the hypothesis space is unique.

By uniqueness, we mean that each line in the\overline{X}-zspace corresponds to only one point in the hypothesis space. To demonstrate this concept, imagine any line in the\overline{X}-zspace. Suppose this line can be represented by two different points,\widehat{w}^{(1)}and\widehat{w}^{(2)}, in the hypothesis space. Using the equationz_i=\overline{x}\widehat{w}_i, we can express this line in two ways: first as

and then as

Both equations hold for any values of\overline{x}.

Now, if we consider both expressions together, we conclude that the difference between them is equal to zero:

This indicates that

which proves the uniqueness of mapping a line to a point in the hypothesis space.

Similarly, we can consider uniqueness in the reverse direction. Suppose we have a point in the hypothesis space and want to determine whether it corresponds to a single line in the\overline{X}-zspace. Starting from this point, we might attempt to construct two different lines,z^{(1)}=\overline{x}\widehat{w}andz^{(2)}=\overline{x}\widehat{w}.

However, using the same principles and equations,z^{(1)}-z^{(2)}=0, we find that any two lines constructed from the same point in the hypothesis space are actually the same line. This is confirmed by the fact that the difference between the two such lines is zero, which means that they are identical.

These arguments emphasize uniqueness. A line defined by its slope and bias is uniquely determined by the parameters\widehat{w}, which is important for correct data encoding. Uniqueness ensures that a dataset can be effectively represented and interpreted in the context of the hypothesis of affine transformations, which in turn allows machine-learning models to learn reliably from these data.

5.4.3 Uniqueness of Encoding: Independent of the Activation Function

Note that applying an activation function to an affine transformation does not affect the uniqueness of its encoding. This is due to the properties of activation functions.

Activation functions commonly used in neural networks are strictly monotonic. This means that they either always increase or always decrease, but not both. This monotonicity property guarantees that the order of the input data is preserved after applying the activation function. As a result, if two different inputs lead to the same output in an affine transformation, then after applying a monotonic activation function they will still have a unique representation. Thus, the uniqueness property of encoding is not lost.

From this we obtain a key requirement for activation functions: they must be monotonic. This requirement is satisfied by all activation functions considered in this textbook, thereby preserving the uniqueness of affine-transformation encoding even after the transformations are modified through activation functions.

5.5 Gradient of the Prediction Function

The gradient of the prediction function indicates how the prediction function changes in response to changes in the learning parameters, such as the weightswand the biasb. The gradients of these parameters can be expressed using simple formulas:

The gradient with respect to the weights shows that the gradient of the prediction function with respect to the weights is directly proportional to the feature variablesx. This means that the stronger the influence of a feature on the prediction, the larger the gradient of the weight for that feature will be.

The gradient with respect to the bias is always equal to one. This is because, in machine-learning equations, the bias is often represented as an additional feature(x_0)whose value is always 1. Therefore, changing the bias affects the prediction by a unit amount regardless of other factors.

When regularization techniques are used, different regularization parameters can be considered for the weights and the bias. Regularization is a method used to prevent model overfitting by adding a penalty to the loss function for large weights. Given the different nature of the gradients of weights and bias, different regularization strategies can be applied to these parameters.

When the\overline{x}\widehat{w}formulation is used, we have

which can be used in automatic differentiation in machine-learning processes.

5.6 Affine Transformation Array (ATA)

This concept assumes the creation of a network in which every output neuron is connected to every input neuron. Such a structure is also known as a fully connected or densely connected network. In this network, one neuron is copied verticallyktimes, creating a complex connection structure between input and output neurons and making it possible to map from a space withpfeatures into a space withkoutputs.

The prediction functions in such an ATA network can be expressed by a special equation:

Here,zrepresents a vector of prediction functions. Each element of this vector,z_i(\widehat{w}_i;x), is a separate prediction function for the corresponding output neuron.

In this model,\widehat{w}_jis the vector of learning parameters for thej-th neuron in the output layer, including the weights(w_{0j},w_{1j},w_{2j},\ldots,w_{pj})and the bias(b_j). The bias is included in the weight vector as an additional parameter, which simplifies mathematical calculations. The matrix\widehat{W}=[b\ W]^Tcombines all these learning-parameter vectors for each neuron in the output layer. It consists of the weightsWand the bias vectorb, forming a structure of size(p+1)\times k.

When we consider the complete vector of ATA learning parameters, it is represented as

which includes the learning-parameter vectors for each of thekneurons. This vector can be represented in a flattened form of the matrix\widehat{W}. The total number of learning parameters in such a system isP=(p+1)\times k, which indicates the rapid growth of the hypothesis space in ATA networks of this type.

The equationZ(\widehat{w};x)=xW+b=\overline{x}\widehat{W}, which we used to represent prediction functions, is the matrix form of affine transformations. It shows how each prediction functionz_i(w_j,b_j)is computed independently using unique weightsw_jand a biasb_jfor each neuron. This provides the independence of each prediction function from the others, which is an important aspect in solving multidimensional regression problems using a dataset withpfeatures.

5.7 Predictability of High-Order Functions in a Deep Network

5.7.1 The Role of Activation Functions

In the process of building deep neural networks, the initial step is to apply a stack of prediction functions,

which are then modified by nonlinear activation functions. This transformation leads to the creation of a new set of features, denoted byx_i^{(new)},i=1,2,\ldots,k. These new features are distinguished by the fact that they are linearly independent of the original featuresx_i, wherei=1,2,\ldots,p. This means that they provide unique information absent from the original data. These new features are then used as input data for the next layer of the neural network, allowing a new set of learning parameters to be introduced for that layer.

Because of their nonlinearity, activation functions ensure the creation of outputs of each affine transformation assembly (ATA) layer that are linearly independent of the outputs of the previous ATA layer. This independence is key because it allows each subsequent ATA layer to process data in a more complex way than would be possible under linear dependence of outputs. This successive addition of layers leads to the formation of a deep neural network.

As a result of this multilayer process, in which each layer is a stack of affine transformations supplemented by nonlinear activation functions, an extremely complex deep neural network is formed. This, in turn, produces a complex prediction function capable of processing and analyzing data at a much deeper level.

In addition, depending on the particular transformations applied to the prediction functions, different types of neural networks can be developed for different tasks. For example, replacing affine transformations with spatial filters makes it possible to create convolutional neural networks (CNNs), which are effectively used for object detection. At the same time, the use of temporal filters can lead to recurrent neural networks (RNNs), which are suitable for analyzing time sequences. This demonstrates the flexibility and power of deep neural networks in solving various machine-learning tasks.

5.7.2 Forming a Deep Network by Combining ATAs

In this section, we consider how the new features defined in the equation

can be used as input data for subsequent layers of a deep neural network (DNN). For clarity, consider a simple DNN with architecture 4-2-3, where the numbers denote the number of neurons in each layer.

Here we use the number in parentheses to denote the layer number.

First layer (2 neurons):

Input data: 4 independent featuresx_i^{(1)}, wherei=1to4.

Process: Two affine transformationsz_i^{(1)}(i=1,2), using the4\times 2weight matrixW^{(1)}and the bias vectorb_i^{(1)}(i=1,2). Each transformation,z_1^{(1)}andz_2^{(1)}, uses its own set of weightsw_{i1}^{(1)}andw_{i2}^{(1)}, respectively, as well as the corresponding biasesb_1^{(1)}andb_2^{(1)}.

Activation function: After the affine transformations,z_1^{(1)}andz_2^{(1)}pass through the nonlinear activation function\varphi, creating two new featuresx_i^{(2)}(i=1,2). These features depend nonlinearly on the original datax_i^{(1)}.

Second layer (3 neurons):

Input data: The obtained featuresx_i^{(2)}(i=1,2)are used in three affine transformationsz_i^{(3)}(i=1to3)in the second layer. Here the2\times 3weight matrixW^{(2)}and the bias vectorb_i^{(2)}(i=1to3)are used. After this transformation, the data can be processed by a nonlinear activation function, completing the formation of the second layer.

The process described above can be repeated for additional layers, thereby increasing the depth of the neural network. The number of neurons in each layer may vary.

The hypothesis space increases through stacking and combining: stacking increases dimensionality in a multiplicative way, while combining increases it additively.

Prediction functions may reside in an extremely high-dimensional space\mathbb{W}^{P}for a DNN. For the simple 4-2-3 deep network, the dimension of the hypothesis space becomes

In general, for a networkp-q-r-k,

The vector of all trainable parameters in an MLP becomes

whereN_Lis the total number of hidden layers in the MLP.

The notation .flatten() refers to a method in the Python programming language that is usually applied to arrays or matrices, in particular NumPy arrays. This method “flattens” a multidimensional array, transforming it into a one-dimensional array. In other words, it takes an array of any dimensionality and lays out all its elements in one flat, one-dimensional array.

Note that, in practice, we may not perform the flattening described above. It is given to demonstrate the growth of the dimension of the hypothesis space. In actual computations, we can simply group the parameter arrays in a Python list and use an autograd algorithm to automatically perform the necessary forward and backward computations when training the MLP. Computations in such a high-dimensional space are performed numerically.

5.7.3 Example: A 1 → 1 → 1 Network

Consider the operation of the simplest neural network with configuration 1 → 1 → 1. This network consists of three layers, each containing only one neuron. For this model, we apply a linear prediction function analogous to the equationz(\widehat{w};x)=\overline{x}\widehat{w}, and a sigmoid activation function(\sigma)for both the hidden layer and the last layer.

The network begins at the first layer, where the input signal

which can be normalized in the range from -1 to 1, passes through the sigmoid activation function. This process is described by the equation

The output value of this layer,x^{(2)}, is then passed to the next layer.

In the second, hidden, layer, a similar transformation process occurs, which can be described by the equation

The output of this layer,x^{(3)}, is the result of the network.

For a more accurate representation of the network operation, we use the Taylor expansion of the sigmoid activation function:

which allows us to approximate the network output as a third-order polynomial of the input valuex. This representation shows how the output values of the network depend on the weight coefficients and biases used in the neurons. The constants of this polynomial can be determined by fitting, that is, by training the weights and biases of the network using a dataset.

This analysis demonstrates that a neural network with configuration 1 → 1 → 1 is capable of approximately modeling third-order functions. Unlike a simpler 1 → 1 network, which is limited to first-order functions, adding an additional layer improves the predictive capabilities of the network.

Similar principles can be applied to other types of activation functions. Using a Taylor expansion, functions of even higher orders can be approximated by restricting the higher-order terms of the series.

This emphasizes that adding layers to a neural network allows it to model more complex nonlinear behavior. This is one of the reasons deep neural networks are considered powerful tools, especially if they are trained effectively.

In conclusion, increasing the depth of a neural network can be compared with increasing the order of shape functions in physics-based models, such as the finite element method or meshfree methods. At the same time, increasing the number of neurons in a network layer is analogous to increasing the number of elements or nodes in these physical models.

5.8 Universal Prediction Theory

Universal Prediction Theory discusses how deep neural networks, or deepnets, can be created and used to solve complex prediction tasks. These networks have unique properties that make them exceptionally powerful in machine learning and artificial intelligence:

The ability of the linear prediction function, known as an affine transformation, to accurately model any function up to first order. This means that each neuron in the network can effectively process input data and perform a unique transformation of these data, regardless of whether an activation function is used.

The independence property of approximation functions, that is, affine transformations, of individual neurons from one another. This is achieved through independent connections in an ATA, or Affine Transformation Array, which allows each neuron to process information independently and contribute to the overall network output.

In each layer of a deep network, new independent features are created by nonlinear activation functions. This gives deep networks the ability to extract and process complex and non-obvious properties of data, which is especially important when working with complex tasks.

The ability of deep networks, through the sequential combination of affine transformations with nonlinear activation functions, to predict complex nonlinear functions up to an arbitrarily high order. This allows deep networks to approximate and model the behavior of complex systems, making them indispensable in many areas.

These properties are the reasons why different types of deep networks can create complex mappings from input features to target labels contained in data. Universal Prediction Theory states that a deep network with a sufficient number of layers and neurons equipped with nonlinear activation functions can be tuned to predict hidden features in data if it is trained correctly.

However, realizing this capability requires the application of several techniques. It is important to determine an appropriate deep-network structure for specific task types and to find optimal learning parameters. The quality and applicability of the trained model also depend on the quality of the dataset used, which is determined by its representativeness for the modeled problem, its correctness, its size, the distribution of data points, and the noise level.

Thus, Universal Prediction Theory emphasizes the potential of deep neural networks in solving complex tasks, confirming their importance and effectiveness in the modern world of artificial intelligence.

5.9 Nonlinear Affine Transformations

Note that, in the formulations above, the featuresx_i,i=1,2,\ldots,p, are used in affine transformations as linear basis functions. Nevertheless, it is important to understand that these basis functions can also be nonlinear. Consider, for example, a one-dimensional problem. In the case of linear approximation, the feature vector is formed as

which is the simplest form of including the original feature and the free term.

However, when second-order approximation is involved, often referred to as nonlinear regression, the feature vector is extended to the form

This allows the model to include quadratic elements, thereby increasing its ability to model more complex dependencies.

In addition, if it is known that a certain function may be useful as a basis function, it can be added to the feature vector. For example, if it is assumed that\sin(x)is important for modeling the data, it can be included in the feature vector as

This approach to using nonlinear functions as feature bases is also related to the concept of support vector machines (SVMs), where kernel functions are used to classify linearly inseparable classes. Such nonlinear feature bases or kernels are sometimes called feature functions.

In neural-network models, higher-order basis functions and enrichment functions can be used in higher-dimensional spaces. For example, in a two-dimensional space, features may include not only linear and quadratic elements but also mixed terms such asx_1x_2, as well as nonlinear functions such as\sin(x).

It is important that, in high-dimensional tasks, a feature space that includes nonlinear bases may reach extremely large dimensions. In such cases, the so-called kernel trick can be useful because it makes it possible to avoid excessive dimensional growth. This approach makes it possible to work effectively with nonlinear features while minimizing the risks associated with an excessively high-dimensional feature space.

5.10 Feature Functions in Physics-Based Models

In physics-based models, special attention is paid to the selection of feature functions that play an important role in approximating physical phenomena. These feature functions, known as basis functions, are tools for modeling field variables such as displacements, stresses, velocities, pressure, and others that obey physical laws. This emphasizes that the fundamental laws of physics, whether in strong or weak form, can be adequately represented and approximated using these basis functions.

One of the most common examples of this approach can be found in the finite element method, where higher-order polynomial basis functions are used. These functions provide a more accurate representation of physical variables and therefore make it possible to model physical processes more accurately. In smoothed finite element methods, basis functions are also used, but they are expanded through so-called enrichment functions. For example, to model a stress field with singularities in certain regions of space, the function\sqrt{r}can be added to the basis functions.

In meshfree methods, such as methods with radial basis functions (RBFs), distance-based basis functions are used. This makes it possible to model physical processes in spaces without an explicitly specified mesh, which is important in cases where traditional mesh methods are ineffective or inconvenient.

In linear mechanics problems governed by physical laws, high-order basis functions and special basis functions are often used to describe system behavior more accurately. Even though the resulting system equations remain linear with respect to the field variables, using these complex feature functions makes it possible to capture the necessary characteristics of the system. This confirms the main principle: to adequately represent system characteristics, which may be either explicitly specified through physical laws or hidden in data, feature functions or basis functions of appropriate complexity must be used. Thus, even in cases where models remain linear with respect to field variables, using complex feature functions allows higher accuracy and adequacy in modeling physical processes.