Introduction

1.1 Intuitive Thinking and Machine Learning: Paths to Solving Complex Problems

We constantly encounter various problems and try to solve them using timely technologies. Each of us knows that many everyday tasks are solved spontaneously and quickly, often without conscious effort. This is because, from birth, we learn to deal with such problems; therefore, our solutions are encoded in the neural cells of the brain.

Solving complex problems, especially in science or engineering, requires deep reflection and possibly extensive research. But what should be done when it is necessary to answer complex questions quickly and correctly? Some scientists and engineers can do this, but only within their own specialization, thanks to intensive preparation or training in specialized courses.

What if an ordinary person also wants to be able to solve complex problems with the same speed and efficiency? One path is to take a specialized training course. An alternative may be machine learning, which involves developing a computer model with a mechanism capable of learning from experience or data in order to provide fast and reliable solutions to certain tasks.

Solving problems in science or engineering is usually much more difficult. This is because we humans can only experience or observe the phenomena associated with those problems. However, many phenomena are difficult to observe, and their basis lies in complex logic. Scientists try to reveal this logic by developing theories, laws, or principles that can help describe these phenomena in the best possible way. These theories are then formulated as algebraic, differential, or integral system equations governing the key variables involved in the phenomena. The next step is to find a method that will solve these equations for variables that change in space and time. The final step is to verify the theory through observations and/or experiments that measure the values of these variables.

A confirmed theory is used to create models that solve problems in which similar phenomena are observed. This type of model is called a model based on physics, or a model based on physical laws.

The process described above is what people have done in their effort to understand nature, and by now we have made significant progress. In this process, many research areas have been created, such as physics, mathematics, biology, and others, which are now sciences.

However, understanding nature is only part of the task. People strive to invent and create new things. A deep understanding of various phenomena allows us to do this, and we have built everything around us: buildings, bridges, aircraft, space stations, automobiles, ships, computers, mobile phones, the Internet, communication systems, energy systems, and much more. In this process, we have created many areas of development that we call engineering.

An understanding of biology has helped us develop medicines, methods for treating diseases in humans and animals, as well as ways to treat plants and the environment, including the development of appropriate policies and strategies. In this process, we have created many research areas, including medicine, agriculture, and ecology.

In human history, countless theories, laws, methods, technologies, and so on have been developed in various fields of science, engineering, and biology. It is enough here to give a general idea of how a problem can be solved by relying on the laws of physics.

It should be noted that in nature, engineering, and society there are many problems for which it is difficult to find suitable laws of physics for an accurate and efficient solution. Therefore, it is important to develop alternative approaches.

1.2 From Data to Solutions: A Variety of Approaches in Machine Learning

In science, engineering, biology, and everyday life, there are many complex problems for which either the governing physical laws are unknown or the solutions of equations based on these laws are too labor-intensive. Nevertheless, for such tasks there are often data obtained from observations, measurements, or historical records. If the data are sufficient and of good quality, computer models that learn from these data can be developed.

Various types of effective artificial neural networks (NNs) with diverse configurations have been developed for solving practical tasks, including multilayer perceptrons (MLPs), convolutional neural networks (CNNs), and recurrent neural networks (RNNs).

Machine learning imitates the learning process of the biological brain, which may contain an enormous number of neurons. From the perspective of data use, the main categories of machine learning include supervised learning, which uses data with true labels; unsupervised learning, which uses unlabeled data; and reinforcement learning, which uses a predefined environment.

Let us consider the variety of approaches in machine learning for solving diverse analytical tasks. These tasks include binary classification, that is, dividing data into two classes; multiclass classification, that is, dividing data into more than two classes; clustering, or grouping data based on their similarity; regression, or predicting continuous values; feature extraction, or identifying the most significant attributes of data; anomaly detection, or identifying unusual or deviating data; and inverse analysis, or identifying cause-and-effect relationships.

Methodologies and algorithms include linear and logistic regression, for predicting values or probabilities respectively; decision trees, which represent decisions as branching structures; support vector machines, which seek the best separating hyperplane between classes; naive Bayes, a simple probabilistic classifier; multilayer perceptrons and artificial neural networks, which are complex networks for function approximation; the k-nearest neighbors method, which classifies based on proximity to samples; random forest, an ensemble of decision trees; Gradient Boosting algorithms, which sequentially improve models while minimizing errors; principal component analysis, which reduces data dimensionality; K-means and Mean-Shift, which are clustering methods; autoencoders, which are neural networks for data compression and reconstruction; and the Markov decision process, which models a sequence of decisions while taking transition probabilities into account. Some of these algorithms will be considered in this book.

In this textbook, the main attention will be paid to neural-network-based models, because they make it possible to create rigorous theories and predictive models. Machine learning is an active area of research and development in which new models, including cognitive machine-learning models, are studied. However, this textbook will not consider the topic of manipulating various machine-learning models.

1.3 From Data Collection to Model Deployment: Stages in Creating Machine Learning

The process of training machine-learning models is complex and multi-stage, and includes the following key steps.

Data collection. This is the first and one of the most important stages; it involves collecting a sufficient amount of high-quality data needed to train the model. Data may be obtained in various ways: through direct collection using sensors or surveys, by importing from open or commercial databases, and also through computer synthesis methods, when real data are supplemented with artificially generated examples to increase the size and diversity of the training sample.

Data preprocessing. At this stage, the data are cleaned of errors, anomalies, and omissions. This may include normalization, standardization, duplicate removal, handling missing values, and other preprocessing techniques whose purpose is to improve the quality and homogeneity of the data before they are used in the model.

Model selection. Here the structure of the machine-learning model is determined, including selection of the algorithm type, for example linear regression, neural networks, or decision trees, and the hypothesis that is best suited to solving the task on the basis of the available data.

Algorithm tuning. At this stage, a learning algorithm is developed or selected and will be used to tune the model parameters. It is important to account for the balance between the model’s learning capacity, which depends on the number of parameters and model complexity, and the size of the available dataset, in order to avoid overfitting. If necessary, regularization methods are applied to improve the generalization ability of the model.

Model initialization. Before training begins, the model parameters are initialized. This may be done by randomly selecting initial values or by using parameters from a pretrained model, which is especially relevant when there are not enough training data available to train the model from scratch.

Model training. During this process, the model is trained on the prepared dataset using the selected optimization algorithm. Training involves adjusting the model parameters so as to minimize the difference between predicted and actual values, that is, the model error, on the training dataset.

Model testing. After training, the model is evaluated on an independent test dataset that was not used during training. This makes it possible to check how well the model generalizes the knowledge it has acquired to new data and to detect possible overfitting.

Model deployment. After successful testing, the model is ready to be deployed in real conditions to solve similar tasks. Deployment may include integrating the model into production systems, developing appropriate software, or embedding the model in devices and applications where it will be used for automatic decision-making or for providing recommendations based on new data.

1.4 Mathematical Concepts, Variables, and Spaces

For convenience of discussion, certain variables and spaces will be used in this book. It is noted that the book considers only real numbers, except in cases where geometrically closed operations are required. Let us consider two examples.

Example 1: regression. Imagine that a company manufactures sports equipment and wants to develop a machine-learning model for predicting the service life of running shoes depending on the material type, weight, and sole type. The goal is to predict how many kilometers of running the shoes will withstand before critical wear occurs. To do this, a series of tests is carried out with different combinations of materials, weights, and soles; as a result, a dataset of 10,000 points is collected, each containing the mentioned characteristics and the actual mileage before critical wear. These data are used to train and validate a regression model.

Example 2: classification. Suppose that a medical institution needs to automate disease diagnosis based on blood-test results. The task of the machine-learning model is to classify tests as “normal” or “pathological” on the basis of hemoglobin levels, leukocyte count, and ESR indicators, that is, erythrocyte sedimentation rate. To do this, blood-test data are collected from 10,000 patients; for each patient, the levels of the specified parameters and the diagnosis made by a physician are recorded. This dataset, which contains both class labels, normal or pathological, and measured parameters, is used to train a classification model capable of automating the initial diagnostic process based on blood tests.

These examples help explain typical tasks that can be solved effectively using a machine-learning model.

1.4.1 Feature Space

Machine learning uses datasets containing p observed or measured real-valued variables in the space of real numbers R; these variables are often called features. In our two examples, p = 3. We can define a p-dimensional feature space, which is a vector space over the real numbers from R with a defined inner product. A vector for an arbitrary point is written in row form as x = [x1, x2, ..., xp].

The origin is at the point x = [0, 0, ..., 0], which is standard for all vector spaces.

Note that by default we define all vectors as rows, as is usually done in Python programming. A column vector is considered a special case of a matrix with one column and many rows. It is clear that the feature space is a special case, with defined vector operations, of a real space.

The quantities xi, where i = 1, 2, ..., p, are also called linear basis functions, not to be confused with basis vectors, because a linear combination of them gives a new vector x that still lies in the feature space.

A data point with p features is a discrete point in the space, and the corresponding vector is expressed as

xi = [xi1, xi2, ..., xip],  i = 1, 2, ..., m

where m is the number of measurements, observations, or data points in the dataset. It is also often called the number of samples in the dataset. For the two examples, m = 10,000.

The data points xi, where i = 1, 2, ..., m, can be stacked into a dataset denoted as X. This is done for convenience of formulation. We do not form such a matrix in computations, because it is usually very large for large datasets with large m.

1.4.2 Affine Space

Affine space is a concept used in machine learning to describe an extended feature space. In practice, this means that an additional dimension is added to the ordinary feature space, which consists of p dimensions, in order to include a constant value. This is often done to account for the free term, or intercept, in a linear model. This constant term is denoted as x0 and is traditionally assigned the value 1. Thus, the feature vector in affine space takes the form

x = [1, x1, x2, ..., xp].

This augmented feature vector represents a complete basis of linear functions, which makes it possible to construct any linear function in the feature space. The presence of one at the beginning of each feature vector makes affine transformations possible. These include not only linear operations such as scaling and rotation, but also translation, which is important for many machine-learning tasks.

Affine space differs from vector space in that it has no clearly defined origin, or zero vector. In affine space, the sum of two vectors does not necessarily belong to the same space, and the affine space itself does not include the zero vector.

Within affine space, a vector belongs to the extended space, but its endpoint is restricted by a hyperplane. In textbooks, such a space is sometimes described as having a “pseudo-dimension” that is one less than its true dimension.

In function approximation theory, a moment matrix assembled from a set of data points, where i = 1, 2, ..., m, is a convenient way to organize data for formulating the approximation problem. In computational practice, however, such a matrix may not be formed explicitly, but it serves as a theoretical basis for constructing and solving machine-learning problems.

It is important to note that affine transformations that preserve the affine properties of the space play an important role in many aspects of machine learning, and their properties and applications will be considered in detail in later chapters of the book.

1.4.3 Label Space

Let us consider a labeled dataset for creating a supervised machine-learning model. We introduce variables that are real numbers. For Example 1, k = 1, and we can define a label space over the real numbers. It is a vector space. A vector in the label space can be written as y = (y1, y2, ..., yk).

In a dataset, a label is associated with a data point. The label for the data point xi, denoted as yi, may be expressed as

yi = [yi1, yi2, ..., yik],  i = 1, 2, ..., m.

For Example 1, the values yij, where i = 1, 2, ..., 10000 and j = 1, represent 10,000 real numbers in the label space. For Example 2, k = 2, and each label yi1 or yi2 may take the value 0 or 1; nevertheless, the labels are still considered within the context of the label space.

These labels, where i = 1, 2, ..., m, may be collected into a label set Y, although in practice we will not always do this in computations.

Typically, affine transformations end at the output layer of a neural network and create a vector in the label space so that a loss function can then be constructed for “final supervision.”

1.4.4 Hypothesis Space

The learning parameters in a machine-learning model are continuous variables that exist in the hypothesis space over the real numbers.

Learning parameters are also called trainable parameters. We use these terms interchangeably. Trainable parameters include the weights and biases in each layer. The “hat” over w implies that it is the collection of all weights and biases, so for all learning parameters we use a single notation in vector form. Their dimension P depends on the type of hypothesis used, including the configuration of neural networks or machine-learning models. These parameters always work with feature vectors; as a result, intermediate feature vectors are obtained in a new feature space or in the label space through a properly designed architecture.

These parameters must be updated, which involves vector operations. To ensure convergence, it is important that the vector of all learning parameters have important vector properties, such as a scalar product, the existence of a norm, and satisfaction of the Cauchy-Schwarz inequality, and so on. We will carry out such proofs repeatedly in this textbook. Consequently, we require the hypothesis space to be a vector space, so that each update of the current learning parameters leads to new parameters that still lie in the same vector space until they converge.

Note that learning parameters are usually matrices or column vectors, which may be regarded as a special case of a matrix. A typical machine-learning model may contain several matrices of different sizes. These matrices form affine transformations that act on features in affine spaces. A component in the “vector” of the hypothesis space may in fact be a matrix, and therefore it is not so easy to understand intuitively. The simplest way, which is also valid, is to “flatten” all matrices and then “join” them together to form a tall vector, and then treat it as an ordinary vector. We constantly perform this type of flattening and joining in Python. Thus, the flattened tall vector in the hypothesis space can be written in a general form as

w = [w1, w2, ..., wP].

In subsequent chapters we will discuss this in more detail for different models, including estimation of the dimension P.

1.4.5 Mathematical Modeling in Machine Learning: From Hypothesis to Optimization

Finally, we can mathematically define machine-learning (ML) models for prediction as a mapping operator.

This means that an ML model uses a dataset X with labels to train its parameters and creates a mapping, or giant functions, that makes a prediction in the label space for any point in the feature space.

The ML model shown in the equation is in fact a data-parameter converter: it converts the given dataset into learning parameters during training and then converts the parameters back to create a prediction for a given set of feature variables. It can also be considered mathematically as a giant function with components in the feature space, governed, or parameterized, by the learning parameters. When the parameters are tuned, a set of giant functions over the feature space is obtained.

On the other hand, this set of giant functions can also be considered as continuous, differentiable functions of these parameters for any given data point in the dataset, and these functions can also be used to form a differentiable loss function. Such a loss function may be the error between these giant functions and the corresponding labels obtained in the dataset. It can be considered a functional of prediction functions, which in turn are functions of the learning parameters in a vector space. Training consists in minimizing such a loss function for all data points in the dataset by updating the learning parameters to their optimizers. This general picture will be explicitly converted into formulas in later chapters. The success factors for creating a high-quality ML model include:

the type of hypothesis; 2. the number of trainable parameters; 3. the quality of the dataset, meaning its representativeness for the problem being solved, including correctness, size, distribution of data points over the feature space, and noise level; 4. techniques for finding the optimizer of the trainable parameters in order to reproduce the label in the dataset as well as possible.

We will discuss this in detail in later chapters for different machine-learning models.

Concepts of spaces are useful in our subsequent analysis of the predictive properties of machine-learning models. Readers may find these concepts difficult to understand at this stage, so it is recommended that they have only an approximate idea for now and return to this section when reading the corresponding chapters, where these ideas will be used in specific ML tasks.

Note: there are also ML models for discrete feature variables, and the trainable parameters may be non-continuous. Such methods are often developed on the basis of intuitive rules and techniques, and we will discuss some of them. The concepts of spaces may not be directly applicable, but they can often help.

1.5 Fundamentals of Preparing for Training Machine-Learning Models

1.5.1 Components of ML Training

Several components are needed for the successful training of a machine-learning (ML) model, and each of them plays an important role in developing and evaluating the model’s performance.

The first and main component is a dataset. Data may be collected in various ways, including observations, experiments, and modeling based on physical laws. These data provide the factual information that the ML model will use for training. Usually a dataset is divided into training and test samples, with a standard ratio of 75% of the data for training and 25% for testing. It is important that the test dataset be independent of the training dataset so that the model’s performance can be evaluated objectively.

The second element is data labels, which are necessary for supervised-learning tasks. Labels serve as the “answers” or “target variables” that the model tries to predict. They are important for the training process, because the model uses these labels to tune its parameters and minimize prediction error.

The third component is preliminary information about the dataset, which may include knowledge about data quality and its key features. This knowledge helps in selecting the most appropriate machine-learning algorithm and determining whether regularization methods need to be applied to reduce overfitting and improve the model’s generalization ability.

The fourth element is the availability of suitable software tools and algorithms. Depending on the task and the dataset, it may be necessary to use particular software packages, libraries, or frameworks that provide the necessary functions and methods for data processing, model training, and performance evaluation.

Together, these components provide the resources needed for effective training and testing of machine-learning models, making it possible to create powerful and reliable prediction systems.

1.5.2 The Spectrum of Data in Machine Learning: From Collection to Application

Data are the fundamental basis of machine learning, providing information for building and training models. Their diversity is as broad as the areas of application of machine learning. Visual data include images and photographs obtained from a wide variety of sources, from mobile devices to professional medical equipment such as computed-tomography and magnetic-resonance scanners. Computer-generated data may come from models based on physical principles or from other reliable artificial models. Text data range from public documents to social media and email. Audio and video data enrich the set of available sources for analysis and interpretation.

However, it is important to realize that data quality and correspondence to a specific application area are necessary for creating reliable machine-learning models. Attempts to use a model outside its training domain may lead to unpredictable and potentially dangerous results. This is relevant in cases where the model makes decisions that may affect health, safety, or important social issues. Therefore, it is important to evaluate the data carefully and make sure that the model has been tested and verified on data that adequately represent the conditions of its future use.

1.6 Physics and Data: Modeling Relationships

Machine-learning models, in general, are slow learners but fast predictors, while models based on the laws of physics do not need training, since they use existing laws, but are slow in prediction. This is because the strategies for physics-law-based models and data-based models are very different. ML models use datasets to train parameters, whereas physics-law-based models use laws to determine parameters.

However, at the level of detailed computational methodology, many of the techniques used in both models are in fact identical or very similar. For example, when we express one variable as a function of other variables, both models use basis functions, such as polynomial functions, radial basis functions (RBF), or both. When constructing objective functions, both models use a least-squares formulation. In addition, the regularization methods used in both are also quite similar. Therefore, these models should not be studied in complete isolation. Physical laws and engineering regularities may be deeply connected and mutually adaptable. This awareness can be useful for better understanding and further developing more effective methods for both models by exchanging ideas and techniques from one to the other.

In general, for computational methods based on physical laws, such as the general form of meshless methods, we understand fairly well why and how the method works in theory. Therefore, we are confident about what we will obtain when the method is used for a problem. However, for data-based methods this is not always the case. Therefore, it is important to develop fundamental theories for data-based methods. In this book we will try to discuss similarities and differences when a computational method is used in both models.

1.7 About the Textbook

This book offers an introductory guide to machine learning, focusing on the basic concepts, fundamental theories, and key computational techniques needed to develop a variety of machine-learning models. Instead of striving for completeness, which is impossible given the breadth and dynamism of the field of machine learning, the book seeks to provide a firm understanding of the foundations on which further study can be built. It is important to note that machine learning is a field in which theoretical knowledge must be combined with practical skills; thanks to the availability of numerous online resources, readers are given tools for practical application and further study.

Special attention in the book is paid to topics that are insufficiently covered in the available literature. Among them are an in-depth study of the predictability of linear functions, an investigation of affine transformations and their role in building deep networks, as well as a detailed consideration of universal prediction theory, the concept of transforming data into parameters, and many other topics.

The book is structured so that readers can study its chapters in any order, with minimal need to consult cross-references. This approach allows readers to immerse themselves more deeply in topics of interest without losing the thread of study.

1.7.1 Intended Audience

The textbook is written for beginners who want to study the fundamentals of machine learning, including university students, graduate students, researchers, and professionals in engineering and the sciences. Engineers and practitioners who want to learn how to create machine-learning models may also find this book useful.

Machine learning is still a rapidly developing field of research. There are many complex problems that provide broad opportunities for research aimed at developing new methods and algorithms. At present, this is a very active research topic. Various techniques are being developed every day. The authors hope that this book will be useful in studying existing machine-learning models and in developing new ones.