Chapter 4

Learning Models Based on Statistics and Probability

This chapter examines several aspects of probability theory and statistics that are relevant to machine-learning models, as well as methods for computing them using Python. All code presented in this textbook, and in this chapter in particular, can be found at https://sohoware.ru/SohoBook/. Creating a machine-learning model is usually aimed at prediction, classification, or identification based on available data and knowledge about those data. Predictions may be deterministic or probabilistic. Often we want to...

96min 18,127Words 14Materials

Key ideas

Estimation and Analysis of Event Probabilities
Controlled Random Sampling: Approaches and Strategies
Fundamentals of Probability Theory
Random Number Distributions
Uniform Distribution
Gaussian Distribution

Practice assignment

Take a small list or dictionary of data, convert it into a NumPy array, and compute two simple statistics. Connect the result with "Learning Models Based on Statistics and Probability".

Open lab

Learning Models Based on Statistics and Probability

Theme

Open lab

Creating a machine-learning model is usually aimed at prediction, classification, or identification based on available data and knowledge about those data. Predictions may be deterministic or probabilistic. Often we want to predict the probability that a particular event will occur, which can be extremely useful and practical for solving certain problems.

For example, engineers who maintain aircraft may want to estimate the probability of engine failure based on records and/or diagnostic data. For a physician, it may be important to predict the probability that a patient will develop a critical disease in the near future based on the patient’s medical record, diagnostic data, and the current epidemiological situation. Medical organizations are interested in predicting the probability of a pandemic. All of these tasks require methods for quantitatively estimating the probability of events. This can be a complex research area in which machine-learning models can be helpful.

This chapter focuses on basic concepts, theories, formulations, and computational techniques that may be needed to build machine-learning models based on probability and statistics. At the end of the chapter, a classification model based on the Naive Bayes classifier will be presented.

4.1 Estimation and Analysis of Event Probabilities

4.1.1 Controlled Random Sampling: Approaches and Strategies

In machine learning, it is often necessary to choose numbers randomly. This is relevant when performing tasks related to modeling random processes, generating test data, or splitting a sample into training and testing groups. To carry out such tasks, we can use the capabilities of the Python programming language, in particular its random module, which provides a wide range of functions for generating random numbers and samples.

One of the main tools for generating random numbers in Python is the random.randint(a, b) function, which returns a random integer N such that a ≤ N ≤ b, providing a uniform probability distribution in the specified range from a to b.

Example of using this function to generate random integers:

import random  # Import the module for working with random numbers.

# Set the parameters for number generation.
na, nb, n = 1, 100, 5  # 'na' and 'nb' define the range; 'n' is the number of values.

# Generate and print n random numbers in the range from 'na' to 'nb'.
for i in range(n):
    print(random.randint(na, nb), ' ', end='')  # Generate and print a random number.

print('\n')  # Move to a new line after printing all numbers.

# Repeat the process to demonstrate the independence of generations.
for i in range(n):
    print(random.randint(na, nb), ' ', end='')

# Print two sets of random numbers to demonstrate their independence.

59 69 26 31 23
70 44 43 77 85

In the code above, we generated two sets of five random integers. Each generation produces different numbers, which emphasizes their randomness and independence from one another.

Now let us consider the use of the random.seed() function, which initializes the internal random-number generator so that reproducible sequences of random numbers can be obtained. This is especially useful in situations where it is necessary to ensure repeatability of experiments or analyses.

import random

# Set the parameters for number generation.
na, nb, n = 1, 100, 5  # 'na' and 'nb' define the range; 'n' is the number of values.

random.seed(1)  # Initialize the random-number generator with seed value 1.

# Generate and print n random numbers with a fixed seed.
for i in range(n):
    print(random.randint(na, nb), ' ', end='')

print('\n')  # Move to a new line.

# Repeat the process with the same seed to demonstrate reproducibility.
random.seed(1)  # Use the same seed.
for i in range(n):
    print(random.randint(na, nb), ' ', end='')

# As a result, we obtain two identical sets of random numbers.

18 73 98 9 33
18 73 98 9 33

Setting a seed value makes it possible to obtain repeatable results every time the code is run. This is important for ensuring the reproducibility of experiments in scientific research and software development.

The random.seed() function initializes the internal state of the random-number generator, making the generated sequence of numbers predictable. This is necessary when debugging programs and in scenarios where specific results must be reproduced for demonstration or testing.

Although numbers generated by standard random-number-generation functions appear random, they are only pseudorandom because they come from a deterministic process. In classical computational theory, where computers operate according to strictly defined instructions, true randomness is difficult to achieve. Instead, pseudorandomness is used, meaning that numbers are generated algorithmically and can be predicted if the initial state of the generator is known.

Now let us consider a specific example of generating pseudorandom real numbers in the range from 0 to 1. When the random.random() function is used, each number in the sequence is generated based on the previous value, creating a series of numbers that appears random.

Let us analyze the code shown below:

import random  # Import the module for working with random numbers.

# Setting a seed for the random-number generator ensures reproducible results.
random.seed(1)  # This can be changed to any other number to obtain a different sequence.

n = 5  # Number of generated values.

for i in range(n):  # Loop for generating n random real numbers.
    print(random.random())  # Prints a random real number from 0 to 1.

0.13436424411240122
0.8474337369372327
0.763774618976614
0.2550690257394217
0.49543508709194095

This code generates pseudorandom real numbers in the range from 0 to 1. To study the effect of the seed on the generated sequence, one can change the value passed to random.seed(), or temporarily comment out this line. In that case, the current system time will be used as the seed, which will cause different sequences to be generated each time the code is executed.

4.2 Fundamentals of Probability Theory

Probability is a numerical measure that reflects the likelihood that an event will occur or the accuracy of a prediction. For example, consider a case where the probability of structural failure is 0.1. This can be expressed mathematically as follows:

\Pr(failure = "yes") = 0.1

In this context, we are dealing with only one random variable that can take two possible discrete values: “yes” with probability 0.1 and “no” with probability 0.9. This distribution of a random variable is known as the Bernoulli distribution. In the more general case, when different events are considered, situations may arise with a larger number of discrete random variables, as well as variables with continuous distributions.

Statistics studies methods of sampling, interpretation, and analysis of data about events. Machine learning is based on datasets associated with certain events, and therefore statistical analysis helps us understand these datasets and possibly make predictions based on probability.

To perform statistical analysis of datasets using Python, we begin by importing the required libraries, including TensorFlow:

import tensorflow as tf  # Import the TensorFlow library for machine learning.
import numpy as np       # Import the NumPy library for working with arrays.

# Example code demonstrating probability calculation using TensorFlow.
probability_of_failure = tf.constant(0.1)  # Define the probability of failure as a TensorFlow constant.
probability_of_success = 1 - probability_of_failure  # Compute the probability of successful operation.

# Directly compute and print the probabilities without needing to use a session.
print(f"Probability of failure: {probability_of_failure.numpy()}")
print(f"Probability of success: {probability_of_success.numpy()}")

Probability of failure: 0.10000000149011612
Probability of success: 0.8999999761581421

Let us consider a simple event: rolling a die with six identical faces, each marked with a unique number from 1 to 6. In this context, the random variable under consideration can take six different discrete values. Suppose that such markings introduce no distortion, meaning that the die is fair, and do not affect the outcome of the roll.

Our goal is to determine the probability that a particular number will appear on the upper face of the die after a series of rolls. To do this, we can conduct “numerical” experiments by virtually rolling the die on a computer many times and counting how often the number of interest appears on the top face.

These experiments allow us to estimate the probability that a particular number will appear on the top face of the die based on the frequency of its occurrence during many rolls. This approach is based on the law of large numbers, which states that the more trials are performed, the closer the relative frequency of an event approaches its theoretical probability.

Now let us consider a code example that can be used to conduct such experiments:

import tensorflow as tf

# Set the seed for reproducibility.
tf.random.set_seed(123)

# Create a tensor representing the uniform probability distribution for numbers on the top face of a die.
pr = tf.fill([6], 1 / 6)  # All values are equal to 1/6, representing equally likely outcomes.

# Print the probability distribution.
print('Probability distribution for each face:', pr.numpy())

# Select one value from the distribution, which corresponds to rolling a die.
n_top = tf.random.categorical(tf.math.log([pr]), 1)  # Use tf.random.categorical for sampling.

# Print the die-roll result.
print('Number on the top face =', n_top.numpy()[0][0] + 1)  # Add 1 because indexing starts at 0.

Probability distribution for each face: [0.16666667 0.16666667 0.16666667 0.16666667 0.16666667 0.16666667]
Number on the top face = 1

Each time, the die-roll result was 1. This may seem surprising, but it is quite possible, especially when the seed for the random-number generator is set to the same value before each run of the code. Setting the same seed ensures that the sequence of random numbers generated in each run will be the same, which may cause the same “random” number to be generated each time.

To see different results on each run, remove the line with tf.random.set_seed(123) or set different seed values in each run. This will cause the random-number generator to produce different sequences of numbers each time the code is run, allowing different die-roll results to be obtained.

In this problem, we assume that the theoretical or “true” probability that a particular number appears on the top face of the die is 1/6, which is approximately 0.1667. A single die roll creates a one-dimensional array, or a tensor in TensorFlow, with one element corresponding to the number on the die’s top face. To compute probabilities, we will perform many rolls, allowing statistics to produce more accurate results. This is achieved by specifying the tensor size in the tf.random.categorical() function:

import tensorflow as tf

n_surfaces = 6  # Number of possible values on the faces of the die.
n_tosses = 18  # Number of rolls.

# Set the seed for reproducibility.
tf.random.set_seed(1)

# Logarithms of the probabilities for each face of the die.
logits = tf.math.log([1.0 / n_surfaces] * n_surfaces)

# Obtain roll results.
toss_results = tf.random.categorical(logits[tf.newaxis, :], n_tosses)

print("Rolled", n_tosses, "times.")
print("Roll results:", tf.squeeze(toss_results).numpy())  # Remove extra dimensions and convert to a NumPy array.

Rolled 18 times.
Roll results: [1 0 2 4 1 0 0 1 3 0 1 3 0 4 0 1 5 1]

In this example, we performed 18 rolls, which created a tensor with 18 elements. Using tf.random.set_seed(1) ensures reproducible results. In a controlled experiment, we may obtain, for example, the value 5 once in 18 rolls, giving the probability Pr(die = "5") = 1/18. If 3 appears three times, then Pr(die = "3") = 3/18 = 1/6, and so on. To estimate probabilities more accurately, let us increase the number of rolls.

import tensorflow as tf

n_surfaces = 6  # Number of possible values on the faces of the die.
n_tosses = 18  # Number of rolls.

# Set the seed for reproducibility.
tf.random.set_seed(1)

# Logarithms of the probabilities for each face of the die.
logits = tf.math.log([1.0 / n_surfaces] * n_surfaces)

# Obtain roll results.
toss_results = tf.random.categorical(logits[tf.newaxis, :], n_tosses)

print("Rolled", n_tosses, "times.")
print("Roll results:", tf.squeeze(toss_results).numpy())  # Remove extra dimensions and convert to a NumPy array.

n_t = 20  # New number of rolls.

# Conduct a new series of rolls and print the results.
new_toss_results = tf.random.categorical(logits[tf.newaxis, :], n_t)
print("New roll results:", tf.squeeze(new_toss_results).numpy())

Rolled 18 times.
Roll results: [1 0 2 4 1 0 0 1 3 0 1 3 0 4 0 1 5 1]
New roll results: [0 1 3 2 3 3 2 1 0 1 2 5 2 2 4 5 1 2 5 2]

import tensorflow as tf
import numpy as np

n_surfaces = 6  # Number of possible values on the faces of the die.
n_tosses = 2000  # Number of rolls.

# Set the seed for reproducibility.
tf.random.set_seed(1)

# Logarithms of the probabilities for each face of the die.
logits = tf.math.log([1.0 / n_surfaces] * n_surfaces)

# Obtain roll results.
toss_results = tf.random.categorical(logits[tf.newaxis, :], n_tosses)
toss_results = tf.squeeze(toss_results)  # Remove extra dimensions.

# Count how many times each digit occurs.
counts = tf.math.bincount(toss_results, minlength=n_surfaces)

# Compute the probabilities for each die face.
probabilities = counts / n_tosses

print('Total rolls:', n_tosses)
print('Probability of each of the 6 faces:', probabilities.numpy())
print('Theoretical (true) probabilities:', [1.0 / n_surfaces] * n_surfaces)

Total rolls: 2000
Probability of each of the 6 faces: [0.1655 0.1495 0.1645 0.166  0.1675 0.187 ]
Theoretical (true) probabilities: [0.16666666666666666, 0.16666666666666666, 0.16666666666666666, 0.16666666666666666, 0.16666666666666666, 0.16666666666666666]

In the following code, we use TensorFlow functionality to count how many times each face of the die appears over a series of rolls and normalize these data relative to the total number of rolls at each stage:

import tensorflow as tf
import numpy as np

np.set_printoptions(suppress=True)  # Prevent the use of scientific notation for large numbers.

n_surfaces = 6  # Number of faces of the die.
n_tosses = 2000  # Number of rolls.

# Set the seed for reproducibility.
tf.random.set_seed(1)

# Logarithms of the probabilities for each face of the die.
logits = tf.math.log([1.0 / n_surfaces] * n_surfaces)

# Obtain roll results.
toss_results = tf.random.categorical(logits[tf.newaxis, :], n_tosses)
toss_results = tf.squeeze(toss_results)  # Remove extra dimensions.

# Initialize result records.
record = tf.Variable(tf.zeros((n_surfaces, n_tosses), dtype=tf.float32))

# Count and record the results for each roll.
for i in range(n_tosses):
    # Count each face up to and including the current roll.
    counts = tf.math.bincount(toss_results[:i + 1], minlength=n_surfaces, maxlength=n_surfaces)
    record[:, i].assign(tf.cast(counts, tf.float32))

# Normalize the results.
x = tf.range(1, n_tosses + 1, dtype=tf.float32)
observations = record / x

# Print the results.
print("Results after the first roll:\n", observations[:, 0].numpy())
print("Results after the first 10 rolls:\n", observations[:, 10].numpy())
print("Results after the first 1000 rolls:\n", observations[:, 999].numpy())

Results after the first roll:
[0. 1. 0. 0. 0. 0.]
Results after the first 10 rolls:
[0.36363637 0.36363637 0.09090909 0.09090909 0.09090909 0.        ]
Results after the first 1000 rolls:
[0.157 0.155 0.163 0.17  0.18  0.175]

This simple experiment allows us to obtain 1000 observations for the six possible values of a uniform distribution, where each of the six faces of the die has the same probability of appearing. After 1000 die rolls and calculation of the probability for each of the six faces, we usually obtain values in the range from 0.14 to 0.19. These probabilities change slightly with each new experiment because of the random nature of the process. If we perform 10,000 rolls in each experiment, the resulting probabilities will be very close to the theoretical value 1/6 ≈ 0.1667. Readers can easily repeat this using the TensorFlow code provided.

Now let us visualize the “numerical” results of the experiment using TensorFlow and the matplotlib library for plotting.

import tensorflow as tf
from matplotlib import pyplot as plt
import numpy as np

np.set_printoptions(suppress=True)  # Prevent the use of scientific notation for large numbers.

n_surfaces = 6  # Number of faces of the die.
n_tosses = 2000  # Number of rolls.

# Set the seed for reproducibility.
tf.random.set_seed(1)

# Logarithms of the probabilities for each face of the die.
logits = tf.math.log([1.0 / n_surfaces] * n_surfaces)

# Obtain roll results.
toss_results = tf.random.categorical(logits[tf.newaxis, :], n_tosses)
toss_results = tf.squeeze(toss_results)  # Remove extra dimensions.

# Initialize result records.
record = tf.Variable(tf.zeros((n_surfaces, n_tosses), dtype=tf.float32))

# Count and record the results for each roll.
for i in range(n_tosses):
    # Count each face up to and including the current roll.
    counts = tf.math.bincount(toss_results[:i + 1], minlength=n_surfaces, maxlength=n_surfaces)
    record[:, i].assign(tf.cast(counts, tf.float32))

# Normalize the results.
x = tf.range(1, n_tosses + 1, dtype=tf.float32)
observations = record / x

# Print the results.
print("Results after the first roll:\n", observations[:, 0].numpy())
print("Results after the first 10 rolls:\n", observations[:, 10].numpy())
print("Results after the first 1000 rolls:\n", observations[:, 999].numpy())

# Assume that observations is a tensor with the experimental results.
# For example: observations = tf.random.uniform([6, 1000], minval=0, maxval=1)

# Plot observed probabilities for each die face.
for i in range(6):
    plt.plot(observations[i].numpy(), label=f"Observed probability P(face={i + 1})")

# Add a horizontal line for the theoretical probability.
plt.axhline(y=0.166667, color='black', linestyle='dashed', label="Theoretical probability 1/6")

# Add a legend and display the plot.
plt.legend()
plt.show()

The more experiments we conduct, the closer the obtained probability approaches the theoretical value 1/6. This is a manifestation of the law of large numbers, according to which the average value of the results of repeatedly performed experiments converges to the expected value as the number of trials increases.

The preceding example, involving simple die-rolling events, clearly demonstrates several basic principles and methods used in statistical analysis and probability computation for more complex events. It shows how important it is to understand these basic concepts when analyzing and interpreting results in more complex statistical and probabilistic models.

In the context of complex events, such as multiple die rolls under different conditions or the analysis of events with many variables, the principles demonstrated in this simple example can be extended and adapted. This may include the use of more complex statistical methods and probability distributions, as well as computational tools and software for processing and analyzing large volumes of data.

4.3 Random Number Distributions

In machine learning, random sampling of numbers is often needed. Depending on the type of task, the data of a variable may have different distributions. Numerical sampling of data should be based on a specified or assumed type of distribution. At the beginning of this chapter, we used a uniform distribution for this purpose. Now we will consider this issue in more detail.

4.3.1 Uniform Distribution

Numbers generated from a uniform distribution should have equal chances of falling anywhere in the specified range. To test the uniformity of numbers generated with the random.randint() function, we can run it a large number of times, say one million, and see how these numbers are distributed. The following code is used for this purpose:

import numpy as np  # Import the library for working with arrays.
import matplotlib.pyplot as plt  # Import the library for data visualization.
import random  # Import the module for generating random numbers.

na, nb, n = 0, 99, 100  # Set the initial and final values of the range and the number of values.
counts = np.zeros(n)  # Create an array to count how many times each number is generated.

# Create a figure for displaying histograms.
fig, axes = plt.subplots(2, 3, figsize=(15, 8), sharex=True)
axes = axes.reshape(6)  # Reshape the array of axes for convenient access.

n_samples = 1000001  # Set the number of samples.

# Generate random numbers and count them.
for i in range(1, n_samples):
    counts[random.randint(na, nb)] += 1  # Increase the counter for the generated number.

    # Visualize the number distribution at different stages of generation.
    if i in [10, 100, 1000, 10000, 100000, 1000000]:
        axes[int(np.log10(i)) - 1].bar(np.arange(na + 1, nb + 2), counts)  # Build a histogram.

plt.show()  # Display the histograms.

This code uses the random.randint(na, nb) function to generate random integers in the range from na to nb, inclusive. We then count how many times each number was generated and visualize this with histograms at different stages of the process: after 10, 100, 1000, 10,000, 100,000, and 1,000,000 generated numbers. This allows us to visually assess how uniformly the numbers are distributed.

Textbook illustration: Learning Models Based on Statistics and Probability — 4.3.1 Uniform Distribution

According to the law of large numbers, as the sample size increases, the average value of the results of a random sample approaches the mathematical expectation, or mean, of the entire population. In the context of a uniform distribution, this means that the more numbers we generate, the closer the distribution of these numbers will be to an ideal uniform distribution, where every number in the specified range has an equal chance of being selected.

With small sample sizes, significant deviations from uniformity are possible because random fluctuations can lead to disproportionate representation of some numbers. However, as the sample size increases, these random fluctuations average out, and the distribution of numbers becomes increasingly uniform. This demonstrates that sufficiently large samples are important for obtaining reliable statistical conclusions.

In the context of machine learning, it is worth noting that the quality and reliability of models largely depend on the quantity and quality of the data used. Uniformly distributed data can be especially important in tasks where equal representation of all possible values of a variable must be ensured, so that the model can learn correctly and avoid being biased toward particular ranges of values.

4.3.2 Gaussian Distribution

The normal distribution, also known as the Gaussian distribution, describes many phenomena in nature. It is described by the following probability density function for the variable x:

p(x) = \frac{1}{\sigma\sqrt{2\pi}} e^{-\left(\frac{x - \mu}{\sqrt{2}\sigma}\right)^2}

where μ and σ are the mean and standard deviation of the distribution, respectively. The normal distribution is often denoted as $\mathcal{N}(\mu, \sigma^2)$ . In particular, when μ = 0 and σ = 1, we are dealing with the standard normal distribution, denoted as $\mathcal{N}(0, 1)$ , and its density function simplifies to:

p(x) = \frac{1}{\sqrt{2\pi}} e^{-\left(\frac{x}{\sqrt{2}}\right)^2}

The gauss() function from the random module in the NumPy library makes it convenient to generate numbers that follow a normal distribution.

Let us visualize the density function defined in the equation:

p(x) = \frac{1}{\sigma\sqrt{2\pi}} e^{-\left(\frac{x - \mu}{\sqrt{2}\sigma}\right)^2}

The bell-shaped curve is probably already familiar to you.

import numpy as np
import matplotlib.pyplot as plt
from random import gauss

mu, sigma, n = 0.0, 0.1, 10  # Mean (mu), standard deviation (sigma), and number of generated values (n).

# Generate n random numbers from a normal distribution.
for i in range(n):
    print(f'{gauss(mu, sigma):.4f} ', end='')

# Define the variable x.
x = np.arange(-0.5, 0.5, 0.001)

# Define the Gaussian function.
def gf(mu, sigma, x):
    return 1 / (sigma * np.sqrt(2 * np.pi)) * np.exp(-0.5 * ((x - mu) / sigma)**2)

# Visualize the density function.
plt.figure(figsize=(6, 4))
plt.plot(x, gf(mu, sigma, x), label="Gaussian function")
plt.title("Normal Distribution Density Function")
plt.xlabel("Value")
plt.ylabel("Probability density")
plt.legend()
plt.show()

Figure: A typical normal distribution, also known as a Gaussian distribution.

To generate random samples from a Gaussian distribution and compare them with the “true” Gaussian distribution, we can use the np.random.normal() function. This function allows us to generate samples following a normal distribution with specified mean and standard-deviation parameters. After generating samples, we can visualize the resulting data as a histogram and compare them with the theoretical normal-distribution curve.

import numpy as np
import matplotlib.pyplot as plt

# Set the parameters of the normal distribution.
mu, sigma = 0, 0.1  # Mean and standard deviation.

# Number of generated random samples.
n = 500

# Generate n random samples from a normal distribution.
samples = np.random.normal(mu, sigma, n)

# Histogram of the samples.
count, bins, ignored = plt.hist(samples, 80, density=True, alpha=0.6, color='g', label='Sample histogram')

# Gaussian function for the “true” distribution.
def gf(mu, sigma, x):
    return 1 / (sigma * np.sqrt(2 * np.pi)) * np.exp(-0.5 * ((x - mu) / sigma)**2)

# Plot the theoretical normal-distribution curve.
plt.plot(bins, gf(mu, sigma, bins), linewidth=2, color='r', label='Theoretical distribution')

plt.title("Comparison of the Sample Histogram with the Theoretical Normal Distribution")
plt.xlabel("Value")
plt.ylabel("Probability density")
plt.legend()
plt.show()

This example shows how the generated samples are distributed in comparison with the theoretical normal distribution, demonstrating approximate agreement between them.

NumPy provides the ability to generate samples from roughly 40 different types of distributions.

4.4 Entropy and Probability Theory

For given probabilities of random variables in a statistical event, the corresponding entropy can be computed. Entropy is a measure of the uncertainty of a probability distribution for a given event. It is the scalar product of the probability vector, which contains the probability values of the random variable, and its negative logarithm.

For given probabilities of the random variables of a statistical event, the corresponding entropy can be estimated. This is a measure of the uncertainty of the probability distribution for the event. It is the dot product of the probability vector, which contains the probability values of the random variable, and its negative logarithm. The entropy $H_{\mathbf{p}}$ for an event with probability vector $\mathbf{p}$ is expressed as

H_{\mathbf{p}} = - \left( \sum_i p_i \log p_i \right) = - \mathbf{p} \cdot \log(\mathbf{p})

where $p_i$ is the probability of the $i$ -th possible value of the variable, and $\sum_i p_i = 1$ . The vector $\mathbf{p}$ is the vector that contains these probabilities. The negative sign is necessary because entropy is positive, while $\log(p_i)$ is always negative for $0 \le p_i \le 1$ . In computations, we often normalize it by dividing by the total number of possible values.

In computations, normalization is often performed by dividing by the total number of possible values. Entropy is often used in machine learning when constructing objective functions, because it is a measure of uncertainty that must be minimized.

Because the logarithm is used frequently, let us examine it in more detail using the log() function from the NumPy library.

import numpy as np
import matplotlib.pyplot as plt

p = np.arange(0.01, 1.0, .01)  # Create an array of probabilities from 0.01 to 1 with step 0.01.
logp = -np.log(p)  # Compute the negative logarithm of each value in the array.

# Plot the negative logarithm.
plt.plot(p, logp)
plt.xlabel('Probability p')  # Label the X-axis.
plt.ylabel('Negative logarithm of p')  # Label the Y-axis.
plt.title('Value of the Negative Logarithm of Probability')  # Chart title.
plt.show()  # Display the chart.

Figure: Logarithmic function of probability.

Let us mention several important features that explain why the logarithm is used so often in machine learning:

The function $-\log(p)$ changes monotonically with the argument $p$ , which is important for optimization algorithms widely used in machine learning. Monotonicity is important because taking the logarithm does not affect the location of stationary points of the original function. This ensures the reliable operation of optimization algorithms, because it guarantees that minimizing or maximizing the loss function will lead to an unambiguous optimal solution.

Imagine that you are walking up a staircase: each next step is slightly higher than the previous one. A monotonic function behaves in the same way: with one step forward, that is, with an increase in $p$ , the value of the function changes in one direction as well; in the case of $-\log(p)$ , it decreases. This is necessary for solving machine-learning problems because it helps “find” the optimal solution without confusion, just as you would walk up the stairs toward your goal without being distracted by other paths.

The function $-\log(p)$ decreases as the probability $p$ increases. This inverts the trend of probability and makes it a suitable measure for entropy, especially in the range of high probabilities. When the probability of an event is high, the level of uncertainty is low, because we are almost certain that it will occur; this reduces the entropy value. Similarly, when the probability of an event is low, the level of uncertainty is also low, because we are almost certain that the event will not occur. In this case, we use the probability itself in the entropy equation.

Imagine that you have a scale from 0 to 100, where 100 means complete certainty about something and 0 means complete uncertainty. If something has probability 90 or higher, you are almost certain about it, and your “uncertainty,” or the “surprise” if it happens, will be low. If the probability of something is very small, say 10 or lower, you are also almost certain that it will not happen, and again your “uncertainty” is low. This is how the function $-\log(p)$ works: it decreases as you become more and more certain about the outcome.

Entropy $H_{\mathbf{p}}$ is a combination of probability and its negative logarithm as a product, as shown in the equation. This combination provides the necessary behavior and is correctly defined for our purposes by using the properties of the logarithm function.

This can be compared with measuring the “interestingness” or “unexpectedness” of information. If something is very likely, as day follows night, then it is not very “interesting” or “unexpected,” and therefore its “interestingness,” or entropy, is small. If something is less likely, such as rolling a six with a standard die, it is more “interesting” or “unexpected,” and its “interestingness” is higher. By using probability and its negative logarithm, we can create a formula, entropy, that helps measure this “interestingness” or “unexpectedness” in different situations, making it very useful in machine learning for understanding how well we can predict the outcomes of different events.

The following examples demonstrate how the entropy function works.

4.4.1 Example 1: Probability and Its Entropy

Consider an event with a variable that takes two values. We make one observation, which produces the probability vector $\mathbf{q}_1$ with entries corresponding to the two probabilities of the two variables. Then we make another observation, which produces the probabilities $\mathbf{q}_2$ . We would like to estimate the entropy of the probabilities of these two observations.

In this example, we see the contradictory behavior of $\mathbf{p}$ and $-\log(\mathbf{p})$ . The vector $\mathbf{q}_1$ has either low or high probability values for its two variables, which means low uncertainty and therefore low computed entropy. On the other hand, $\mathbf{q}_2$ has two probabilities in the middle for both variables, which means that it is highly uncertain. The computed entropy is high, as expected.

4.4.2 Example 2: Variation of Entropy

Suppose an event is described by a variable that can take two values. First, we make an observation that leads to the probability vector $\mathbf{q}_1$ with probabilities of the corresponding two outcomes. Then we make another observation, obtaining the probability vector $\mathbf{q}_2$ . Our task is to estimate the entropy of the probabilities of these two observations.

For a better understanding of the principles of working with entropy, consider the following program code:

import numpy as np

# Function for calculating entropy.
def calculate_entropy(probabilities):
    # Apply the negative logarithm to each element of the probability vector.
    entropy = -np.sum(probabilities * np.log(probabilities))
    return entropy

# Probability vector for the first observation.
q1 = np.array([0.5, 0.5])  # Example probabilities for two outcomes.

# Probability vector for the second observation.
q2 = np.array([0.9, 0.1])  # Example showing a difference in probabilities.

# Calculate and print entropy for each probability vector.
entropy_q1 = calculate_entropy(q1)
entropy_q2 = calculate_entropy(q2)

print(f'Entropy for q1: {entropy_q1:.4f}')  # Print entropy for q1 with 4 decimal places.
print(f'Entropy for q2: {entropy_q2:.4f}')  # Print entropy for q2 with 4 decimal places.

Entropy for q1: 0.6931
Entropy for q2: 0.3251

import numpy as np

# Probability vector for the first observation with low uncertainty:
# strong confidence in the occurrence of the event, because the variable
# has either a very high or a very low probability of being observed.
q1 = np.array([0.999, 0.001])

# Probability vector for the second observation with high uncertainty:
# low confidence in the occurrence of the event, because the variable
# has neither a high nor a low probability of being observed.
q2 = np.array([0.5, 0.5])

# Compute and print negative logarithms of probabilities
# to demonstrate their influence on uncertainty.
print('q1=', q1, ' -log(q1)=', -np.log(q1))  # Negative logarithm increases the value for small p.
print('q2=', q2, ' -log(q2)=', -np.log(q2))

# Compute entropy as a measure of system uncertainty.
H_q1 = -np.dot(q1, np.log(q1)) / len(q1)  # Entropy for q1.
H_q2 = -np.dot(q2, np.log(q2)) / len(q2)  # Entropy for q2.

print('H_q1=', H_q1, 'H_q2=', H_q2)

q1= [0.999 0.001] -log(q1)= [1.00050033e-03 6.90775528e+00]
q2= [0.5 0.5] -log(q2)= [0.69314718 0.69314718]
H_q1= 0.003953627556116044 H_q2= 0.34657359027997264

In this example, we see the opposite behavior of $p$ and $-\log(p)$ . The vector $\mathbf{q}_1$ contains probability values that are either very low or very high, which indicates low uncertainty and, consequently, low computed entropy. On the other hand, the vector $\mathbf{q}_2$ contains probabilities close to the middle for both variables, which indicates high uncertainty. The computed entropy, as expected, is high.

To show the change in entropy visually, let us create artificial events $\mathbf{q}_1$ with a variable that takes two possible values. Let the probabilities of these two values be $\mathbf{v}_1$ and $\mathbf{v}_2$ , changing inversely so that the sum of the probabilities is 1. We use the following code to calculate the change in entropy depending on changes in $\mathbf{v}_1$ and $\mathbf{v}_2$ :

import numpy as np
import matplotlib.pyplot as plt

# Event with a variable that takes two values.
v1 = np.arange(0.01, 1.0, .05)  # Initialize v1 with step 0.05 from 0.01 to 1.0.
gap = (v1[1] - v1[0]) * len(v1) / 3.  # Compute the gap width between histogram bars.
v1 /= (v1[0] + v1[-1])  # Normalize v1 to keep the sum of probabilities equal to 1.
v2 = v1[::-1]  # Create v2 as a mirror image of v1.

# Print v1 and v2 values for checking.
print(v1, np.sum(v1))
print(v2, np.sum(v2))

# Check that v1 and v2 sum to 1 for each pair of values.
print(v1 + v2, np.sum(v1 + v2) / 2)

# Visualize probabilities with a histogram.
xtick = range(len(v1))  # Labels for the x-axis.
plt.bar(range(len(v1)), v1, width=gap * 1.2, alpha=.9)  # Histogram for v1.
plt.bar(range(len(v2)), v2, width=gap, alpha=.9)        # Histogram for v2.
plt.xlabel('Event ID, first series: v1, second series: v2')
plt.ylabel('Probability')
plt.xticks(xtick)
plt.show()

H_qf = np.array([])  # Initialize an array for the entropy of q1 events.

# Calculate entropy for each pair of probabilities.
for q1 in list(zip(v1, v2)):
    H_qf = np.append(H_qf, -(np.dot(q1, np.log(q1))) / 2)  # Compute and append entropy.

# Visualize the change in entropy.
plt.plot(v1, H_qf)
plt.xlabel('Probability, v1 (v2 = 1 - v1)')
plt.ylabel('Event entropy')
plt.title('Event Entropy')
plt.show()

[0.01030928 0.06185567 0.11340206 0.16494845 0.21649485 0.26804124
 0.31958763 0.37113402 0.42268041 0.4742268  0.5257732  0.57731959
 0.62886598 0.68041237 0.73195876 0.78350515 0.83505155 0.88659794
 0.93814433 0.98969072] 10.0
[0.98969072 0.93814433 0.88659794 0.83505155 0.78350515 0.73195876
 0.68041237 0.62886598 0.57731959 0.5257732  0.4742268  0.42268041
 0.37113402 0.31958763 0.26804124 0.21649485 0.16494845 0.11340206
 0.06185567 0.01030928] 9.999999999999998
[1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.] 10.0

As the charts show, entropy reaches its maximum when the probabilities $\mathbf{v}_1$ and $\mathbf{v}_2$ are equal to 0.5, which corresponds to the greatest uncertainty of the outcome. Entropy is minimal at the edges of the range, which corresponds to cases where the outcome of events is almost determined.

4.4.3 Example 3: Entropy for Events with Variables Taking Different Numbers of Uniformly Distributed Values

Consider events in which a variable can take different numbers of possible values. It is assumed that the probability distribution for this variable is uniform for all such events. Our goal is to determine how the entropy of the probability distribution changes depending on the number of event variables.

Entropy is a measure of uncertainty or unpredictability in a probability distribution. In the context of a uniform distribution, the more possible values a variable can take, the higher the uncertainty and, accordingly, the entropy.

import matplotlib.pyplot as plt
import numpy as np

# Initialize variables.
N = 0       # Current number of possible values of the variable.
max_v = 100  # Maximum number of possible values.
Ni = np.array([])    # Array for storing the number of variables.
H_qf = np.array([])  # Array for storing entropy values.

# Generate a uniform distribution and compute entropy for each N.
while N < max_v:
    N += 1  # Increase the number of possible values of the variable.
    Ni = np.append(Ni, N)  # Record the current number of variables.
    qf = np.ones(N) / N  # Generate a uniform distribution.
    H_qf = np.append(H_qf, -np.dot(qf, np.log(qf)) / len(qf))  # Compute and record entropy.

# Print results.
print('Probability distribution:', qf[0:max_v:10])
print('H_qf=', H_qf[0:max_v:10])

# Visualize results.
plt.plot(Ni, H_qf)
plt.xlabel('Number of variables, all with equal probability')
plt.ylabel('Entropy')
plt.title('Events with Variables of a Uniform Distribution')
plt.show()

Probability distribution: [0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01]
H_qf= [-0.          0.21799048  0.14497726  0.11077378  0.09057493  0.07709462
 0.06739137  0.06003774  0.05425246  0.04956988]

In this code, we study how entropy changes as the number of possible values of a variable increases within a uniform distribution. We start with one possible value and gradually increase this number to 100, computing and recording the entropy for the current distribution at each step.

The results show that:

Entropy is zero when $N = 1$ ; this means that if there is only one possible value, there is no uncertainty and we know the outcome exactly. 2. Entropy reaches its maximum at $N = 3$ ; this indicates the greatest uncertainty for the given number of possible values. 3. For very large $N$ , entropy decreases, implying a reduction in uncertainty because the probability of each individual variable becomes very small due to the assumption of a uniform distribution for all variables.

This study emphasizes the relationship between the number of possible states of a system and the degree of uncertainty, or entropy, of that system in the context of a uniform distribution.

4.4.4 Cross-Entropy

Cross-entropy is a concept that we often encounter in machine learning and statistics, and it helps us measure how well our predictions match real data. To understand this concept, imagine that we have a program that tries to guess what the user has drawn on the screen.

Suppose our program can predict three different things: a cat, a tree, and a car. When the user draws a picture, the program analyzes it and says how likely it is that the picture is a cat, a tree, or a car. These probabilities form our “predicted distribution.”

The “true distribution” is what the user actually drew. If the user drew a cat, then the true distribution will be 100% cat and 0% everything else.

Cross-entropy helps us understand how well our predicted distribution matches the true one. If the predictions are very close to reality, cross-entropy will be low. If they are far from the truth, cross-entropy will be high.

Let us consider a code example that demonstrates how cross-entropy can be computed in Python:

import numpy as np

# True distribution: the user drew a cat.
true_distribution = np.array([1, 0, 0])  # Cat, tree, car.

# Predicted distribution: the program thinks it is most likely a cat.
predicted_distribution = np.array([0.7, 0.2, 0.1])

# Compute cross-entropy.
cross_entropy = -np.sum(true_distribution * np.log(predicted_distribution))

print(f"Cross-entropy: {cross_entropy}")

Cross-entropy: 0.35667494393873245

We define the true and predicted distributions as arrays, where each array element represents the probability that the drawn picture is a cat, a tree, or a car. Then we compute cross-entropy using the formula and print the result.

Cross-entropy is often used in statistics. The cross-entropy of a distribution $\mathbf{q}$ with respect to a distribution $\mathbf{p}$ is defined as follows:

H_{\mathbf{pq}} = - \left( \sum_i p_i \log q_i \right) = - \mathbf{p} \cdot \log(\mathbf{q})

In general, cross-entropy is a measure of the similarity of two distributions from the same space. In machine learning, we are interested in the cross-entropy of the predicted probability $\mathbf{q}$ with respect to the true probability $\mathbf{p}$ .

In this context, $H_{\mathbf{pq}}$ can serve as a measure of the performance of a prediction model and is therefore often used as an objective function or loss function in machine-learning models.

Note the following properties:

Cross-entropy is not symmetric: $H_{pq} \ne H_{qp}$ if $p \ne q$ .

The inequalities $H(p, q) \ge H(p)$ , $H_{pq} \ge H_p$ , and $H_{qp} \ge H_q$ hold. The difference represents the Kullback-Leibler divergence, which is always positive. When the two distributions coincide, cross-entropy becomes the entropy studied in the previous section. In that case, all three inequalities mentioned become equalities.

Consequently, in machine-learning models, even if the prediction is perfect, cross-entropy will still not be zero, because the true distribution itself may have entropy. If $H_{\mathbf{p}}$ is the entropy of the true distribution, then the cross-entropy $H_{\mathbf{pq}}$ is bounded below by $H_{\mathbf{p}}$ . It can be zero only when the true distribution has no uncertainty at all, meaning that the probabilities of all variables are zero except one, which is equal to 1.

Next, let us examine several simple examples.

4.4.5 Example 4: Cross-Entropy in Prediction Quality

In the section on cross-entropy in the context of prediction, consider an example of using this method to evaluate the quality of a weather forecast. Suppose we aim to predict the probability of precipitation on the next day based on current meteorological data. As an example, our prediction is that the probability of rain is 90%, and the probability of clear weather is 10%. This is our assumption, or “forecast.”

Next, imagine that we have exact weather data for the next day, which we obtained, for example, by using a time machine. According to these data, the actual probability of rain is 99%, and the probability of sunny weather is only 1%. These values are the “true” probabilities.

To compare our forecast with the true probability, we use a method known as cross-entropy. Cross-entropy allows us to quantify the difference between the probability distributions predicted by the model and the true probabilities. A minimal value of cross-entropy indicates that the forecast is as close as possible to the truth, whereas a large value indicates a significant discrepancy between the forecast and the actual data.

In programming, this process involves calculating cross-entropy to compare the probability distributions provided by the forecast and the actual observations. This makes it possible to assess how effectively the model can predict true outcomes and is an important tool in machine learning and data analysis.

Consider a simple event with a variable that can take two possible values. Suppose we have a good forecast; let us examine how it can be measured using cross-entropy.

Case of a good forecast:

import numpy as np

# Predicted probability of two outcomes.
q_good = np.array([0.9, 0.1])  # Predicted probabilities for two values.
y = np.array([0.99, 0.01])     # True probabilities of two values.

p = y       # True distribution.
q = q_good  # Predicted distribution.

# Print p and q values, as well as their logarithms.
print('p=', p, ' log(p)=', -np.log(p))
print('q=', q, ' log(q)=', -np.log(q))

# Calculate and print entropy for p and q.
print(' Entropy: Hp=', -np.dot(p, np.log(p)) / len(p),
      ' Hq=', -np.dot(q, np.log(q)) / len(q))

# Calculate and print cross-entropy.
print('\nCross-entropy: Hpq=', -np.dot(p, np.log(q)) / len(p),
      ' Hqp=', -np.dot(q, np.log(p)) / len(q))

p= [0.99 0.01] log(p)= [0.01005034 4.60517019]
q= [0.9 0.1] log(q)= [0.10536052 2.30258509]
Entropy: Hp= 0.028000767177423672 Hq= 0.1625414866957241
Cross-entropy: Hpq= 0.06366638071559423 Hqp= 0.2347811604334802

np.array([0.9, 0.1]) and np.array([0.99, 0.01]) define the predicted and true probabilities of the two possible outcomes, respectively. - -np.log(p) and -np.log(q) are used to compute the natural logarithm, ln, of the elements of arrays p and q, which is part of the formula for calculating entropy and cross-entropy. - np.dot(p, np.log(p)) and np.dot(p, np.log(q)) compute the dot product between two arrays, which is needed to calculate entropy and cross-entropy. Division by len(p), the number of elements in p, makes it possible to normalize the result. - Printing the entropy values, Hp, and the cross-entropy values, Hpq and Hqp, gives an idea of the quality of the forecast. Low cross-entropy Hpq indicates good forecast quality of q with respect to the true distribution p.

Note that the values of Hpq and Hqp are different, which demonstrates the asymmetry of cross-entropy. At the same time, we expect $H_{pq} \ge H_p$ and $H_{qp} \ge H_q$ , confirming that cross-entropy is greater than or equal to the corresponding entropy. This property is used to evaluate the quality of predicted distributions with respect to the true distribution.

We can see that the cross-entropy $H_{\mathbf{pq}}$ is low, which indicates that the prediction $\mathbf{q}$ is good. Note that $H_{\mathbf{pq}} \ne H_{\mathbf{qp}}$ , $H_{\mathbf{pq}} \ge H_{\mathbf{p}}$ , and $H_{\mathbf{qp}} \ge H_{\mathbf{q}}$ .

4.4.6 Example 5: Cross-Entropy with a Poor Forecast

Imagine a simple card game in which we have a deck of two cards: one red and one blue. The goal of the game is to guess which card will be drawn. In our experiment, we will make a forecast and then compare it with the actual outcome in order to evaluate how good our forecast is.

Forecast: Suppose we assume that the probability of drawing the red card is 10% (0.1), and the probability of drawing the blue card is 90% (0.9). - Truth: In reality, the red card appears with probability 99% (0.99), and the blue card with only 1% (0.01).

To understand how much our forecast differs from the truth, we use a concept from computer science called cross-entropy. In Python, this can be done with a few lines of code:

import numpy as np

# Define the forecast and the true probabilities.
q_bad = np.array([0.1, 0.9])  # Forecast.
y = np.array([0.99, 0.01])    # Truth.

# Calculate cross-entropy.
cross_entropy = -np.dot(y, np.log(q_bad))

print(f"Cross-entropy: {cross_entropy}")

Cross-entropy: 2.2806128472206835

In this example, cross-entropy is approximately 2.28, which indicates a significant discrepancy between your forecast and the actual probabilities.

Cross-entropy is used to measure the “distance” between two probability distributions and is often applied in classification and machine-learning tasks to optimize models. In our card-game analogy, a large value of cross-entropy means that if you made bets based on your forecast, you would often be wrong, because your forecast differs greatly from reality.

The magnitude of cross-entropy shows the difference between our forecast and real outcomes. The higher this number, the less accurate our forecast was. In our case, the high value of cross-entropy indicates that the assumption that the blue card would appear with probability 90% was far from the truth.

Consider once again a simple event in which the variable can take two possible values. This time, we assume a poor forecast and analyze how this is reflected in cross-entropy.

In the case of a completely wrong forecast:

import numpy as np

# Predicted probabilities of the two values (poor forecast).
q_bad = np.array([0.1, 0.9])

# True probabilities of the two values.
y = np.array([0.99, 0.01])

p = y      # Truth.
q = q_bad  # Forecast.

# Print true and predicted probabilities with their logarithms.
print('p=', p, ' log(p)=', -np.log(p))
print('q=', q, ' log(q)=', -np.log(q))

# Calculate entropy for the true distribution and the forecast.
print(' Entropy: Hp=', -np.dot(p, np.log(p)) / len(p),
      ' Hq=', -np.dot(q, np.log(q)) / len(p))

# Calculate cross-entropy.
print('\nCross-entropy: Hpq=', -np.dot(p, np.log(q)) / len(p),
      ' Hqp=', -np.dot(q, np.log(p)) / len(q))

p= [0.99 0.01] log(p)= [0.01005034 4.60517019]
q= [0.1 0.9] log(q)= [2.30258509 0.10536052]
Entropy: Hp= 0.028000767177423672 Hq= 0.1625414866957241
Cross-entropy: Hpq= 1.1403064236103417 Hqp= 2.072829100487316

In this code:

q_bad represents a poor forecast of event probabilities. - y contains the true probabilities of the events. - p and q are used for convenience and denote the true and predicted probabilities, respectively. - np.log() is used to compute the natural logarithm of the probabilities, which is necessary for calculating entropy and cross-entropy. - Console output makes it possible to see the probability values, their logarithms, and the values of entropy and cross-entropy.

The results show that the cross-entropy value Hpq is high, which indicates the poor quality of the forecast q. It should also be noted that Hpq is not equal to Hqp; in general, $H_{pq} \ge H_p$ and $H_{qp} \ge H_q$ , which demonstrates the asymmetry of cross-entropy and its dependence on which distribution is considered true and which is considered predicted.

We can see that the cross-entropy $H_{\mathbf{pq}}$ is high, which indicates the poor quality of the forecast $\mathbf{q}$ . Note that $H_{\mathbf{pq}} \ne H_{\mathbf{qp}}$ , $H_{\mathbf{pq}} \ge H_{\mathbf{p}}$ , and $H_{\mathbf{qp}} \ge H_{\mathbf{q}}$ .

Now we are ready to discuss the Kullback-Leibler divergence.

4.5 Kullback-Leibler Divergence

The Kullback-Leibler divergence is a metric used to quantify differences between two probability distributions. In this case, a probability distribution can be regarded as a mathematical model describing the probability that various events or outcomes will occur in an experiment or observed process.

For clarity, imagine that each of two observers has their own idea of how certain events or elements are distributed in the observed world. This idea is expressed through “bags” filled with marbles of different colors, where each color corresponds to a certain event or element, and the number of marbles of a particular color reflects the probability of its occurrence according to the observer.

The Kullback-Leibler divergence makes it possible to evaluate how strongly these two representations differ, that is, how much one “bag” differs from the other. If the “bags” are identical, meaning that the observers’ representations of the event distribution coincide completely, the Kullback-Leibler divergence will be zero. However, the more differences there are between the “bags,” the larger the KL-divergence value will be, indicating a more significant divergence in the observers’ representations of the distribution of events or elements.

In a more formal mathematical context, KL-divergence is computed as the sum of products of the event probabilities in one distribution and the logarithm of the ratio of these probabilities to the probabilities of the corresponding events in another distribution. This makes it possible not only to determine that differences between distributions exist, but also to quantify the degree of these differences.

Thus, by using the concept of KL-divergence, we can not only compare different representations of the distribution of events or elements in different contexts, but also analyze changes in data.

In Python, we can represent a probability distribution as a list of probabilities, where each list element represents the probability of a certain event.

Suppose we have two distributions, P and Q, represented in Python as lists. We want to compute how much Q differs from P using KL-divergence.

import numpy as np

# Define two distributions P and Q.
P = np.array([0.1, 0.2, 0.7])  # Distribution P.
Q = np.array([0.1, 0.3, 0.6])  # Distribution Q.

# Compute the KL-divergence from Q to P.
KL_divergence = np.sum(P * np.log(P / Q))

# Print the result.
print(f"Kullback-Leibler divergence from Q to P: {KL_divergence}")

Kullback-Leibler divergence from Q to P: 0.026812454257447993

The Kullback-Leibler divergence, or KL-divergence, is a measure of the relative entropy of one distribution with respect to another. For two given distributions $\mathbf{p}$ and $\mathbf{q}$ , the KL-divergence from $\mathbf{q}$ to $\mathbf{p}$ is defined as

D_{KL}(\mathbf{p} \parallel \mathbf{q}) = \sum_i p_i \cdot [\log p_i - \log q_i] = \mathbf{p} \cdot [\log(\mathbf{p}) - \log(\mathbf{q})]

This is also called the relative entropy of $\mathbf{q}$ with respect to $\mathbf{p}$ , which can be considered the true or reference distribution. Using the definitions of entropy and cross-entropy, we have

D_{KL}(\mathbf{p} \parallel \mathbf{q}) = H_{\mathbf{pq}} - H_{\mathbf{p}}.

Note that the KL-divergence of $\mathbf{q}$ relative to $\mathbf{p}$ differs from the divergence of $\mathbf{p}$ relative to $\mathbf{q}$ . We also have

D_{KL}(\mathbf{p} \parallel \mathbf{q}) \ge 0,

and equality holds only if $\mathbf{p} = \mathbf{q}$ .

Below are two simple examples of KL-divergence.

4.5.1 Example 1: KL-Divergence of a Good Prediction Distribution

Imagine that you have two acquaintances, each of whom loves ice cream but prefers different flavors. One of them usually chooses chocolate flavor, in 99% of cases, and only rarely prefers strawberry, in 1% of cases. This preference distribution can be denoted as PP.

Now let us try to predict which ice-cream flavor your friend will choose next time based on previous experience. You may assume: “I think he will choose chocolate with probability 90% and strawberry with probability 10%.” This is your assumed flavor distribution; let us call it QQ.

The Kullback-Leibler divergence, or KL-divergence, is a method for evaluating how accurately your assumed distribution QQ agrees with the actual distribution PP. It makes it possible to evaluate how well you know your friend’s preferences.

If your assumption almost coincides with your friend’s real choices, the KL-divergence will be minimal, indicating the accuracy of your assumption. However, if the assumption differs greatly from reality, the divergence will be significant, indicating the need to study your friend’s preferences better.

It should be remembered that KL-divergence is asymmetric; this means that what matters is not only how accurately you guessed your friend’s preferences, but also how much the real preferences differ from your assumption. This is similar to comparing your friend’s impressions of ice cream chosen according to your advice with your own impressions of ice cream chosen according to your friend’s true preferences.

As a result, the code example shows that the KL-divergence values turned out to be small, indicating that the prediction was sufficiently accurate.

Consider a simple event with a variable that can take two possible values. Suppose we have a good prediction of a distribution relative to the true distribution. How is this measured using the Kullback-Leibler divergence (KL-divergence)?

In the case of a good prediction, we can define the true or reference distribution p and the predicted distribution $q$ . The key question is how close the predicted distribution $q$ is to the true distribution $p$ , and this is what KL-divergence measures.

KL-divergence is not symmetric, which means that $D_{pq}$ is not equal to $D_{qp}$ . This is an important aspect showing that the difference between the predicted and true distributions is not the same in both directions. We will examine this using specific values.

import numpy as np

# True or reference distribution.
p = np.array([0.99, 0.01])  # Probabilities of the two possible outcomes.

# Predicted distribution.
q_good = np.array([0.9, 0.1])  # Good distribution prediction.

# Calculate logarithms for each element in the distributions.
log_p = -np.log(p)
log_q_good = -np.log(q_good)

# Print values for the true and predicted distributions.
print(f'p={p}, log(p)={log_p}')
print(f'q={q_good}, log(q)={log_q_good}')

# Calculate KL-divergence from p to q and from q to p.
Dpq = np.sum(p * (log_p - log_q_good)) / len(p)
Dqp = np.sum(q_good * (log_q_good - log_p)) / len(p)

# Print KL-divergence values.
print(f'Dpq={Dpq}, Dqp={Dqp}')

# Output: We observe that both KL-divergences are positive and have low values,
# which indicates the good quality of the prediction q.

p=[0.99 0.01], log(p)=[0.01005034 4.60517019]
q=[0.9 0.1], log(q)=[0.10536052 2.30258509]
Dpq=-0.035665613538170556, Dqp=-0.07223967373775611

In this code, we define the true distribution p and the predicted distribution q_good, compute the logarithms of these distributions, and then calculate the KL-divergence in both directions, $D_{pq}$ and $D_{qp}$ .

Note that $D_{\mathbf{pq}} \ne D_{\mathbf{qp}}$ .

4.5.2 Example 2: KL-Divergence for a Poorly Predicted Distribution

Imagine that we have a coin that can land heads or tails, and we are trying to guess how often this will happen. We make an assumption, but it turns out not to be very accurate. Here is how we can check this using a special calculation called KL-divergence.

import numpy as np

# Our assumption about how often the coin lands on different sides.
# The true chances are 99% heads and 1% tails.
p = np.array([0.99, 0.01])

# Our incorrect assumption is 10% heads and 90% tails.
q = np.array([0.1, 0.9])

# Calculate and show how often we think the coin lands on each side.
print('We think that heads occurs in', q[0] * 100, '% of cases')
print('We think that tails occurs in', q[1] * 100, '% of cases')

# Count our errors in the assumption.
# Error when assuming that heads occurs more often than it really does.
Dpq = np.sum(p * (np.log(p) - np.log(q)))

# Error when assuming that tails occurs more often than it really does.
Dqp = np.sum(q * (np.log(q) - np.log(p)))

# Show how badly we were wrong.
print('Our error when we think that heads lands more often:', Dpq)
print('Our error when we think that tails lands more often:', Dqp)

We think that heads occurs in 10.0 % of cases
We think that tails occurs in 90.0 % of cases
Our error when we think that heads lands more often: 2.224611312865836
Our error when we think that tails lands more often: 3.8205752275831846

The assumption that the coin lands heads in only 10% of cases and tails in 90% is significantly different from reality, where heads occurs in 99% of cases and tails in 1%. This means that our assumption about the probability distribution is very inaccurate.

The obtained KL-divergence values, Dpq and Dqp, show how large an error was made in the assumptions. The larger the KL-divergence value, the worse the predicted distribution.

KL-divergence is not symmetric; this means that the error value changes depending on whether we compare the predicted distribution with the true one, Dpq, or the true distribution with the predicted one, Dqp. In our case, the error that occurs when we assume that tails occurs more often than heads, Dqp, is larger than the error in the reverse assumption, Dpq. This shows that the mismatch between prediction and reality can have different consequences depending on the direction of comparison.

This example emphasizes the importance of accurate prediction of probability distributions in statistics and machine learning. Inaccurate predictions can lead to significant errors, which in turn can influence decisions made on the basis of these predictions.

Consider a simple event associated with a variable that can take two possible values. Suppose we have an inaccurate prediction of the probability distribution compared with the true or reference distribution. We examine how this divergence is measured using KL-divergence in the following code example:

import numpy as np

# Poor forecast.
p = np.array([0.99, 0.01])  # True distribution.
q = np.array([0.1, 0.9])    # Predicted distribution.

# Print distribution values and their logarithms.
print('p=', p, ' log(p)=', -np.log(p))
print('q=', q, ' log(q)=', -np.log(q))

# Calculate KL-divergence from p to q and from q to p.
Dpq = np.dot(p, (np.log(p) - np.log(q))) / len(p)
Dqp = np.dot(q, (np.log(q) - np.log(p))) / len(p)

# Print KL-divergence values.
print('KL-divergence from p to q (Dpq)=', Dpq)
print('KL-divergence from q to p (Dqp)=', Dqp)

p= [0.99 0.01] log(p)= [0.01005034 4.60517019]
q= [0.1 0.9] log(q)= [2.30258509 0.10536052]
KL-divergence from p to q (Dpq)= 1.112305656432918
KL-divergence from q to p (Dqp)= 1.910287613791592

We see that both KL-divergence values, $D_{pq}$ and $D_{\mathbf{qp}}$ , are positive. They both have high values, indicating that the prediction $q$ is inaccurate. It should also be noted that $D_{pq} \ne D_{\mathbf{qp}}$ , which demonstrates the asymmetry of KL-divergence.

4.6 Fundamentals of Binary Cross-Entropy in Machine Learning

When we work with classification tasks in Python, especially when our task is to determine whether an object belongs to one of two classes, for example “dog” or “cat,” we often use a method called binary cross-entropy to evaluate the effectiveness of our algorithms.

Binary cross-entropy can be imagined as a way of measuring differences between what our program predicts and the actual answer. If our program is very confident in its prediction but is wrong, binary cross-entropy will be high, indicating a significant error. On the other hand, if the program’s prediction is accurate, the binary cross-entropy value will be low, indicating good program performance.

In the context of Python, we can calculate binary cross-entropy using a function that takes two main parameters: the true labels, for example “1” for dogs and “0” for cats, and the predicted probabilities that each object belongs to class “1.” The function then computes a “penalty” for each prediction based on the difference between the prediction and the true label, and returns the total “penalty” for all predictions, which is the binary cross-entropy.

Using binary cross-entropy as a metric allows us to improve our machine-learning algorithms by tuning them so as to minimize the differences between predictions and true labels, making our models more accurate in classification tasks.

import numpy as np

def binary_cross_entropy(true_labels, predicted_probs):
    """
    Compute binary cross-entropy between true labels and predicted probabilities.

    Parameters:
    true_labels (numpy array): an array of true labels (0 or 1).
    predicted_probs (numpy array): an array of predicted probabilities of belonging to class 1.

    Returns:
    float: the binary cross-entropy value.
    """

    # Minimum value added to prevent taking the logarithm of zero.
    epsilon = 1e-15

    # Adjust predicted probabilities to prevent logarithm computation errors.
    predicted_probs = np.clip(predicted_probs, epsilon, 1 - epsilon)

    # Compute binary cross-entropy.
    bce = -np.mean(true_labels * np.log(predicted_probs) +
                   (1 - true_labels) * np.log(1 - predicted_probs))

    return bce

# Example of use.
true_labels = np.array([1, 0, 1, 1, 0])  # True labels (1 = dog, 0 = cat).
predicted_probs = np.array([0.9, 0.2, 0.8, 0.95, 0.1])  # Predicted probabilities that the image contains a dog.

bce = binary_cross_entropy(true_labels, predicted_probs)

print(f"Binary cross-entropy: {bce}")

Binary cross-entropy: 0.14166028566632455

In this code, the binarycrossentropy function takes two arguments: truelabels, an array of true labels, and predictedprobs, an array of predicted probabilities. Inside the function, the np.clip operation is used to restrict the predicted probabilities, preventing logarithmic errors caused by probabilities equal exactly to 0 or 1. Then the mean value of the weighted logarithmic losses for each example is computed, which is the final binary cross-entropy for the dataset.

Binary cross-entropy is a measure of divergence between two probability distributions for a binary-classification problem. It is used in machine learning and statistics to measure the “distance” between the true distribution $\mathbf{p}$ and the predicted distribution $\mathbf{q}$ of probabilities. In the context of machine learning, where $\mathbf{p}$ represents the true class label, 0 or 1, and $\mathbf{q}$ is the predicted probability of belonging to class 1, binary cross-entropy is defined as follows:

H_{\mathbf{pq}}^{B} = - \left( \sum_i p_i \cdot \log q_i + (1 - p_i) \cdot \log(1 - q_i) \right)

For two given distributions $\mathbf{p}$ and $\mathbf{q}$ , the binary cross-entropy of $\mathbf{q}$ with respect to $\mathbf{p}$ is defined as

H_{\mathbf{pq}}^{B} = - \left( \sum_i p_i \cdot \log q_i + (1 - p_i) \cdot \log(1 - q_i) \right) = - \mathbf{p} \cdot \log(\mathbf{q}) - (\mathbf{1} - \mathbf{p}) \cdot \log(\mathbf{1} - \mathbf{q})

In machine-learning models, we usually assume that $\mathbf{p}$ is the true distribution, which may take values 0 or 1 and therefore cannot be logarithmized. Binary cross-entropy can be regarded as a measure of the entropy of the predicted probability relative to the true probability. It takes into account both the probability $\mathbf{p}$ and $\mathbf{q}$ , and the inverse probability $(1 - \mathbf{p})$ and $(1 - \mathbf{q})$ , and computes the entropy of both. In essence, this doubles cross-entropy, giving a slightly strengthened measure of mismatch between the predicted distribution and the true one. It is often used to measure model performance and is used as one type of loss function.

Consider several examples.

4.6.1 Example 1: Binary Cross-Entropy for a Good Prediction Distribution

Binary cross-entropy is a measure of how well a predicted probability distribution corresponds to the true distribution. In this example, we are dealing with a good forecast, which means that the predicted probability distribution corresponds well to the true distribution. For example, if the true probability that an event will occur is 0.8 and our model predicts probability 0.85, then the difference between these probabilities is small; consequently, the binary cross-entropy will be low, indicating good prediction quality.

Imagine that you are trying to predict whether it will rain tomorrow. The true probability of rain tomorrow is 80% (0.8). Your weather-prediction model gives a rain probability of 85% (0.85). Since your prediction is very close to the true probability, the difference between these two probabilities is small; consequently, the binary cross-entropy will be low. This indicates that your forecast is fairly good.

Consider a simple event whose variable can take four possible values. Suppose we have a good prediction of the distribution relative to the true or reference distribution. We investigate how this is measured using binary cross-entropy with the following code:

import numpy as np

# True probability distribution.
p = np.array([1.0, 0.0, 0.0, 0.0])  # truth.

# Predicted probability distribution.
q = np.array([0.9, 0.04, 0.03, 0.03])  # forecast.

# Complementary probabilities for the true and predicted distributions.
p_conv = 1.0 - p  # complement of the truth.
q_conv = 1.0 - q  # complement of the forecast.

# Print true and predicted probabilities and their complements.
print(p, q, ' complements:', p_conv, q_conv)

# Compute cross-entropy.
cHpq = -np.sum(np.dot(p, np.log(q))) / len(p)

# Compute binary cross-entropy.
bcHpq = -np.sum(np.dot(p, np.log(q)) + np.dot(p_conv, np.log(q_conv))) / len(p)

# Print cross-entropy and binary cross-entropy values.
print('Cross-entropy cHpq:', cHpq)
print('Binary cross-entropy bcHpq:', bcHpq)

[1. 0. 0. 0.] [0.9  0.04 0.03 0.03] complements: [0. 1. 1. 1.] [0.1  0.96 0.97 0.97]
Cross-entropy cHpq: 0.02634012891445657
Binary cross-entropy bcHpq: 0.05177523128687465

In this example, we are dealing with a situation in which the event variable can take one of four values. It is assumed that distribution q is our prediction for the true distribution p. We use cross-entropy to evaluate the quality of our prediction relative to the true distribution. Cross-entropy cHpq measures the difference between two distributions, whereas binary cross-entropy bcHpq takes into account both event probabilities and their complementary values, providing a more complete measure of differences.

Binary cross-entropy turns out to be approximately twice as large as ordinary cross-entropy, which agrees with expectations. This is because binary cross-entropy takes into account not only the probabilities that events will occur, as in ordinary cross-entropy, but also the probabilities that events will not occur, which makes it a more informative measure for evaluating the quality of binary classifiers.

4.6.2 Example 2: Binary Cross-Entropy for a Poorly Predicted Distribution

In this example, we consider a case in which the predicted probability distribution differs significantly from the true distribution. For example, if the true probability of an event is 0.8 and the model predicts 0.3, then the difference between the predicted and true probabilities is large. As a result, the binary cross-entropy value will be high, indicating poor prediction quality.

Using the same weather-forecast scenario, suppose that the true probability of rain tomorrow is 80% (0.8), but your model predicts a rain probability of only 30% (0.3). Here the difference between the model’s prediction and the true probability is significant, leading to high binary cross-entropy. This means that your forecast is far from the truth and is not of good quality.

Consider an event whose variable can take four possible values. Suppose we have a poor distribution prediction compared with the true distribution. Let us again examine how this is measured using binary cross-entropy with the following code:

import numpy as np

# Initialize a poor prediction and the true distribution.
q = np.array([0.4, 0.2, 0.3, 0.1])  # prediction.
p = np.array([1.0, 0.0, 0.0, 0.0])  # true distribution.

# Calculate the complements of the prediction and the true distribution.
p_conv = 1.0 - p  # complement of the true distribution.
q_conv = 1.0 - q  # complement of the prediction.

# Print the true distribution, prediction, and their complements.
print(f"Truth: {p}, Prediction: {q}, Complement of truth: {p_conv}, Complement of prediction: {q_conv}")

# Calculate cross-entropy.
cHpq = -np.sum(p * np.log(q)) / len(p)

# Calculate binary cross-entropy.
bcHpq = -np.sum(p * np.log(q) + p_conv * np.log(q_conv)) / len(p)

# Print results.
print(f"Cross-entropy cHpq: {cHpq}")
print(f"Binary cross-entropy bcHpq: {bcHpq}")

Truth: [1. 0. 0. 0.], Prediction: [0.4 0.2 0.3 0.1], Complement of truth: [0. 1. 1. 1.], Complement of prediction: [0.6 0.8 0.7 0.9]
Cross-entropy cHpq: 0.22907268296853875
Binary cross-entropy bcHpq: 0.4003674356962309

In this example, binary cross-entropy, bcHpq, shows the magnitude of divergence between the predicted and true distributions. As the results show, binary cross-entropy is approximately twice as large as cross-entropy, cHpq, indicating an enhanced measure of divergence.

This increase is important because binary cross-entropy takes into account not only the probability that the event occurs, but also the probability that it does not occur, making it a more sensitive measure for evaluating prediction quality in binary-classification tasks.

4.6.3 Example 3: Binary Cross-Entropy for a More Uniform True Distribution: Good Prediction

In this example, the true probability distribution is more uniform, which may mean that the event has approximately equal chances of occurring or not occurring. A good prediction in such a case will also assume a uniform probability distribution close to the true one. Binary cross-entropy in this case will be lower, indicating good agreement between the predicted and true distributions.

Suppose we are trying to predict whether a fair coin toss will result in heads or tails. The true probability of each outcome is 50% (0.5). If your model predicts the probability of heads as 52% (0.52) and tails as 48% (0.48), then your prediction is fairly uniform and close to the true distribution. Binary cross-entropy in this case will be relatively low, indicating a good forecast.

In the two previous examples, we considered cases with extreme true distributions, whose probabilities were 1.0 and 0. In both examples, we observed an improved entropy measure using binary cross-entropy. In this example, we consider a more uniform true distribution and study the behavior of binary cross-entropy.

Consider a forecast and a true distribution with more uniform probabilities:

import numpy as np

# Predicted distribution.
q = np.array([0.4, 0.2, 0.3, 0.1])  # Forecast.

# True distribution.
p = np.array([0.3, 0.3, 0.2, 0.2])  # Truth, a fairly uniform distribution.

# Calculate the complements of the forecast and truth for binary cross-entropy.
p_conv = 1.0 - p  # Complement of the truth.
q_conv = 1.0 - q  # Complement of the forecast.

# Print the true and predicted distributions, as well as their complements.
print("Truth:", p, "Forecast:", q, ' Complements: Truth:', p_conv, 'Forecast:', q_conv)

# Calculate ordinary cross-entropy.
cHpq = -np.sum(p * np.log(q)) / len(p)

# Calculate binary cross-entropy.
bcHpq = -np.sum(p * np.log(q) + p_conv * np.log(q_conv)) / len(p)

# Print cross-entropy results.
print('Cross-entropy cHpq:', cHpq)
print('Binary cross-entropy bcHpq:', bcHpq)

Truth: [0.3 0.3 0.2 0.2] Forecast: [0.4 0.2 0.3 0.1] Complements: Truth: [0.7 0.7 0.8 0.8] Forecast: [0.6 0.8 0.7 0.9]
Cross-entropy cHpq: 0.36475754318911824
Binary cross-entropy bcHpq: 0.585609240747465

In this code, we first define the predicted (q) and true (p) probability distributions. Then we compute the complements of these distributions, pconv and qconv, which are needed to calculate binary cross-entropy. Two entropy metrics are computed and printed: ordinary cross-entropy, cHpq, and binary cross-entropy, bcHpq.

In this case, it is also observed that binary cross-entropy does not lead to an improvement. This shows that although binary cross-entropy may be more informative in some situations, it does not always provide better distinction between predicted and true distributions, especially when the true distribution is more uniform.

4.6.4 Example 4: Binary Cross-Entropy for a More Uniform True Distribution: Forecast Quality

Here we are also dealing with a more uniform true probability distribution, but the prediction differs significantly from this distribution. For example, if the true distribution assumes that the probabilities of an event occurring and not occurring are equal, while the prediction is strongly biased toward one outcome, binary cross-entropy will be high, indicating poor prediction quality.

Returning to the coin example, suppose your model predicts the probability of heads as 80% (0.8) and tails as 20% (0.2). Here your prediction differs significantly from the uniform true probability distribution, 50% for heads and 50% for tails. As a result, binary cross-entropy will be high, indicating poor prediction quality because it does not correspond to the expected uniform probability distribution.

In the previous two examples, we considered cases with extreme true distributions: their probabilities were equal to 1.0 and zeros. For both examples, we observed an improved entropy measure using binary cross-entropy. In this example, we consider a more uniform true distribution and study the behavior of binary cross-entropy.

In this example, we consider a case in which the forecast distribution (q) and the true distribution (p) are more uniform, which allows us to examine in more detail the effectiveness of binary cross-entropy as a measure of forecast accuracy.

import numpy as np

# Predicted distribution.
q = np.array([0.4, 0.2, 0.3, 0.1])  # Forecast.

# True distribution.
p = np.array([0.3, 0.3, 0.2, 0.2])  # Truth, a fairly uniform distribution.

# Calculate complements of the forecast and truth.
p_conv = 1.0 - p  # Complement of the truth.
q_conv = 1.0 - q  # Complement of the forecast.

# Print values for clarity.
print("Truth:", p, "Forecast:", q, ' Complements:', p_conv, q_conv)

# Calculate cross-entropy.
cHpq = -np.sum(p * np.log(q)) / len(p)

# Calculate binary cross-entropy.
bcHpq = -np.sum(p * np.log(q) + p_conv * np.log(q_conv)) / len(p)

# Print results.
print('Cross-entropy cHpq:', cHpq)
print('Binary cross-entropy bcHpq:', bcHpq)

Cross-entropy cHpq: 0.36475754318911824
Binary cross-entropy bcHpq: 0.585609240747465

In this case, we also find that binary cross-entropy does not lead to an improvement. This shows that although binary cross-entropy can be a useful measure for evaluating forecast quality, especially when the true distribution has extreme probability values, its effectiveness may be less obvious in cases with a more uniform distribution.

By considering different examples, from good forecasts to poor predictions and from uniform to nonuniform true distributions, we see how binary cross-entropy helps determine how well a model predicts the probability that events will occur.

In the weather-forecasting and coin-toss examples, we saw that low binary cross-entropy indicates that the predicted probability distribution is close to the true distribution, demonstrating the high quality of the model. Conversely, high binary cross-entropy indicates a significant discrepancy between the predicted and true probabilities, which is a sign of poor prediction quality.

4.7 Bayesian Statistics

Bayesian statistics is a methodology for estimating the probability of events based on available data. To illustrate it at an accessible level, we can use the analogy of a box of candies that contains red and blue candies in an unknown ratio. If you draw candies and find that most of them are red, you might conclude that red candies dominate in the box. However, if you later receive information that the initial number of red and blue candies was equal, the probability estimate changes: the larger number of red candies drawn may have occurred by chance. Bayesian statistics makes it possible to analyze such situations mathematically, refining assumptions on the basis of new data. This approach is widely used to refine forecasts and make decisions based on available information, which makes it a valuable tool in scientific and practical applications.

Let us test this idea with Python. We will create a simple program that “draws” candies from a “box” and helps us update our assumption about the number of candies of each color.

import random

# Initial number of candies of each color.
red = 50
blue = 50

# Function that simulates drawing one candy.
def draw_candy():
    global red, blue
    if random.random() < red / (red + blue):
        red -= 1
        return "red"
    else:
        blue -= 1
        return "blue"

# Simulate drawing 10 candies.
for _ in range(10):
    print(draw_candy())

# See how the number of candies has changed.
print(f"Red candies remaining: {red}")
print(f"Blue candies remaining: {blue}")

This example helps us understand how Bayesian statistics works: we start with an initial assumption and then update it as we receive new information—in this case, by drawing candies from the box.

Consider a statistical event that includes more than one random variable and in which the variables occur jointly. When we deal with multiple random variables of this kind, we are often interested in the joint probability $\Pr(A,B)$ : the probability that $A = a$ and $B = b$ occur simultaneously for given elements $a$ and $b$ . Clearly, for any values $a$ and $b$ , $\Pr(A,B) \leq \Pr(A = a)$ , because $\Pr(A = a)$ is measured regardless of what happens to $B$ . For events $A$ and $B$ to occur jointly, event $A$ must occur and event $B$ must occur as well, and vice versa. Therefore, the probability of the joint occurrence of $A$ and $B$ cannot be greater than the probability of each of them separately.

The joint probability $\Pr(A,B)$ when $A$ is known leads to conditional probability, denoted $\Pr(B \mid A)$ , which represents the probability of the occurrence of $B$ under the condition that $A$ has already occurred. This brings us to the important Bayes theorem.

By definition, we have

\Pr(A,B) = \Pr(B \mid A)\Pr(A).

By symmetry, the following equality is also valid:

\Pr(A,B) = \Pr(A \mid B)\Pr(B).

Therefore, we obtain Bayes’ formula:

\Pr(A \mid B) = \frac{\Pr(B \mid A)\Pr(A)}{\Pr(B)}.

# Initialize probabilities.
p_A = 0.3          # Probability of event A.
p_B_given_A = 0.6  # Probability of event B given A.
p_B = 0.2          # Probability of event B.

# Calculate the conditional probability of A given B using Bayes' theorem.
p_A_given_B = (p_B_given_A * p_A) / p_B

# Print the result.
print(f"Conditional probability of A given B: {p_A_given_B:.2f}")

Conditional probability of A given B: 0.90

p_A represents the prior probability of event A, that is, the probability before any data are obtained.

pBgiven_A is the probability of event B occurring if event A is known to have occurred.

p_B is the prior probability of event B.

pAgiven_B is calculated as the conditional probability of event A occurring under the condition that event B has already occurred, using Bayes’ theorem.

4.7.1 Classification Using the Naive Bayes Approach

Classification with the naive Bayes algorithm is a statistics-based learning method used to determine the category of new data on the basis of previously studied data. This method is called “naive” because it assumes that all features in the data are independent of one another, which is not always true in reality.

At the beginning, information is collected about already known data, where each data item is clearly assigned to a particular category. This allows the algorithm to learn the characteristics associated with each category.

At the next stage, the algorithm calculates the probabilities that certain features appear in each category, based on the data collected earlier. For example, if most objects in one category have a particular feature, the algorithm will treat the presence of that feature in a new object as evidence in favor of that object belonging to this category.

When the algorithm is given a new object with a set of features, it uses the previously calculated probabilities to determine which category the object most likely belongs to. The algorithm evaluates the probabilities for each category and selects the one with the highest probability.

The naive Bayes algorithm is effective for classification tasks because of its simplicity and ability to process large volumes of data quickly. However, its “naivety”—the assumption that features are independent—can be a limitation in complex tasks where features are interrelated.

Install the scikit-learn library:

pip install scikit-learn

The following libraries are used in Python for machine-learning and text-processing tasks, and each has its own purpose:

from sklearn.modelselection import traintest_split: This function from the scikit-learn library is used to split a dataset into training and test samples. This makes it possible to evaluate model performance on data it has not seen before and thereby test its generalization ability.

from sklearn.feature_extraction.text import CountVectorizer: CountVectorizer is used to convert text data into numerical form by creating a so-called bag of words. It counts how many times each word from the training set occurs in each document and converts texts into feature vectors that can be used in machine learning.

from sklearn.naive_bayes import MultinomialNB: MultinomialNB is a naive Bayes algorithm for multinomial features and is often used in text classification. This algorithm is based on Bayes’ theorem and assumes feature independence within each class. It is well suited for working with vectorized text data, for example data transformed with CountVectorizer.

from sklearn.metrics import accuracyscore: This function is used to evaluate the quality of machine-learning models. accuracyscore calculates classification accuracy, that is, the share of correctly predicted labels among all predictions. Accuracy is one of the simplest and most commonly used performance metrics for classification models.

Together, these tools can be used to create and evaluate machine-learning models, in particular for text classification. For example, they can be used to develop a system that automatically categorizes emails as spam or not spam.

# Import the required libraries.
from sklearn.model_selection import train_test_split  # Split data into training and test samples.
from sklearn.feature_extraction.text import CountVectorizer  # Convert text into numerical form.
from sklearn.naive_bayes import MultinomialNB  # Create and use a naive Bayes classifier.
from sklearn.metrics import accuracy_score  # Calculate prediction accuracy.

# Example text messages and corresponding labels: 1 = spam, 0 = not spam.
messages = [
    "free tickets send SMS",
    "let us meet for lunch",
    "special offer buy now",
    "hello how are you",
    "do not miss your chance to get a discount",
]

labels = [1, 0, 1, 0, 1]

# Split the data into training and test samples.
messages_train, messages_test, labels_train, labels_test = train_test_split(
    messages,
    labels,
    test_size=0.4,
    random_state=42,
)

# Vectorize the text data.
vectorizer = CountVectorizer()
messages_train_vectorized = vectorizer.fit_transform(messages_train)
messages_test_vectorized = vectorizer.transform(messages_test)

# Train the naive Bayes classifier.
classifier = MultinomialNB()
classifier.fit(messages_train_vectorized, labels_train)

# Predict on the test data.
predictions = classifier.predict(messages_test_vectorized)

# Evaluate model accuracy.
accuracy = accuracy_score(labels_test, predictions)
print(f"Model accuracy: {accuracy}")

Model accuracy: 0.5

A model accuracy of 0.5 indicates that the model correctly classified only one of the two test examples. This is an expected result when working with very small datasets, where machine-learning models often have difficulty generalizing because the number of training examples is limited.

In this particular case, increasing test_size to 0.4 made it possible to use two examples for testing, which gave us slightly more information about how the model performs on new data. Nevertheless, an accuracy of 50% shows that the model could improve significantly with a larger and more diverse training dataset.

4.7.2 Problem Formulation

A popular algorithm known as the naive Bayes classifier was developed on the basis of Bayesian statistics. Consider an event with $p$ variables $\mathbf{x} = \{x_1, x_2, \ldots, x_p\} \in \mathbb{X}^p$ . We assume that each variable $x_i$ is independent of the others. For a given label $y$ , the conditional probability for $\mathbf{x}$ is expressed as

p(\mathbf{x} \mid y) = \prod_i p(x_i \mid y).

On the basis of Bayes’ theorem, we have the following formula:

p(y \mid \mathbf{x}) = \frac{p(\mathbf{x} \mid y)p(y)}{p(\mathbf{x})} = \frac{\prod_i p(x_i \mid y)\,p(y)}{p(\mathbf{x})}.

Although we may not know $p(\mathbf{x})$ , the probability that $\mathbf{x}$ occurs in the event, this may not be necessary because it only concerns normalization when computing $p(y \mid \mathbf{x})$ . Therefore, we can instead use the following formula:

p(y \mid \mathbf{x}) \propto p(\mathbf{x} \mid y)p(y) = \prod_i p(x_i \mid y)\,p(y).

4.7.3 Analysis of Machine-Learning Methods in Handwritten-Digit Recognition

We can now consider how a naive Bayes classifier is coded to identify handwritten digits. We will use the well-known MNIST dataset to train this classifier. The MNIST dataset contains a total of 70,000 images, 60,000 for training and 10,000 for testing, of handwritten digits from 0 to 9; all these images are labeled. The images were collected from employees of the U.S. Census Bureau and students from American high schools.

The digit-classification task is then reduced to computing the probability that a given image $\mathbf{x}$ is the digit $y$ : $p(y \mid \mathbf{x})$ . Any image $\mathbf{x}$ contains $p$ pixels $x_i$ for $i = 1, 2, \ldots, p$ , and each pixel $x_i$ can take the value 1 when it is on or 0 when it is off; therefore it is a binary variable.

Equation (4.12) can then be used, in which we need to estimate $p(y)$ and $p(x_i \mid y)$ . Both can be computed using the MNIST training dataset for each digit. For example, among all 60,000 images in the MNIST training set, the digit 4 occurs 5,800 times, so the probability is $p(y = 4) = \frac{5800}{60000}$ . To estimate $p(x_i \mid y)$ , each pixel $x_i$ is binary, and $p(x_i = 0 \mid y) = 1 - p(x_i = 1 \mid y)$ . The estimate of $p(x_i = 1 \mid y)$ can be obtained by counting how many times pixel $i$ is on for label $y$ and then dividing this number by the total number of images with label $y$ . In this simple algorithm, all we need to do is count the MNIST training images for each label $y$ .

Before working with MNIST, make sure that Python certificates are installed so that you do not encounter an error such as SSL: CERTIFICATEVERIFYFAILED.

To solve the CERTIFICATEVERIFYFAILED problem, you need to install SSL certificates for the Python environment you are using. If you use the version of Python included with macOS or installed from python.org, this problem is usually solved by running the Install Certificates.command script that comes with your Python installation.

Open Finder and go to the Python installation directory. If you installed Python from python.org, it may be in a directory such as /Applications/Python 3.x, where 3.x corresponds to the Python version you installed.

Inside the Python directory, find the file named Install Certificates.command.

Double-click the Install Certificates.command file to run the script. This will open a terminal window and install the required SSL certificates for your Python environment.

After running this script, try running your code again. It should now be able to download the MNIST dataset without SSL-certificate verification errors.

If you use a virtual environment or a Python installation from a source other than python.org, you may need to make sure that SSL certificates are available for your Python environment. Sometimes this can be achieved by installing the certifi package through pip:

pip install certifi

The certifi package includes a collection of root certificates for verifying the trustworthiness of SSL certificates when verifying the identity of TLS hosts. However, the effectiveness of this solution may depend on how your Python environment is configured and how it is set up to work with SSL certificates.

To correct the SSL: CERTIFICATEVERIFYFAILED certificate error in a Python environment on Windows, make sure that your Python environment has access to the necessary SSL certificates. The instructions below describe how to correct this error for Windows users.

Open Command Prompt or PowerShell with administrator privileges. To do this, click the Start button, begin typing cmd or PowerShell, then right-click the corresponding application and choose “Run as administrator.”

Find the directory where Python is installed. If you installed Python from the official python.org website, the directory will usually look approximately like this: C:\Users\YOURUSERNAME\AppData\Local\Programs\Python\Python3X, where Python3X is the version of Python you installed. If you are not sure where Python is installed, you can run the where python command in Command Prompt to find it.

Go to the directory where Python is installed by using the cd command with the correct path to the directory. For example:

cd C:\Users\YOUR_USER_NAME\AppData\Local\Programs\Python\Python3X

Inside this directory, run the script to install certificates. You can do this by executing the following command:

.\Scripts\pip.exe install --upgrade certifi

This command installs or updates the certifi package, which contains a set of root certificates.

After installing or updating certifi, try running your code again.

To solve the SSL: CERTIFICATEVERIFYFAILED certificate problem in a Python environment on Ubuntu, follow these steps:

Open a terminal. You can do this by pressing Ctrl + Alt + T on the keyboard.

Update the package list and install updates to make sure all system packages are current. Run the following commands:

sudo apt update
sudo apt upgrade

Install the ca-certificates and openssl packages, which contain a set of generally accepted certificates, if they are not already installed:

sudo apt install ca-certificates openssl

If you use Python installed through the apt package manager, certificates should already be configured. If Python was installed in another way, you can install certificates manually.

To install or update certificates for your Python environment, run the following command:

sudo /usr/local/bin/python3 -m pip install --upgrade certifi

Replace /usr/local/bin/python3 with the path to the Python interpreter you use if it is different.

After running these commands, try running your Python code again. It should now handle SSL connections correctly without certificate-verification errors.

If you have a Python environment installed through pyenv or virtualenv, make sure that certificates are available for these environments as well, because they may use their own certificate sets. In that case, install certifi inside each virtual environment:

pip install --upgrade certifi

The entire training process reduces to counting:

import tensorflow as tf
import numpy as np

# Load MNIST data using TensorFlow.
mnist = tf.keras.datasets.mnist
(training_images, training_labels), _ = mnist.load_data()

# Normalize the images.
training_images = training_images / 255.0

# Initialize counters for probabilities.
num_labels = 10       # Number of labels: digits from 0 to 9.
pixel_count = 28 * 28 # MNIST image size: 28x28.
p_y = np.zeros(num_labels)  # Probability of each digit.
p_xi_given_y = np.zeros((num_labels, pixel_count))  # Probability of each pixel for each digit.

# Counting to estimate probabilities.
for label in training_labels:
    p_y[label] += 1  # Count occurrences of each digit.

for i, image in enumerate(training_images):
    label = training_labels[i]

    # Convert the image into a binary vector.
    binary_image = np.round(image).flatten()
    p_xi_given_y[label] += binary_image  # Count occurrences of active pixels for each digit.

# Normalize to obtain probabilities.
p_y /= len(training_labels)
p_xi_given_y /= np.sum(p_xi_given_y, axis=1, keepdims=True)

# Add a small epsilon to probabilities before taking logarithms.
epsilon = 1e-9
p_y += epsilon
p_xi_given_y += epsilon

# Function for classifying a new image.
def classify(image):
    binary_image = np.round(image).flatten()

    # Compute the log probability of each digit for the given image.
    log_prob_y = np.log(p_y) + np.dot(binary_image, np.log(p_xi_given_y.T))

    # Return the digit with the highest probability.
    return np.argmax(log_prob_y)

# Test the classifier on a new image (example).
test_image = training_images[0]
predicted_digit = classify(test_image)
print("Predicted digit:", predicted_digit)

Predicted digit: 5

4.7.4 Naive Bayes Classification Algorithm

# Import the required libraries.
import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt

# Load the MNIST dataset using TensorFlow Keras.
(mnist_train_images, mnist_train_labels), (mnist_test_images, mnist_test_labels) = tf.keras.datasets.mnist.load_data()

# Function for transforming images.
def transform(images, labels):
    # Data transformation: pixels are considered “on” when their value is >= 128.
    transformed_images = np.floor(images / 128).astype(np.float32)
    return transformed_images, labels.astype(np.float32)

# Transform the training and test datasets.
train_images, train_labels = transform(mnist_train_images, mnist_train_labels)
test_images, test_labels = transform(mnist_test_images, mnist_test_labels)

# Display an example image from the dataset.
image_index = 8888  # Arbitrary image index.
plt.imshow(train_images[image_index], cmap="Greys")
plt.title(f"Label: {train_labels[image_index]}")
plt.show()

# Initialize arrays for probability counting.
ycount = np.ones(10)       # For each of the 10 digits.
xcount = np.ones((784, 10)) # For each of the 784 pixels in each digit.

# Process the training dataset.
for image, label in zip(train_images, train_labels):
    x = image.reshape((784,))
    y = int(label)
    ycount[y] += 1      # Count the number of each digit.
    xcount[:, y] += x   # Count “on” pixels for each digit.

# Compute probabilities p(x_i | y).
for i in range(10):
    xcount[:, i] = xcount[:, i] / ycount[i]

# Compute the probability p(y).
py = ycount / np.sum(ycount)

# Display the “trained” model.
fig, figarr = plt.subplots(1, 10, figsize=(15, 15))
for i in range(10):
    figarr[i].imshow(xcount[:, i].reshape((28, 28)), cmap="hot")
    figarr[i].axes.get_xaxis().set_visible(False)
    figarr[i].axes.get_yaxis().set_visible(False)
plt.show()

# Print the probabilities for each digit.
print(py)

[0.09871688 0.11236461 0.09930012 0.10218297 0.09736711 0.09035161
 0.09863356 0.10441593 0.09751708 0.09915014]

Figure: One sample image of a handwritten digit from the MNIST dataset.

Figure: Average appearance of handwritten digits.

This code loads the MNIST training and test datasets using TensorFlow, transforms the images so that pixels are considered “on” at values of 128 and above, and performs calculations to determine the probability of each digit and the probability of each pixel for each digit on the basis of the training dataset. At the end of the code, the resulting probability distributions for each digit are visualized; they represent the average appearance of each digit based on the training data.

4.7.5 Testing the Naive Bayes Model

We now examine the performance of this statistics-based model using the MNIST test dataset. The training, which is simply counting over the training dataset, was completed above and gives us $p(x_i = 1 \mid y)$ and $p(y)$ . For a given image $\mathbf{x}$ from the test dataset, we compute the probability corresponding to label $y$ , namely $p(y \mid \mathbf{x})$ , using Equation (4.12), where $\log p(\mathbf{x} \mid y)$ is in turn computed using the trained model. To avoid a chain of multiplications of small probability values, we compute the following logarithms instead, known as log-probability:

\log p(y \mid \mathbf{x}) \approx \log p(\mathbf{x} \mid y) + \log p(y) = \sum_i \log p(x_i \mid y) + \log p(y).

For a given image $\mathbf{x}$ , the feature $x_i$ is binary and takes either the value 1 or the value 0. Because we use a model trained to compute probabilities, we have

\begin{cases} p(x_i = 1 \mid y) = p(x_i = 1 \mid y), & \text{if the prediction is that } x_i \text{ is on},\\ p(x_i = 0 \mid y) = 1 - p(x_i = 1 \mid y), & \text{if the prediction is that } x_i \text{ is off}. \end{cases}

The equation can be written in a single form using a mathematical trick:

p(x_i \mid y) = p(x_i = 1 \mid y)^{x_i}\left(1 - p(x_i = 1 \mid y)\right)^{1 - x_i}.

This is the general equation for computing the probability of an event with binary variables, using the trained model to predict the probability of a positive variable. Ultimately, we have

\sum_i \log p(x_i \mid y) = \sum_i \left[x_i \log p(x_i = 1 \mid y) + (1 - x_i)\log\left(1 - p(x_i = 1 \mid y)\right)\right].

It is now clear that testing essentially consists of measuring the binary cross-entropy between the distribution of the given image, the true distribution, and the distribution of the average image of a labeled digit computed from the dataset, the model distribution. Therefore, we can write the equation directly using the binary cross-entropy formula.

To avoid recalculating logarithms, we precompute $\log p(y)$ for all $y$ , as well as $\log p(x_i \mid y)$ and $\log(1 - p(x_i \mid y))$ for all pixels.

# Import the required libraries for data processing and visualization.
import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt

# Load the MNIST dataset, which contains handwritten digits for training machine-learning models.
(mnist_train_images, mnist_train_labels), (mnist_test_images, mnist_test_labels) = tf.keras.datasets.mnist.load_data()

# Function for preprocessing images.
def transform(images, labels):
    """
    Convert the source images into binary form, with pixels equal to 0 or 1.

    Pixels are considered “on” (1) if their value is >= 128; otherwise they are “off” (0).
    The function receives images and labels as input and returns transformed images and labels.
    """
    transformed_images = np.floor(images / 128).astype(np.float32)
    return transformed_images, labels.astype(np.float32)

# Apply the transformation function to the training and test datasets.
train_images, train_labels = transform(mnist_train_images, mnist_train_labels)
test_images, test_labels = transform(mnist_test_images, mnist_test_labels)

# Visualize an example image after transformation.
image_index = 8888  # Select a random index for demonstration.
plt.imshow(train_images[image_index], cmap="Greys")
plt.title(f"Label: {int(train_labels[image_index])}")
plt.show()

# Initialize arrays for probability counting.
ycount = np.ones(10)        # Initial value 1 prevents division by zero (Laplace smoothing).
xcount = np.ones((784, 10)) # Likewise, to avoid division by zero.

# Process the training dataset to train the naive Bayes classifier.
for image, label in zip(train_images, train_labels):
    x = image.reshape((784,))  # Convert the image into a vector.
    y = int(label)
    ycount[y] += 1             # Account for occurrences of each digit.
    xcount[:, y] += x          # Sum “on” pixels for each digit.

# Normalize to obtain probabilities p(x_i | y).
for i in range(10):
    xcount[:, i] /= ycount[i]

# Compute the prior probability of each digit p(y).
py = ycount / np.sum(ycount)

# Visualize the “trained” model: show the probability that each pixel is “on” for each digit.
fig, figarr = plt.subplots(1, 10, figsize=(15, 15))
for i in range(10):
    figarr[i].imshow(xcount[:, i].reshape((28, 28)), cmap="hot")
    figarr[i].axis("off")
plt.show()

# Compute and visualize probabilities for each digit on test images.
logxcount = np.log(xcount)        # Take logarithms to avoid floating-point problems.
logxcountneg = np.log(1 - xcount) # Log probability of the pixel being “off.”
logpy = np.log(py)                # Log prior probability.

fig, figarr = plt.subplots(2, 10, figsize=(15, 3))

ctr = 0       # Counter for the subplot index.
y_true = []   # List for storing true labels.
pxm = np.array([]) # Array for storing maximum probabilities.
xi_pred = []  # List for storing predicted labels.

# Evaluate the model on test images.
for image, label in zip(test_images, test_labels):
    x = image.reshape((784,))
    y_true.append(int(label))

    logpx = logpy.copy()
    for i in range(10):
        # Compute the logarithm of the joint probability P(x | y)P(y).
        logpx[i] += np.dot(logxcount[:, i], x) + np.dot(logxcountneg[:, i], 1 - x)

    logpx -= np.max(logpx)  # Normalize to prevent numerical overflow.
    px = np.exp(logpx)      # Return from logarithms to probabilities.
    px *= py                # Multiply by the prior probability.
    px /= np.sum(px)        # Normalize to obtain conditional probabilities P(y | x).

    pxm = np.append(pxm, np.max(px))
    xi_pred.append(np.argmax(px))

    figarr[1, ctr].bar(range(10), px, color="blue")
    figarr[1, ctr].axis("off")
    figarr[0, ctr].imshow(x.reshape((28, 28)), cmap="hot")
    figarr[0, ctr].axis("off")

    ctr += 1
    if ctr == 10:
        break

plt.show()

# Print classification results.
y_true = np.array(y_true)
xi_pred = np.array(xi_pred)
print("True labels: ", y_true)
print("Predicted digits:", xi_pred)
print("Correct?", np.equal(y_true, xi_pred))
print("Maximum probability:", pxm)

True labels:  [7 2 1 0 4 1 4 9 5 9]
Predicted digits: [7 2 1 0 4 1 4 9 4 9]
Correct? [ True  True  True  True  True  True  True  True False  True]
Maximum probability: [1.         1.         1.         1.         1.         1.
 0.99999999 0.9999996  0.99999758 0.99999899]

This code is an implementation of a naive Bayes classifier for recognizing handwritten digits from the MNIST dataset, using the NumPy, TensorFlow, and Matplotlib libraries for data processing and visualization. It loads MNIST, converts the images into binary format, where pixels are considered “on” or “off” depending on their intensity, and then trains the model by counting the probabilities that “on” pixels appear for each digit and the probabilities of each digit in the training set. After training, the model is tested on the test dataset: for each image, the probability of belonging to each class, that is, each digit, is computed; the class with the highest probability is selected; and the prediction results are visualized along with their probabilities and compared with the true labels.

4.9 Conclusion

The test results show that the classifier presented above made an error in classifying one of the first ten digits in the test dataset. Specifically, the ninth digit, which should have been recognized as “5,” was incorrectly classified as “4.” It is important to note that the classifier’s confidence in this incorrect prediction was high and close to 1. This indicates possible weaknesses in the assumptions underlying the classification model. One such assumption is that each pixel of the image is generated independently of the others, based only on its class label. Reality, however, is much more complex, because an image of a digit is a complex function that includes relationships between pixels. The limitations of single statistical information in this context emphasize the need for more advanced methods for image-classification tasks.

Naive Bayes classifiers such as the one considered here were popular in the 1980s and 1990s, especially in applications such as spam filtering. In modern image processing, however, they have given way to more powerful methods, such as convolutional neural networks (CNNs).

An alternative approach is to use cross-entropy or binary cross-entropy to evaluate prediction quality. Using the training dataset, we can compute the probability $p(x_i \mid y_i)$ for each image $x_i$ and class label $y_i$ , where $i = 1, 2, \ldots, 9$ . Then, by calculating the cross-entropy between $p(x_i \mid y_{test})$ for the test image and $p(x_i \mid y_j)$ for each possible class label $y_j$ , we can determine the $y_j$ with the smallest cross-entropy as the most likely class label for the test image.

This example illustrates the value of statistical analysis in classification tasks and in machine learning as a whole. It also demonstrates how statistical indicators can be computed for specific datasets. Despite some limitations, naive classifiers remain a powerful tool in the machine-learning arsenal, especially in tasks where strict physical laws are not applicable. They are widely used in medical applications, recommendation systems, text classification, and the creation of forecasts based on real data. The Scikit-learn library can be used to train naive Bayes classifiers, as it provides the convenient and flexible sklearn.naive_bayes module. In addition, TensorFlow is widely used to develop more complex machine-learning models, including deep neural networks. TensorFlow offers an extensive set of tools and APIs for designing, training, and deploying both simple and complex deep-learning models, making it a valuable complement to Scikit-learn for solving more complex machine-learning problems.

Check yourself

Which idea best describes the focus of "Learning Models Based on Statistics and Probability"?

In machine learning, theoretical definitions are useful to check with numerical examples and visualizations.

Which actions help reinforce the chapter material?

Take quiz