Learning

Machine learning: Training strategy – tutorial

Tutorial index:

4. Training strategy

The training strategy determines how a neural network learns from a data set. It combines a loss, which measures the model error, with an optimization algorithm, which updates the model parameters.

OpenNN selects suitable defaults from the neural network task. The loss, optimizer, stopping criteria, batch processing, and validation behavior can also be configured.

Contents

Loss
Optimization algorithms
Training control and defaults

Loss

The loss defines what the neural network must learn. It combines an error term with an optional regularization term:

$loss = error + regularization$

Error term

The error measures the difference between the outputs from the neural network and the targets in the data set.

The training error is calculated on the training samples. The validation error monitors generalization on the validation samples. The testing samples are reserved for the final evaluation and do not update the model.

OpenNN provides the following general-purpose error methods:

Mean squared error.
Mean absolute error.
Normalized squared error.
Weighted squared error.
Cross-entropy error.
3D cross-entropy error.
Minkowski error.

Mean squared error (MSE)

The mean squared error penalizes large differences between outputs and targets. It is the default error for approximation, forecasting, and auto-association.

$MSE = \frac{1}{2 N} \sum_{i = 1}^{N} {‖ y_{i} - t_{i} ‖}^{2}$

Mean absolute error (MAE)

The mean absolute error averages the absolute differences between outputs and targets. It is less sensitive to large individual errors than squared losses.

$MAE = \frac{1}{M} \sum_{j = 1}^{M} | y_{j} - t_{j} |$

Normalized squared error (NSE)

The normalized squared error divides the squared error by the target variance. A value close to one represents prediction around the target mean, while zero represents a perfect prediction.

$NSE = \frac{\sum {‖ y - t ‖}^{2}}{\sum {‖ t - \bar{t} ‖}^{2}}$

Weighted squared error (WSE)

The weighted squared error is designed for imbalanced binary classification. OpenNN calculates class weights from the positive and negative training samples and gives both classes a balanced contribution.

$WSE \propto \frac{1}{2} \sum_{i = 1}^{N} w_{t_{i}} {(y_{i} - t_{i})}^{2}$

Cross-entropy error

Cross-entropy measures probabilistic classification. OpenNN uses binary cross-entropy for one output and categorical cross-entropy for multiple outputs. Multiclass models require a softmax output.

$BCE = - \frac{1}{N} \sum [t \log (y) + (1 - t) \log (1 - y)]$

3D cross-entropy error

The 3D cross-entropy error trains language models and other token sequences. It averages categorical cross-entropy over active tokens and also reports token accuracy and perplexity.

Minkowski error (ME)

The Minkowski error uses a power between one and two to reduce the influence of large residuals. Its default power is 1.5. This loss currently runs on CPU.

$ME = \frac{1}{p N} \sum_{i = 1}^{N} {| y_{i} - t_{i} |}^{p}$

Regularization term

Regularization penalizes large parameters to control model complexity. It is optional, and no regularization is applied by default.

L1 regularization.
L2 regularization.

L1 regularization

L1 regularization adds the sum of the absolute parameter values to the loss.

$R_{1} = λ \sum | θ |$

L2 regularization

L2 regularization adds the sum of the squared parameter values to the loss.

$R_{2} = λ \sum θ^{2}$

Loss function

The loss forms a surface over the neural network parameters. Training searches this surface for parameter values that minimize the selected objective.

Optimization algorithms

The optimization algorithm updates the neural network parameters to reduce the loss. OpenNN currently provides four algorithms:

Adaptive moment estimation (Adam).
Stochastic gradient descent (SGD).
Quasi-Newton method.
Levenberg-Marquardt algorithm.

Algorithm	Training mode	Hardware	Best suited for
Adam	Mini-batch	CPU and GPU	Large, image, text, and sequence models
SGD	Mini-batch	CPU and GPU	Controlled learning schedules and large data sets
Quasi-Newton	Full-batch	CPU	Tabular and moderate-size problems
Levenberg-Marquardt	Full-batch	CPU	Small sequential Dense regression models

Adaptive moment estimation (Adam)

Adam combines exponential averages of the gradient and squared gradient with bias correction. Its main parameters are the learning rate, beta 1, and beta 2.

$θ_{t} = θ_{t - 1} - α \frac{{\hat{m}}_{t}}{\sqrt{{\hat{v}}_{t}} + ε}$

Adam is the default optimizer for approximation, forecasting, auto-association, image classification, text classification, and language modeling.

Stochastic gradient descent (SGD)

SGD updates the parameters after every mini-batch. OpenNN supports learning-rate decay, momentum, and Nesterov momentum.

$η_{t} = \frac{η_{0}}{1 + t d}, θ_{t} = θ_{t - 1} - η_{t} g_{t}$

Quasi-Newton method (QNM)

The quasi-Newton method builds a BFGS approximation of the inverse Hessian from successive gradients. An Armijo line search selects the step size.

$θ_{t + 1} = θ_{t} - α_{t} H_{t} g_{t}$

Quasi-Newton uses all training samples in every epoch and runs on CPU. It is the default optimizer for binary and multiclass classification.

Levenberg-Marquardt algorithm (LM)

Levenberg-Marquardt combines the speed of Gauss-Newton with an adaptive damping parameter. It solves the following system for every parameter update:

$(J^{T} J + λ I) Δθ = - J^{T} e$

It runs on CPU and is intended for sum-of-squares problems with sequential Dense trainable layers. Softmax outputs, non-Dense trainable layers, and weighted, cross-entropy, or Minkowski errors are not supported.

Performance considerations

Use Levenberg-Marquardt for small Dense regression models and Quasi-Newton for moderate full-batch problems. Use Adam or SGD for GPU training, large data sets, and convolutional, attention, recurrent, image, or text networks.

The neural network optimizer guide in the Neural Designer blog contains a broader comparison of optimization methods.

Training control and defaults

The same stopping, validation, and execution controls are shared by the OpenNN optimizers.

Control	Purpose
Batch size	Sets the samples processed together. A value of zero selects the largest batch that fits the available memory.
Sample shuffling	Randomizes training batches at every epoch.
Loss goal	Stops training when the requested training error is reached.
Maximum epochs	Limits the number of complete passes through the training data.
Maximum time	Limits the elapsed training time.
Validation failures	Stops after the validation error fails to improve a specified number of times.
Validation period	Controls how often the validation subset is evaluated.
Restore best	Restores the parameters and states from the best validation epoch.
Gradient clipping	Limits the gradient norm to stabilize Adam and SGD.

Stopping criteria

Training can stop after reaching the loss goal, maximum number of epochs, maximum time, or maximum validation failures. Minimum loss decrease is also available for the full-batch Quasi-Newton and Levenberg-Marquardt algorithms.

Validation and best model

Validation samples monitor generalization without updating the parameters. When validation is available, OpenNN stores the best parameters and network states and restores them after training by default.

The testing subset is not used to choose parameters or stop training. It is reserved for the subsequent testing analysis.

Task defaults

OpenNN initializes the loss and optimizer from the neural network task:

Task	Default loss	Default optimizer
Approximation	Mean squared error	Adam
Forecasting	Mean squared error	Adam
Auto-association	Mean squared error	Adam
Binary classification	Weighted squared error	Quasi-Newton
Multiclass classification	Cross-entropy	Quasi-Newton
Image classification	Cross-entropy	Adam
Text classification	Cross-entropy	Adam
Language modeling	3D cross-entropy	Adam

Hardware and batches

Adam and SGD train on CPU or GPU and support CUDA graph execution. OpenNN can size batches from available CPU or GPU memory, shuffle samples, and prefetch batches during training. Quasi-Newton and Levenberg-Marquardt are full-batch CPU algorithms.

Training results

The training result stores the training and validation histories, stopping condition, elapsed and measured training time, final loss, and the epoch restored as the best model.

⇐ Neural Network Model Selection ⇒