Contents
Performance benchmarking
As we know, the volume, variety, and velocity of information stored in organizations are increasing significantly. Therefore, for machine learning tools to be efficient, they must process large amounts of data in the shortest time possible. Key performance indicators typically measured here are data capacity, training speed, inference speed, and model precision.DATA CAPACITY
TRAINING SPEED
MODEL PRECISION
INFERENCE SPEED
Data capacity tests
Nowadays, common datasets used in machine learning might contain thousands of variables and millions of samples. However, when building models with big datasets, machine learning platforms may crash due to memory problems. Therefore, tools capable of processing these volumes of data are necessary. The data capacity of a machine learning platform can be defined as the biggest dataset that it can process. In this way, the tool should perform all the essential tasks with that dataset. We can measure data capacity as the number of samples a machine learning platform can process for a given number of variables. This metric depends on numerous factors:- The programming language in which it is written (C++, Java, Python…).
- The strategies used within the code for the efficient use of memory.
- The optimization algorithms it contains (SGD, Adam, LM…).
- Choose a reference computer (CPU, GPU, RAM…).
- Choose a reference benchmark (data set, neural network, training strategy).
- Choose a reference model (number of layers, number of neurons…).
- Choose a reference training strategy (loss index, optimization algorithm…).
- Choose a stopping criterion (loss goal, epochs number, maximum time…).

Training speed tests
One of the most critical factors in machine learning platforms is the time they need to train the models. Indeed, modeling big data sets is very expensive in computational terms. Training machine learning models with big datasets can take several hours. Moreover, before deploying a model, it is usually necessary to train many candidate models to select the best-performing one. This can make it impractical to use some platforms for some applications. The training speed of a machine learning platform depends on numerous factors:- The programming language in which it is written (C++, Java, Python…).
- The high performance computing (HPC) techniques that it implements (CPU parallelization, GPU acceleration…).
- The optimization algorithms it contains (SGD, Adam, LM…).
- Choose a reference benchmark (data set, neural network, training strategy…).
- Choose a reference computer (CPU, GPU, RAM…).
- Compare the training speed.

Model precision tests
The main objective of machine learning is to develop models to attain high accuracy. We can define precision as the mean error of a model against a testing data set. The precision of a machine learning platform depends on numerous factors:- The optimization algorithm with which it has been trained (HPC).
- The programming language in which it is written (C++, Java, Python…).
- The high-performance computing (HPC) techniques that it implements (CPU parallelization, GPU acceleration…).
- Choose a reference computer (CPU, GPU, RAM…).
- Choose a reference dataset (variables and sample number).
- Choose a reference model (number of layers, number of neurons…).
- Choose a reference training strategy (loss index, optimization algorithm…).
- Choose a stopping criterion (loss goal, epochs number, maximum time…).

Inference speed tests
In many applications, especially real-time ones, the response time of the model is a critical factor. Indeed, an inference time of a few milliseconds can make the model impractical. The inference speed can be defined as the time to calculate the outputs from the model as a function of the inputs. To measure this metric, we use the number of samples per second. The inference speed of a machine learning platform depends on numerous factors.- The programming language in which it is written (C++, Java, Python…).
- The high-performance computing (HPC) techniques that it implements (CPU parallelization, GPU acceleration…).
- Choose a reference computer (CPU, GPU, RAM…).
- Choose a reference input set (variables and samples number).
- Choose a reference model (number of layers, number of neurons…).
