No account yet ?

The purpose of regularization is to limit overfitting and to control variance type error to achieve better performance.

When training a model, regularization makes it possible to impose a constraint to favor simple models at the expense of complex models, which makes it possible to reduce the variance type error and improve the generalization of the solution.

There are many forms of regularization, which depend on the objective sought and the hypotheses set on the problem.

Thus, a Euclidean type regularization in a least squares regression will favor low coefficients while a lasso type regularization used when the number of input variables is high will favor the sparseness of the representation by pushing the algorithm to only take take into account only a small part of the data while ignoring the others.

For neural networks, the most popular regularization methods are:

Dropout in which the weights and parameters of the neural network are randomly replaced by zero during training.

Early Stopping in which learning stops earlier to favor simple models or the Euclidean regularization mentioned above.

Regularization makes it possible to guarantee certain theoretical properties of the algorithms, thus ensuring their good performance, such as stability or better generalization bounds in the case of kernel methods.