Regularization is a technique in machine learning used to avoid the overfitting problem. The concept behind regularization is, model that overfit the data are relatively complex models, for example a polynomial function of degree three or more.
In order to find the best model we define a loss function that calculates how well the model fits the data by calculating the loss. Our aim is to find the model with least loss.
With Regularization we penalize this loss function by adding a complexity term called lambda that should give a higher loss for relatively complex models.
Thus for higher values of lambda high complexity models are rejected and for smaller values of lambda high training error models are rejected.
And finally we get the optimal solution which is in the middle of above two cases.
