Regularization

Overfitting and underfitting:

This image should give you a rough idea of what underfitting and overfitting are.

When does not fit the data correctly it is underfitting and overfitting occurs when the curve fits the data too much.

Underfitting is also called as "High Bias" and overfitting is also called as "High Variance".

Overfitting:

If we have too many features, the learned hypothesis may fit the training set very well, that $cost=0$ but may fail to generalize to new examples.

Addressing overfitting:

• Reduce the number of features, manually.
• Regularization:

Regularization:

• Keep all the features but reduce the magnitude/values of parameters thetas.
• IT works well when we have a lot of features, and each of them contributes a bit to predicting y.
• Whenever we regularize a cost function we add the sum of thetas squared multiplied with some parameter to the un-regularized cost function.

For example, let J(theta) be un regularized then we add $lambda⅀(theta(j)^2)$ Where lambda is some regularization parameter and the summation runs over each theta in Theta matrix.