What is early stopping? | Machine Learning from Scratch

Machine learning models, particularly those trained iteratively using algorithms like Gradient Descent, face the risk of overfitting the training data. One powerful and elegant solution to this challenge is known as “Early Stopping.” In this blog post, we’ll delve into the concept of Early Stopping, explore its effectiveness, and showcase a practical implementation using a high-degree Polynomial Regression model.

Understanding Early Stopping

In the iterative training of machine learning models, such as with Gradient Descent, the goal is to minimize the prediction error on both the training and validation datasets. As the model continues to learn, the training error naturally decreases. However, a critical point is reached where the validation error stops decreasing and may even start to rise. This phenomenon signifies that the model is overfitting the training data, becoming too complex and losing its generalization ability.

Early Stopping is a regularization technique designed to prevent overfitting. Instead of training the model for a fixed number of epochs, Early Stopping monitors the validation error during training. When the validation error reaches its minimum, training is halted, preventing the model from overfitting.

Implementation with Polynomial Regression

Let’s take a closer look at a practical implementation of Early Stopping using Polynomial Regression and Stochastic Gradient Descent (SGD) in Python:

Here is a basic implementation of early stopping:

from sklearn.base import clone
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import SGDRegressor
from sklearn.metrics import mean_squared_error

# Prepare the data with polynomial features and scaling
poly_scaler = Pipeline([
    ("poly_features", PolynomialFeatures(degree=90, include_bias=False)),
    ("std_scaler", StandardScaler())
])
X_train_poly_scaled = poly_scaler.fit_transform(X_train)
X_val_poly_scaled = poly_scaler.transform(X_val)

# Initialize SGDRegressor with warm_start=True
sgd_reg = SGDRegressor(max_iter=1, tol=-np.infty, warm_start=True, penalty=None, learning_rate="constant", eta0=0.0005)

# Early Stopping logic
minimum_val_error = float("inf")
best_epoch = None
best_model = None

for epoch in range(1000):
    sgd_reg.fit(X_train_poly_scaled, y_train)
    y_val_predict = sgd_reg.predict(X_val_poly_scaled)
    val_error = mean_squared_error(y_val, y_val_predict)
    
    if val_error < minimum_val_error:
        minimum_val_error = val_error
        best_epoch = epoch
        best_model = clone(sgd_reg)

Note that with warm_start=True, when the fit() method is called, it just continues training where it left off instead of restarting from scratch.

In this code:

• Polynomial features are created and scaled using a pipeline.
• SGDRegressor is initialized with warm_start=True, allowing training to continue from the previous state.
• The model is trained iteratively, and validation error is monitored.
• Early Stopping is applied by stopping training when the validation error reaches its minimum.

The Beauty of Early Stopping

Geoffrey Hinton aptly referred to Early Stopping as a “beautiful free lunch.” This simple yet powerful regularization technique adds a layer of intelligence to the training process, preventing models from becoming overly complex and ensuring better generalization to unseen data.

In conclusion, Early Stopping is a valuable tool in the machine learning practitioner’s toolbox. By listening to the validation dataset and gracefully stopping training when needed, it helps strike a balance between model complexity and generalization, ultimately leading to more robust and effective machine learning models.