Lesson 7.5: Fighting Overfitting in Neural Networks
A deep neural network is a high-variance model with millions of parameters, making it extremely prone to overfitting. This lesson covers the two most essential and effective regularization techniques used to build robust networks that generalize to new data: Dropout and Early Stopping.
Part 1: The Problem - A Brain That Only Memorizes
The power of a neural network lies in its ability to learn complex, non-linear functions. But with millions of weights, it can easily "cheat" by simply memorizing the training data. The result is a model with near-perfect accuracy on the training set but terrible performance on the unseen test set.
While the L1 and L2 regularization we learned in Module 2 can be applied to the weights of a neural network, deep learning practitioners have developed more powerful, specialized techniques to combat this problem.
Part 2: Dropout - The 'Randomly Forgetting' Technique
The Core Idea: Sabotaging Your Own Network
Dropout is a brilliantly simple and counter-intuitive idea developed by Geoffrey Hinton and his students. During training, at every single forward pass, it **randomly 'drops out' (sets to zero) a fraction of the neurons in a layer**.
This means on every training step, the network is forced to learn with a different, randomly thinned-out architecture.
Imagine two diagrams. Left: a fully connected network. Right: the same network, but 30% of the neurons and their connections are greyed out randomly.
Dropout forces the network to learn **redundant representations**.
Analogy: Imagine a critical project at a company. If you have one "superstar" employee who knows everything, the project is fragile. If they get sick, the project fails. A good manager forces the team to cross-train. They might say, "Today, you can't talk to the superstar. You have to figure it out with the rest of the team."
Dropout does the same thing. It prevents the network from becoming overly reliant on any single neuron or specific path. It forces different neurons to learn to detect the same underlying features, creating a much more robust and distributed representation of the data. This acts as a powerful form of regularization and significantly improves generalization.
At test time, all neurons are used (no dropout), but their outputs are scaled down to account for the fact that more neurons are active than during training.
Part 3: Early Stopping - The 'Quit While You're Ahead' Technique
This is the most intuitive and widely-used regularization technique of all. The idea is to monitor the model's performance not just on the training set, but also on a separate **validation set** during the training process.
The Early Stopping Algorithm
The training process for a neural network proceeds in "epochs," where one epoch is a full pass through the entire training dataset.
- After each epoch, calculate the loss on both the training set and the validation set.
- The training loss should always decrease.
- The validation loss will typically decrease at first, but then it will "bottom out" and start to increase. This is the exact moment the model begins to overfit.
- **Stop training** as soon as the validation loss stops improving (or starts getting worse).
- Save the model weights from the epoch that had the best validation loss.
Imagine a plot with 'Epochs' on the X-axis and 'Loss' on the Y-axis. The training loss curve steadily goes down. The validation loss curve goes down, hits a minimum, and then starts to curve back up.
The 'Common Sense' Regularizer
Early stopping is so effective because it directly optimizes for what we care about: performance on unseen data. It is a simple, powerful, and computationally cheap way to prevent overfitting. It is almost always used in practice.
- Use a powerful, adaptive optimizer like **Adam**.
- Add **Dropout** layers between your main hidden layers to force redundant learning.
- Use **Early Stopping** based on a validation set to find the optimal number of training epochs.
- (Optionally) Add **L2 regularization** (also called "weight decay") to the loss function to keep the model weights small.
A standard, modern approach to training a robust neural network combines these ideas:
This combination provides multiple, complementary layers of defense against overfitting, leading to models that are both powerful and capable of generalizing to new data.
What's Next? The Theory of Power
We've now built a complete, robust neural network. We know how to train it and how to prevent it from overfitting. But this leaves a nagging theoretical question.
We know these models are powerful, but *how* powerful? A linear model can only learn lines. A decision tree can only learn axis-aligned rectangles. What is the fundamental limit of what a neural network can learn?
In the next lesson, we will explore the **Universal Approximation Theorem**, the mind-bending theoretical result that proves a simple neural network with just one hidden layer can, in principle, approximate *any* continuous function.