Overfitting Monster
Don't memorise!
Your mission
Turn up a model's complexity and it fits its training photos better and better, until a monster appears: it has memorised the training data and fails on new photos. Watch two live charts pull apart, find the sweet spot, clean the data, and choose a model that generalises.
Explore
Move the complexity slider from simple to complex, and watch the shading and the two charts.
A model is learning to tell cats (circles) from dogs (squares) by two measurements. You control how complex it may become. The shaded areas show where it would say cat or dog. Two live charts show how it does on its training photos and on separate validation photos. Something lurks at the complex end...
0 of 6 discoveries made
Experiment
Give two models a final exam, find out why the monster makes mistakes, clean the data, and test new animals.
The test set is 30 more photos, kept locked away until the very end. Choose a complexity, then let the model sit its final exam.
A few training photos have the wrong label. A very complex model learns those mistakes too. At level 8, tap a red-outlined validation photo to see why it went wrong.
Three animals the model has never seen. Ask it at different complexities.
Monster-taming experiments
Try each one. They tick themselves off.
0 of 5 experiments done
Challenge
Six rounds: spot overfitting in the numbers, choose the right model, and explain how to tame the monster.
Monster-hunter's guide
- Overfitting: great on training data, poor on new data
- A big gap between training and validation = warning
- Choose the model by its validation accuracy
- The test set is only for the final check
- A good model generalises; it doesn't memorise
Finish all 6 rounds to finish this stage.
Understand
Here is what the monster was showing you, in the words you need for the exam.
What is overfitting?
Overfitting occurs when a model learns the training data too closely and performs poorly on new, unseen examples. At level 8 the model scored 100% on its training photos but only about two in three on the validation photos: it had memorised every training photo, including the mislabelled ones, instead of learning the general pattern.
Key concept: a good model should learn patterns that generalise, not simply memorise its training examples.
The student analogy
- Student A memorises answers: remembers the exact answers to previous questions, performs well on familiar questions, but struggles when the questions change. That is an overfitted model.
- Student B understands concepts: understands the underlying ideas and can solve new and different problems. That is a model that generalises.
Training vs new data
| Model | Training data | Unseen data |
|---|---|---|
| Overfitted | ≈ 98% | ≈ 55% |
| Better-generalising | ≈ 92% | ≈ 88% |
Illustrative example from the lecture, not experimental results. The warning sign is the gap.
Why we need a validation set
Training accuracy always looked best at the most complex setting, so it can't tell you which model to pick. The validation set is used to compare models and tune settings, such as complexity, on data the model did not learn from. The test set is kept back for one final evaluation on held-out data. The very simple settings did badly on both sets: that model was too simple to capture the pattern at all.
How this model worked
To decide about a photo, this model looked at the most similar training photos and went with the majority. At "simple" it asked many neighbours (39 at level 1); at "complex" it copied the single nearest training photo (level 8). Copying single photos means copying their mistakes: the islands in the shading.
What helps?
- Choose the complexity by validation accuracy, not training accuracy.
- Clean the data: mislabelled examples teach wrong patterns.
- Prefer a model that generalises: a slightly lower training score is fine if it does well on new data.
Words to know
- Overfitting
- Learning the training data too closely, then performing poorly on new data.
- Generalise
- Perform well on new examples that were not used for training.
- Training accuracy
- How often the model is right on the data it learned from.
- Validation accuracy
- How often it is right on separate data, used to choose a model.
- Test set
- Held-out data for the final evaluation.
- Model complexity
- How closely a model is allowed to follow individual training examples.
Exam tip
To spot overfitting, compare two numbers: high accuracy on the training data and much lower accuracy on unseen data. Use the student analogy to explain it: memorising answers vs understanding concepts. And say which data to trust: models are compared on validation data and finally evaluated on test data, never only on the data they were trained on.