Why I Don’t Use Neural Network Early Stopping

I was chit-chatting with a work colleague about early stopping. We were next to our workplace coffee-machine / engineer-fueling-station. I almost never use early stopping for neural network training. Briefly, 1.) it’s not really possible to determine when to stop, and 2.) the gain from early stopping doesn’t outweigh the downside cost of having less training data due to the need for validation data.

The graph on the left shows early stopping in theory. Instead of dividing your labeled data into a train (maybe 80% of the data) and a test (20%) dataset as usual, you divide into train (perhaps 70%), validate (15%), and test (15%) datasets. During training, every so often you evaluate the network error/loss on the training and validation data. At some point the validation error will start to increase, indicating that overfitting has started to occur.

Unfortunately, the nice smooth, easy-to-interpret graph on the left almost never happens. The graph on the right shows validation error on an actual training run (I found the graph on the Internet). The graph jumps around and if you stopped training around epoch 50, when validation error starts to increase, you’d be early stopping too early.

There are some scenarios where a conservative form of early stopping is useful. For example, if you are doing programmatic hyperparameter optimization/tuning, a bad early-stop doesn’t hurt too much because you’ll likely get another chance so to speak.

I don’t know much about art, but I suspect that an artist has a difficult time knowing when to stop work on a piece of art. Here are three left-facing portraits by artists whose work I admire a lot. Left: By Hans Jochem Bakker. Center: By Randy Monteith. Right: By Daniel Arrhakis.

This entry was posted in Machine Learning. Bookmark the permalink.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s