Why a Neural Network is Always Better than Logistic Regression

Logistic regression is a technique that can be used for binary classification — making a prediction when the thing to predict can be one of just two possible values. For example, you might want to predict if a person is male (0) or female (1) based on age, annual income, height, weight, and so on.

A neural network is more complex than logistic regression. And, as I show in the diagram below, logistic regression is a subset of a neural network classifier. To cut to the chase, you can simulate a logistic regression model using a neural network with one hidden node with the identity activation function, and one output node with zero bias and logistic sigmoid activation.

The moral of the story is that, in principle, anything you can do with logistic regression you can do with a neural network. Therefore, theoretically, a neural network is always better than logistic regression, or more precisely, a neural network can do no worse than logistic regression.

In the diagram, there are three input values (1.0, 2.0, 3.0). The logistic regression model on the left emits output value 0.5474 and so does the neural network model on the right. For the male-female example, the prediction would be female because the output value is greater than 0.5 (if the value was less than 0.5 the prediction would be male).

Now all of this is “in theory” and “in principle”. In practice, a neural network model for binary classification can be worse than a logistic regression model because neural networks are more difficult to train and are more prone to overfitting than logistic regression. That said however, the bottom line is that when doing binary classification, using a neural network is better in most cases than using logistic regression. And if you’re careful, you should be able to get better results with a neural network.

This entry was posted in Machine Learning. Bookmark the permalink.