Why I Don’t Use Min-Max or Z-Score Normalization For Neural Networks

Normalization is the process of scaling numeric predictor values so that they’re all roughly in the same range, typically 0.0 to 1.0 (min-max normalization) or about -4.0 to +4.0 (z-score normalization). Over the past few years I have become quite convinced that neither of these techniques is as good as simple order-magnitude normalization. The ideas are subtle and are best explained by a concrete example.

Note: Encoding is the process of converting non-numeric data, such as a color variable that can be red, blue, or green into numeric values such as (1, 0, 0), (0, 1, 0), (0, 0, 1). Encoding is a separate topic.

Suppose you have just 10 data items where the predictors are a person’s age and some sort of money account balance:

age    balance   val-to-predict
24      450.00        y
36     2750.00        y
57      367.00        y
28    -1730.00        y
44      310.00        y
39      803.00        y
50      189.00        y
47     4120.00        y
29     -203.00        y
31      402.00        y

If you’re going to feed this data to a neural network system, you want to normalize the predictor data so that the large balance values don’t overwhelm the smaller age values. (There are a few ML techniques that don’t need normalization, but most do.)

If you want to use min-max normalization, first you must do a random train-test split, and then find the min and max of only the train data and then use those two values on the train and test data to compute x’ = (x – min) / (max – min) on the train data and then on the test data. You should not use any knowledge about the test data for normalization!

There are several significant problems here. First, the min and max values will depend on how the data was split, and so no two people will normalize a dataset in the same way. Second, you must record and save the min and max values so you can use them to normalize new, previously unseen data on which you wish to make a prediction. And the min and max values are likely to be very ugly and hard-to-remember, like min = 42.3333, max = 57.1825. Third, you lose the plus-minus sign information, which is sometimes very important. Fourth, the normalized scores are difficult to interpret — what is the actual age of a person whose normalized age is 0.35?

If you use z-score normalization, you have the same problems as min-max normalization.

The normalization technique I prefer doesn’t have a standard name, but I call it order-magnitude normalization. It’s super simple: you divide each predictor column by 10 or 100 or 1000 or etc., so that the magnitude of each predictor is between 0.0 and 1.0. For the age values above, you’d divide each by 100. For the balance values, you’d divide each by 10,000.

age    balance   val-to-predict
0.24    0.0450        y
0.36    0.2750        y
0.57    0.0367        y
0.28   -0.1730        y
0.44    0.0310        y
0.39    0.0803        y
0.50    0.0189        y
0.47    0.4120        y
0.29   -0.0203        y
0.31    0.0402        y

First, notice that this technique does not depend on how the data is split into train-test, or train-validate-test. Therefore you can normalize the source data and then split — a huge savings in effort, trust me. Second, you only have to record a single vector of divisor values, such as (10, 1, 100, 1, 10) which is much simpler to use when normalizing new data items. Third, the order-magnitude normalization technique maintains the plus-minus sign information of the source data. Fourth, normalized values are easy to interpret — a normalized age of 0.35 is 35 years old.

The order-magnitude normalization technique is almost never used by anyone except me, or employees at my workplace who have taken a technical training class from me. In fact, I have never seen the technique used in any research or system (although I’m sure it must be used sometimes).

Does this mean I’m wrong about the advantages of order-magnitude normalization compared to min-max or z-score normalization? I don’t think so. Unfortunately, in machine learning, topics are so complicated that everyone who learns neural-based ML accepts traditional lore as unimpeachable gospel truth. This is understandable because anyone who is starting to learn ML is swamped by complex information.

But in neural ML, sometimes a “fact that everyone knows is true” — such as “you should normalize your numeric data using min-max or z-score” — is just not true at all.

---

Common knowledge that everyone knows is true: Dogs 1.) are inherently able to catch frisbees, 2.) are always great at obstacle courses, 3.) are natural enemies of cats.