## Computing Model Accuracy for Keras Regression Models

I recently upgraded my Keras neural network code library version to 2.6.0 and decided to revisit my three basic examples — Iris (multi-class classification), Banknote (binary classification), and Boston (regression). This morning I refactored my Boston example.

Even though it had only been a few months since I last did the Boston example, I was surprised at how much Keras had changed, and how much my preferred techniques had changed. The goal of the Boston Housing Dataset example is to predict the median house price in one of 506 towns near Boston. There are 13 predictor variables — average number of rooms in a house in the town, tax rate in the town, percentage of Black people in town, and so on.

I used order-magnitude normalization on the numeric predictors, then randomly split the 506-item into a training set (400 items) and a test set (106 items). Preparing even simple data like this is tedious and very time-consuming. See https://jamesmccaffrey.wordpress.com/2021/08/18/preparing-the-boston-housing-dataset-for-pytorch/.

For my new version of the Boston example, one of the main changes I made was to write a much faster function to compute model accuracy. When you do classification with Keras, like the Iris example, you get a built-in accuracy function. But with regression, you must write your own function. For classification, a prediction of a discrete value like “red” is either correct or incorrect. But for regression, a prediction of a house price like 0.49500 = \$49,500 will never be perfectly correct so you must determine if the prediction is within a specified percent of the target value.

When I use the PyTorch library, my standard approach for regression accuracy is:

```loop
get a line of input data
get target value
feed input to model, get predicted value
if predicted is close enough to target
num_correct += 1
else
num_wrong += 1
end-if
end-loop
return num_correct / num_correct + num_wrong
```

This approach examines one data item at a time. The technique is simple, and allows you to examine each item to see why a prediction was correct or wrong. But when using Keras, the item-by-item approach is painfully slow. I’m not sure why Keras is roughly 10 times slower than PyTorch when computing accuracy item-by-item.

So, a quick approach is useful. The idea is to compute all outputs at once.

```read all input data
feed all input, get all predicted values
compare all predicted to all targets
( result is like [1, 0, 0, 1, . . ] )
sum the comparison results
return sum / num_items
```

The ideas are simple and the code is short, but implementation is very, very tricky. Here is my implementation for the Boston data:

```def accuracy_quick(model, data_x, data_y, pct):
n = len(data_x)
oupt = model(data_x)
oupt = tf.reshape(oupt, [-1])  # 1D

max_deltas = tf.abs(pct * data_y)    # max allowable deltas
abs_deltas = tf.abs(oupt - data_y)   # actual differences
results = abs_deltas "lt" max_deltas    # [True, False, . .]

n_correct = np.sum(results)
acc = n_correct / n
return acc
```

Even though a quick accuracy function implementation is tricky, once you know how to compute quick accuracy for one specific regression problem, it’s relatively easy to adapt the code to any other regression problem. The digital artist who goes by the name “batjorge” uses fractal generation software (Mandelbulb) to create interesting images of alien mushrooms and fungi. Are the images accurate? For art, the concept of accuracy doesn’t usually apply.

Code below. Very long.

```# boston_tfk.py
# regression on the Boston Housing dataset
# Keras 2.6.0 in TensorFlow 2.6.0 ("_tfk")
# Anaconda3-2020.02  Python 3.7.6  Windows 10

import os
os.environ['TF_CPP_MIN_LOG_LEVEL']='2'  # suppress CPU warn

import numpy as np
import tensorflow as tf
from tensorflow import keras as K

class MyLogger(K.callbacks.Callback):
def __init__(self, n, data_x, data_y, pct_close):
self.n = n   # print loss & acc every n epochs
self.data_x = data_x
self.data_y = data_y
self.pct_close = pct_close

def on_epoch_end(self, epoch, logs={}):
if epoch % self.n == 0:
curr_loss = logs.get('loss')  # loss on curr batch
print("epoch = %4d  curr batch loss (MSE) = \
%0.6f " % (epoch, curr_loss))

def accuracy_slow(model, data_x, data_y, pct_close):
# item-by-item -- slow -- for debugging
n_correct = 0; n_wrong = 0
n = len(data_x)
for i in range(n):
x = np.array([data_x[i]])  # [[ x ]]
predicted = model.predict(x)
actual = data_y[i]
if np.abs(predicted - actual) "lt" \
np.abs(pct_close * actual):
n_correct += 1
else:
n_wrong += 1
return (n_correct * 1.0) / (n_correct + n_wrong)

def accuracy_quick(model, data_x, data_y, pct):
n = len(data_x)
oupt = model(data_x)
oupt = tf.reshape(oupt, [-1])  # 1D

max_deltas = tf.abs(pct * data_y)    # max allowable deltas
abs_deltas = tf.abs(oupt - data_y)   # actual differences
results = abs_deltas "lt" max_deltas    # [True, False, . .]

n_correct = np.sum(results)
acc = n_correct / n
return acc

def main():
print("\nBoston Housing regression example ")
np.random.seed(1)
tf.random.set_seed(1)

kv = K.__version__
print("Using Keras: ", kv)

train_file = ".\\Data\\boston_train.txt"  # 400 lines
usecols=[0,1,2,3,4,5,6,7,8,9,10,11,12],
delimiter="\t", skiprows=0, dtype=np.float32)
delimiter="\t", skiprows=0, dtype=np.float32)

test_file = ".\\Data\\boston_test.txt"  # 106 lines
usecols=[0,1,2,3,4,5,6,7,8,9,10,11,12],
delimiter="\t", skiprows=0, dtype=np.float32)
delimiter="\t", skiprows=0, dtype=np.float32)

print("\nCreating 13-10-10-1 neural network ")
model = K.models.Sequential()
activation='tanh'))  # hid1
activation='tanh'))  # hid2
activation=None))    # output layer
model.compile(loss='mean_squared_error',
optimizer=opt, metrics=['mse'])

max_epochs = 1000
log_every = 100
my_logger = MyLogger(log_every, train_x,
train_y, 0.15)

print("\nStarting training ")
h = model.fit(train_x, train_y, batch_size=4,
epochs=max_epochs, verbose=0, callbacks=[my_logger])
print("Training finished ")

eval = model.evaluate(train_x, train_y, verbose=0)
#  = loss (mse),  = compile-metrics = 'mse' again
print("\nFinal overall loss (MSE) on train = %0.6f" % eval)
eval = model.evaluate(test_x, test_y, verbose=0)
print("Final overall loss (MSE) on test = %0.6f" % eval)

# loss_list = h.history['loss']  # loss LAST BATCH each epoch
# print(loss_list[len(loss_list)-1])

# train_acc = accuracy_slow(model, train_x, train_y, 0.15)
# print("\nAccuracy on train data = %0.4f" % train_acc)
# test_acc = accuracy_slow(model, test_x, test_y, 0.15)
# print("Accuracy on test data = %0.4f" % test_acc)

train_acc = accuracy_quick(model, train_x, train_y, 0.15)
print("\nAccuracy on train data = %0.4f" % train_acc)
test_acc = accuracy_quick(model, test_x, test_y, 0.15)
print("Accuracy on test data = %0.4f" % test_acc)

print("\nSaving trained model as boston_model.h5 ")
# model.save_weights(".\\Models\\boston_model_wts.h5")
# model.save(".\\Models\\boston_model.h5")

np.set_printoptions(formatter={'float': '{: 0.6f}'.format})
x = np.array([[0.000493, 0.330000, 0.021800, -1, 0.472000,
0.684900, 0.703000, 0.031827, 0.070000, 0.222000,
0.184000, 0.396900, 0.075300]],
dtype=np.float32)
predicted = model.predict(x)
print("\nUsing model to predict median price for features: ")
print(x)
print("\nPredicted price is: ")  # expected = 0.282000
print(predicted)

print("\nEnd demo ")

if __name__=="__main__":
main()
```
This entry was posted in Keras. Bookmark the permalink.