I recently upgraded my Keras neural network code library version to 2.6.0 and decided to revisit my three basic examples — Iris (multi-class classification), Banknote (binary classification), and Boston (regression). This morning I refactored my Boston example.

Even though it had only been a few months since I last did the Boston example, I was surprised at how much Keras had changed, and how much my preferred techniques had changed.

The goal of the Boston Housing Dataset example is to predict the median house price in one of 506 towns near Boston. There are 13 predictor variables — average number of rooms in a house in the town, tax rate in the town, percentage of Black people in town, and so on.

I used order-magnitude normalization on the numeric predictors, then randomly split the 506-item into a training set (400 items) and a test set (106 items). Preparing even simple data like this is tedious and very time-consuming. See https://jamesmccaffrey.wordpress.com/2021/08/18/preparing-the-boston-housing-dataset-for-pytorch/.

For my new version of the Boston example, one of the main changes I made was to write a much faster function to compute model accuracy. When you do classification with Keras, like the Iris example, you get a built-in accuracy function. But with regression, you must write your own function. For classification, a prediction of a discrete value like “red” is either correct or incorrect. But for regression, a prediction of a house price like 0.49500 = $49,500 will never be perfectly correct so you must determine if the prediction is within a specified percent of the target value.

When I use the PyTorch library, my standard approach for regression accuracy is:

loop get a line of input data get target value feed input to model, get predicted value if predicted is close enough to target num_correct += 1 else num_wrong += 1 end-if end-loop return num_correct / num_correct + num_wrong

This approach examines one data item at a time. The technique is simple, and allows you to examine each item to see why a prediction was correct or wrong. But when using Keras, the item-by-item approach is painfully slow. I’m not sure why Keras is roughly 10 times slower than Keras when computing accuracy item-by-item.

So, a quick approach is useful. The idea is to compute all outputs at once.

read all input data read all target values feed all input, get all predicted values compare all predicted to all targets ( result is like [1, 0, 0, 1, . . ] ) sum the comparison results return sum / num_items

The ideas are simple and the code is short, but implementation is very, very tricky. Here is my implementation for the Boston data:

def accuracy_quick(model, data_x, data_y, pct): n = len(data_x) oupt = model(data_x) oupt = tf.reshape(oupt, [-1]) # 1D max_deltas = tf.abs(pct * data_y) # max allowable deltas abs_deltas = tf.abs(oupt - data_y) # actual differences results = abs_deltas "lt" max_deltas # [True, False, . .] n_correct = np.sum(results) acc = n_correct / n return acc

Even though a quick accuracy function implementation is tricky, once you know how to compute quick accuracy for one specific regression problem, it’s relatively easy to adapt the code to any other regression problem.

*The digital artist who goes by the name “batjorge” uses fractal generation software (Mandelbulb) to create interesting images of alien mushrooms and fungi. Are the images accurate? For art, the concept of accuracy doesn’t usually apply.*

Code below. Very long.

# boston_tfk.py # regression on the Boston Housing dataset # Keras 2.6.0 in TensorFlow 2.6.0 ("_tfk") # Anaconda3-2020.02 Python 3.7.6 Windows 10 import os os.environ['TF_CPP_MIN_LOG_LEVEL']='2' # suppress CPU warn import numpy as np import tensorflow as tf from tensorflow import keras as K class MyLogger(K.callbacks.Callback): def __init__(self, n, data_x, data_y, pct_close): self.n = n # print loss & acc every n epochs self.data_x = data_x self.data_y = data_y self.pct_close = pct_close def on_epoch_end(self, epoch, logs={}): if epoch % self.n == 0: curr_loss = logs.get('loss') # loss on curr batch print("epoch = %4d curr batch loss (MSE) = \ %0.6f " % (epoch, curr_loss)) def accuracy_slow(model, data_x, data_y, pct_close): # item-by-item -- slow -- for debugging n_correct = 0; n_wrong = 0 n = len(data_x) for i in range(n): x = np.array([data_x[i]]) # [[ x ]] predicted = model.predict(x) actual = data_y[i] if np.abs(predicted[0][0] - actual) "lt" \ np.abs(pct_close * actual): n_correct += 1 else: n_wrong += 1 return (n_correct * 1.0) / (n_correct + n_wrong) def accuracy_quick(model, data_x, data_y, pct): n = len(data_x) oupt = model(data_x) oupt = tf.reshape(oupt, [-1]) # 1D max_deltas = tf.abs(pct * data_y) # max allowable deltas abs_deltas = tf.abs(oupt - data_y) # actual differences results = abs_deltas "lt" max_deltas # [True, False, . .] n_correct = np.sum(results) acc = n_correct / n return acc def main(): print("\nBoston Housing regression example ") np.random.seed(1) tf.random.set_seed(1) kv = K.__version__ print("Using Keras: ", kv) print("\nLoading Boston data into memory ") train_file = ".\\Data\\boston_train.txt" # 400 lines train_x = np.loadtxt(train_file, usecols=[0,1,2,3,4,5,6,7,8,9,10,11,12], delimiter="\t", skiprows=0, dtype=np.float32) train_y = np.loadtxt(train_file, usecols=[13], delimiter="\t", skiprows=0, dtype=np.float32) test_file = ".\\Data\\boston_test.txt" # 106 lines test_x = np.loadtxt(test_file, usecols=[0,1,2,3,4,5,6,7,8,9,10,11,12], delimiter="\t", skiprows=0, dtype=np.float32) test_y = np.loadtxt(test_file, usecols=[13], delimiter="\t", skiprows=0, dtype=np.float32) print("\nCreating 13-10-10-1 neural network ") model = K.models.Sequential() model.add(K.layers.Dense(units=10, input_dim=13, activation='tanh')) # hid1 model.add(K.layers.Dense(units=10, activation='tanh')) # hid2 model.add(K.layers.Dense(units=1, activation=None)) # output layer opt = K.optimizers.Adam(learning_rate=0.01) model.compile(loss='mean_squared_error', optimizer=opt, metrics=['mse']) max_epochs = 1000 log_every = 100 my_logger = MyLogger(log_every, train_x, train_y, 0.15) print("\nStarting training ") h = model.fit(train_x, train_y, batch_size=4, epochs=max_epochs, verbose=0, callbacks=[my_logger]) print("Training finished ") eval = model.evaluate(train_x, train_y, verbose=0) # [0] = loss (mse), [1] = compile-metrics = 'mse' again print("\nFinal overall loss (MSE) on train = %0.6f" % eval[0]) eval = model.evaluate(test_x, test_y, verbose=0) print("Final overall loss (MSE) on test = %0.6f" % eval[0]) # loss_list = h.history['loss'] # loss LAST BATCH each epoch # print(loss_list[len(loss_list)-1]) # train_acc = accuracy_slow(model, train_x, train_y, 0.15) # print("\nAccuracy on train data = %0.4f" % train_acc) # test_acc = accuracy_slow(model, test_x, test_y, 0.15) # print("Accuracy on test data = %0.4f" % test_acc) train_acc = accuracy_quick(model, train_x, train_y, 0.15) print("\nAccuracy on train data = %0.4f" % train_acc) test_acc = accuracy_quick(model, test_x, test_y, 0.15) print("Accuracy on test data = %0.4f" % test_acc) print("\nSaving trained model as boston_model.h5 ") # model.save_weights(".\\Models\\boston_model_wts.h5") # model.save(".\\Models\\boston_model.h5") np.set_printoptions(formatter={'float': '{: 0.6f}'.format}) x = np.array([[0.000493, 0.330000, 0.021800, -1, 0.472000, 0.684900, 0.703000, 0.031827, 0.070000, 0.222000, 0.184000, 0.396900, 0.075300]], dtype=np.float32) predicted = model.predict(x) print("\nUsing model to predict median price for features: ") print(x) print("\nPredicted price is: ") # expected = 0.282000 print(predicted) print("\nEnd demo ") if __name__=="__main__": main()

You must be logged in to post a comment.