For the past several weeks, I’ve been looking at a deep neural technique for anomaly detection based on an idea called variational autoencoder (VAE) reconstruction probability. My most recent experiment was to apply the technique to the UCI Digits image dataset. I was pleased with the results.
First I generated a tiny 100-item subset from the 3823-item UCI digits training data. Each item is 64 pixel values (an 8 x 8 grayscale image) where each value is an integer between 0 and 16, followed by the label of the represented digit (0 to 9). During processing, the pixel values are normalized by dividing by 16, and the label is divided by 9. Therefore all 65 values are between 0.0 and 1.0.
The demo VAE has a 65-32-[8,8]-8-32-[65,65]-65 architecture. An encoder component compresses each image down to a mean and standard deviation (in the form of log-variance) that defines a multivariate Normal distribution. A decoder component takes the compressed representation and expands it to a second multivariate Normal distribution, which is then sampled to give a reconstructed version of the input image (and its label). I implemented the anomaly detection system using the PyTorch code library.
The system compares the input image X with the second Normal distribution and computes the probability that the image came from the distribution. A small probability suggests the image is anomalous in some way. The probabilities involved are so small, I had to use the logpdf() function rather than the pdf() function.
It took me several days of experimenting to get the demo to run. There are a lot of hyperparameters to deal with. In addition to all the usual deep neural parameters such as architecture, activation functions, learning rate, and so on, a VAE has a beta weight value that affects the custom loss function that combines regular error with Kullback-Leibler divergence.
Anomaly detection using VAE reconstruction probability is a cousin technique to anomaly detection using autoencoder reconstruction error. I suspect that a hybrid anomaly detection sysyem that combines VAE reconstruction probability with autoencoder reconstruction error might be able to achieve state of the art results. Maybe.
Anomaly detection using VAE reconstruction probability is a compicated topic, but I’m making good progress in unwrapping the mysteries. When I’m satisfied my system is correct, I’ll post the code here on my blog site.