In the October 2012 issue of MSDN Magazine, I wrote an article titled “Neural Network Back-Propagation for Programmers”. See http://msdn.microsoft.com/en-us/magazine/jj658979.aspx.
You’ve probably heard the term neural network before. An artificial (software) neural network models biological neurons and synapses to create a system that can be used to make predictions. Examples include predicting football scores and predicting stock market prices. The basic idea is to use a set of training data (with known inputs and outputs) to tune the neural network — that is, find the set of numeric constants (called weights and biases) that result in the best fit of the training data. Then the neural network, using the best constants, can make predictions on new data inputs with unknown outputs. One of the main challenges when working with neural networks is training to find the best set of constants. The most common technique (but by no means the only one) is called back-propagation.
Back propagation uses Calculus to determine the direction and magnitude of neural network errors on the training data, and then modifies the constants appropriately. The process is repeated until some stopping condition is met.
Neural networks are simple conceptually, but the details are extremely tricky. Furthermore, even though there are many online references that explain the interesting math of neural networks, I did not find any good references that provide a clear explanation of how to actually code neural networks and the back-propagation algorithm. So I set out to do so and wrote up what I believe is a clear explanation.