Coding Logistic Regression with Newton-Raphson

The September 2012 issue of Microsoft’s MSDN Magazine has an article I wrote titled, “Coding Logistic Regression with Newton-Raphson”. See Logistic regression is a machine learning technique used to predict which of two categories some data belongs to. The technique works by analyzing a set of existing training data to find the best set of b parameters that generate an equation of the form:

y = 1.0 / (1.0 + exp(-z))


z = b0 + (b1)(x1) + (b2)(x2) + . . . (bn)(xn)

In the example screenshot below, the training data X values are x1 = a patient’s age, x2 = sex, and x3 = cholesterol level. The Y value to predict is death or not. The challenge is to find the set of b0, b1, b2, b3 that lead to the best fit with the training data. There are several techniques to estimate the best values for the b parameters, including a technique based on calculus derivatives called Newton-Raphson. By the way, the actual technique in general is called iteratively reweighted least squares, and Newton-Raphson is one particular way to perform iteratively reweighted least squares. Because of the close connection, the two terms are typically used interchangeably. Logistic regression classification and Naïve Bayes classification are generally considered by my colleagues as the two most fundamental machine learning techniques.

This entry was posted in Machine Learning. Bookmark the permalink.