Author Archives: jamesdmccaffrey

During neural network training, it’s possible to use a momentum factor. Momentum is a technique designed to speed up training. But I hardly ever see momentum used. The main problem with momentum is that it adds another hyperparameter, the momentum … Continue reading →

Suppose you have a source dataset of five items where each item is a person’s height, test-score, age: [64.0, 580.0, 29.0] [66.0, 570.0, 33.0] [68.0, 590.0, 37.0] [69.0, 660.0, 46.0] [73.0, 600.0, 55.0] And suppose you want the Mahalanobis distance … Continue reading →

I contributed to an article titled “Researchers Explore Machine Learning Hyperparameter Tuning Using Evolutionary Optimization” in the November 2022 edition of the Pure AI web site. See https://pureai.com/articles/2022/11/01/evolutionary-optimization.aspx. When data scientists create a machine learning prediction model, there are typically … Continue reading →

The goal of a numerical optimization problem is to find a vector of values that minimize some cost function. The most fundamental example is minimizing the Sphere Function f(x0, x1, .. xn) = x0^2 + x1^2 + .. + xn^2. … Continue reading →