Category Archives: Machine Learning

Why Neural Network Training Momentum Isn’t Used Very Often

During neural network training, it’s possible to use a momentum factor. Momentum is a technique designed to speed up training. But I hardly ever see momentum used. The main problem with momentum is that it adds another hyperparameter, the momentum … Continue reading

Posted in Machine Learning | 1 Comment

Mahalanobis Distance Example Using Python

Suppose you have a source dataset of five items where each item is a person’s height, test-score, age: [64.0, 580.0, 29.0] [66.0, 570.0, 33.0] [68.0, 590.0, 37.0] [69.0, 660.0, 46.0] [73.0, 600.0, 55.0] And suppose you want the Mahalanobis distance … Continue reading

Posted in Machine Learning | Leave a comment

“Researchers Explore Machine Learning Hyperparameter Tuning Using Evolutionary Optimization” on the Pure AI Web Site

I contributed to an article titled “Researchers Explore Machine Learning Hyperparameter Tuning Using Evolutionary Optimization” in the November 2022 edition of the Pure AI web site. See https://pureai.com/articles/2022/11/01/evolutionary-optimization.aspx. When data scientists create a machine learning prediction model, there are typically … Continue reading

Posted in Machine Learning | Leave a comment

Simple Numerical Optimization Using an Evolutionary Algorithm with C#

The goal of a numerical optimization problem is to find a vector of values that minimize some cost function. The most fundamental example is minimizing the Sphere Function f(x0, x1, .. xn) = x0^2 + x1^2 + .. + xn^2. … Continue reading

Posted in Machine Learning | Leave a comment

Traveling Salesman Problem Combinatorial Optimization Using an Evolutionary Algorithm with C#

A combinatorial optimization problem is one where the goal is to place items in a correct order. The classic example is the Traveling Salesman Problem (TSP). Suppose you have n = 20 cities that are numbered 0 to 19. There … Continue reading

Posted in Machine Learning | 1 Comment

Why I Don’t Use Neural Network Early Stopping

I was chit-chatting with a work colleague about early stopping. We were next to our workplace coffee-machine / engineer-fueling-station. I almost never use early stopping for neural network training. Briefly, 1.) it’s not really possible to determine when to stop, … Continue reading

Posted in Machine Learning | Leave a comment

Hyperparameter Tuning Using Evolutionary Optimization

All neural systems have hyperparameter values that must be determined by trial and error. Architecture hyperparameters include things like number of hidden layers, and hidden node activation. Training hyperparameters include things like learning rate and batch size. The system random … Continue reading

Posted in Machine Learning | Leave a comment

Hyperparameter Tuning Using Distributed Grid Descent

I was chit-chatting with one of my work colleagues about an algorithm he created for hyperparameter tuning. The algorithm is called Distributed Grid Descent (DGD). Every neural prediction system has hyperparameters such as training learning rate, batch size, architecture number … Continue reading

Posted in Machine Learning | 1 Comment

A Quick Look at TorchSharp

I ran across an interesting GitHub project called TorchSharp. Briefly, TorchSharp is PyTorch without the Python. Let me try to explain. PyTorch is 1.) a Python language wrapper over the libtorch.dll C++ library that implements low-level tensor functionality, and 2.) … Continue reading

Posted in Machine Learning | Leave a comment

Lasso Regression From Scratch Using Python

Lasso regression is just linear regression with L1 regularization. Let me explain. A linear regression problem is one where the goal is to predict a single numeric value from one or more numeric predictor values. For example, you might want … Continue reading

Posted in Machine Learning | 2 Comments