Multi-Swarm Optimization with C#

Multi-swarm numerical optimization is technique that can be used to find values for a set of variables so that some function is minimized (or more rarely, maximized) in situations where classical techniques based on Calculus are not feasible. For the September 2013 issue of Microsoft’s MSDN Magazine, I wrote an article that explains exactly how multi-swarm optimization works. See

Suppose you have some error function f(x0, x1) = 20 + (x0^2 – 10*cos(2*Pi*x0) + x1^2 – 10*cos(2*Pi*x1) and you want to find the values for x0 and x1 that give a minimum value for f. This function is called Rastrigin’s function and it has many hills and valleys that make it difficult to find the best values for x0 and x1, which are x0 = 0.0 and x1 = 0.0, giving an f value of 0.0.

In most cases you want to do numerical optimization as a part of a larger problem. For example, numerical optimization can be used to find the values for the weights and biases of a neural network, which in turn can be used to make predictions. There are many numerical optimization techniques and all have relative strengths and weaknesses.

Multi-swarm optimization is an extension of particle swarm optimization (PSO). PSO models the behavior of groups such as flocks of birds or schools of fish. The position of each virtual particle represents a potential solution and particles/solutions iteratively move to better positions/solutions until some stopping condition is met. A collection of particles is called a swarm.

One of the weaknesses of regular PSO is that results depend to a large extent on the starting position of the particles. Multi-swarm optimization uses several swarms of particles rather than a single swarm. This approach helps overcome the sensitivity to initial positions. Multi-swarm optimization is very robust and is one of my favorite numerical optimization algorithms.



This entry was posted in Machine Learning. Bookmark the permalink.