Python Tools for Visual Studio

I do the majority of my software development using the C# language, but I also use C, C++, Perl, JavaScript, and others, including Python. Many of my colleagues believe that some languages just “have the right feel” to them. For me, Python is one of those good languages.

Until recently, I had always programmed scripting languages using the plain old Notepad editor. There’s something satisfying about using the simplest editor available. But I decided to try out the Python Tools for Visual Studio (PTVS). I hate the name but really like the tool. In essence, PTVS allows you to edit and run Python programs (OK, “scripts” if you want to get technical) in the Visual Studio program.

PythonToolsForVisualStudioInAction

Let me cut to the chase and say I was very impressed by PTVS. The installation was quick, easy, simple, and trouble-free. I am always fearful of installing third-party tools because if my system gets messed up it can be very painful to recover. After installation, using PTVS was easy and intuitive, although part of this ease-of-use is probably due to my years of experience with Visual Studio.

Now, to be honest, PTVS wasn’t earth-shattering, but three things made writing Python programs much easier than using Notepad. First, the editor colors keywords, string literals, parameters, and so on. I was surprised at how much difference this made. Second, PTVS in VS automatically indents when appropriate. (Important because indentation is syntactically significant in Python). Third, the auto-complete feature called IntelliSense really helps for library function names. In addition to these three features, the debugging mode in Visual Studio was nice too.

Because I’m only a casual user of Python, I don’t know what the industry standard integrated editor is in Python-world. Or if there even is a most-common integrated editor. But for me, because I spend most of my developer hours in Visual Studio, PTVS is perfect.

Posted in Machine Learning, Miscellaneous | Leave a comment

Recap of the 2015 Visual Studio Live Conference in Las Vegas

I presented a workshop “Developing Neural Networks using C#” at the Visual Studio Live conference, March 16-20, 2015. Visual Studio Live is a conference for software developers who use Microsoft technologies. VS Live has been around for many years and currently offers conferences in several cities. The March 2015 event was in Las Vegas, at Bally’s Hotel.

VSLiveInFrontOfTitleSlide

I estimate that Visual Studio Live had somewhere between 1,200 to 2,000 attendees. Most of the attendees I talked to were senior software developers who primarily program using Microsoft technologies, but frequently use non-Microsoft technologies too. Most attendees worked at mid and large size companies like banks, insurance companies, and Web focused companies like Expedia. But there were attendees from small companies too.

Visual Studio Live had about 100 sessions. Talk topics included ASP.NET, SQL Server, Azure, and so on. I think most of the talks were related to Web development, reflecting the technology chaos in that field.

VSLiveBreak

Visual Studio Live is a good conference. What do I mean by that? Several things. First, attendees get good value. All the attendees I talked to said that most, or all, of the talks they heard were very good. Also, about 50% of the attendees in my workshop, had been to a Visual Studio Live conference before. This tells me that they felt they had gotten good value in the past. Finally, I enjoy Visual Studio Live myself — I had a good time, and went back to work with new knowledge and renewed energy.

There are a handful of what I consider good software developer conferences. Compared to these similar conferences, Visual Studio Live is slightly more serious in tone and slightly more homogeneous. Other events sometimes have more peripheral events than Visual Studio Live, and cover things like IT technologies and management. Don’t get me wrong — VS Live is serious, but there are plenty of fun events too. Especially when the venue is Las Vegas.

My all-day neural networks workshop was on Friday and had about 40 attendees. This was a lot more people than I had expected there’d be, and this suggests to me that interest in neural networks is increasing. And this in turn suggests to me that in the coming year or two, there will likely be an increase in interest in other machine learning techniques (such as data clustering) and data science topics (such as the R language).

VSLiveOnBreakFromRight

Conferences like VS Live are too pricey (usually between about $1000 and $3000) for most people to pay for by themselves, so most attendees fees are paid for by their companies. In effect, VS Live is their annual training. I believe that VS Live training is, in the long run, far more effective than having a training company come to an employer’s place of work and deliver training there.

Posted in Conferences | 2 Comments

Neural Network Resilient Back-Propagation (Rprop) using C#

I wrote an article titled “How To Use Resilient Back Propagation To Train Neural Networks” in the March 2015 issue of Visual Studio Magazine. See http://visualstudiomagazine.com/articles/2015/03/01/resilient-back-propagation.aspx.

A neural network (NN) is a software system that makes predictions based on data. For example a NN can predict the winner of a basketball game based on data such as each team’s winning percentage, the average winning percentage of each team’s opponents, and so on.

RPROP_Demo_Run

In essence, a NN is a complicated math function with many numeric constants called weights. Training a NN is the process of using historical data, with known input and output values, to find the values for the NN weights so that the computed output values closely match the known output values in the training data.

There are several algorithms that can be used to train a NN. Each algorithm has pros and cons. The most common technique is called back-propagation. Back-propagation is easy to implement and is usually very fast compared to other training techniques. However, back-propagation requires the user to supply two parameter values, for the “learning rate” and the “momentum factor”. Back-propagation is extremely sensitive to the learning rate and momentum factor values. For example, when the learning rate is 0.05 and the momentum is 0.01 you might get a great NN, but for a learning rate of 0.04 and a momentum of 0.02 you might get a terrible NN.

Resilient back-propagation, Rprop, is a fascinating variation of regular back-propagation. It does not require the user to specify a learning rate and a momentum factor. In some of my experiments, Rprop works incredibly well.

I don’t see Rprop used very much. I suspect this is in part due to the fact that Rprop is much harder to implement than regular back-propagation.

Posted in Machine Learning | Leave a comment

Singular and Plural Words using .NET PluralizationServices

Here’s an example of how to use the .NET PluralizationServices library. I usually work with numbers but sometimes I have to work on natural language processing (NLP) projects. A basic NLP task is converting a plural form of a word (such as “processes” or “mice”) to the singular form (“process”, “mouse”), or vice versa.

DotNetPluralizationServices

It’s a surprisingly difficult task if you try to code from scratch. But the .NET Framework 4.5 introduced a PluralizationServices library that’s very handy. The demo code:

using System;
// add reference to System.Data.Entity.Design
using System.Data.Entity.Design.PluralizationServices; 
// visible by default
using System.Globalization;

namespace SingularAndPlural
{
  class Program
  {
    static void Main(string[] args)
    {
      Console.WriteLine("\nBegin singular and plural words demo");
      Console.WriteLine("Example of .NET PluralizationServices\n");
      CultureInfo ci = new CultureInfo("en-us");
      PluralizationService ps =
        PluralizationService.CreateService(ci);

      string word = "die";

      if (ps.IsSingular(word) == true)
      {
        Console.WriteLine("The word " + word + " is singluar");
        string plural = ps.Pluralize(word);
        Console.WriteLine("The plural form is " + plural);
      }

      Console.WriteLine("\nModifying the service");
      ((ICustomPluralizationMapping)ps).AddWord("die", "dice");
      string pluralAlt = ps.Pluralize(word);
      Console.WriteLine("\nNow the plural form is " + pluralAlt);

      Console.WriteLine("\nEnd demo\n");
      Console.ReadLine();
    } // Main
  } // Program
} // ns

The code should be pretty self-explanatory. The demo converts the singular word “die” to its two plural forms, the default “dies” and an alternative “dice” using methods IsSingular and Pluralize. The service also has methods IsPlural and Singularize.

Posted in Machine Learning

Gradient Descent Training for Logistic Regression

I wrote an article titled “Gradient Descent Training Using C#” in the March 2013 issue of the Microsoft MSDN Magazine. See https://msdn.microsoft.com/en-us/magazine/dn913188.aspx.

Gradient descent is an idea that’s simple once you understand it, but hard to grasp at first. In many machine learning (ML) problems, you need to find the values for a set of weight variables so that some measure of error is minimized. The measure of error will vary as the value of a weight changes. In calculus, a quantity called the partial derivative, is a measure that indicates how much and in what direction (+ or -) a weight value should change in order to lessen error.

GradientGraph

The set of all partial derivatives (one for each weight) is called the gradient. Notice that the term “gradient” is singular but it has multiple components. For simplicity, each partial derivative is often called a gradient even though that’s not technically correct.

Gradient descent adjusts weights so that error is lessened. Graphically, you move down (descent) an error curve using the partial derivative (gradient).

In the MSDN Magazine article, I show how to use gradient descent to find the weight values for a logistic regression problem that predicts a binary result (true or false) for some synthetic data.

Posted in Machine Learning

Inverting a Matrix using C#

Inverting a matrix is a surprisingly difficult challenge. I have my own library of C# matrix routines. I like to control my own code rather than relying on magic black box implementations, and I generally prefer to implement matrices using a plain array-of-arrays style rather than using an OOP approach.

I tested the code below by generating one million random matrices, with a dimension between 10 and 100, where each cell is a random value between -100.0 and +100.0. For each random matrix, I computed its inverse, then multiplied the inverse by the original matrix, and checked if the result was the identity matrix.

int n = rnd.Next(10, 100);
double[][] m = MatrixRandom(n, n, seed);
double[][] i = MatrixInverse(m);
double[][] I = MatrixIdentity(n);
double[][] p = MatrixProduct(m, i);
if (MatrixAreEqual(p, I, 1.0E-8))
  // pass
else
  // fail

The code passed 999,9999 out of 1,000,000 test cases. It failed once because of a round-off error. One of the problems with matrix inversion is that sometimes it just doesn’t work.

The code is listed below. I always have trouble with the less-than and greater-than symbols so I did a text replacement for those symbols.

    static double[][] MatrixCreate(int rows, int cols)
    {
      double[][] result = new double[rows][];
      for (int i = 0; i less-than rows; ++i)
        result[i] = new double[cols];
      return result;
    }

    // --------------------------------------------------

    static double[][] MatrixRandom(int rows, int cols,
      double minVal, double maxVal, int seed)
    {
      // return a matrix with random values
      Random ran = new Random(seed);
      double[][] result = MatrixCreate(rows, cols);
      for (int i = 0; i less-than rows; ++i)
        for (int j = 0; j less-than cols; ++j)
          result[i][j] = (maxVal - minVal) *
            ran.NextDouble() + minVal;
      return result;
    }

    // --------------------------------------------------

    static double[][] MatrixIdentity(int n)
    {
      // return an n x n Identity matrix
      double[][] result = MatrixCreate(n, n);
      for (int i = 0; i less-than n; ++i)
        result[i][i] = 1.0;

      return result;
    }

    // --------------------------------------------------

    static string MatrixAsString(double[][] matrix, int dec)
    {
      string s = "";
      for (int i = 0; i less-than matrix.Length; ++i)
      {
        for (int j = 0; j less-than matrix[i].Length; ++j)
          s += matrix[i][j].ToString("F" + dec).PadLeft(8) + " ";
        s += Environment.NewLine;
      }
      return s;
    }

    // --------------------------------------------------

    static bool MatrixAreEqual(double[][] matrixA,
      double[][] matrixB, double epsilon)
    {
      // true if all values in matrixA == values in matrixB
      int aRows = matrixA.Length; int aCols = matrixA[0].Length;
      int bRows = matrixB.Length; int bCols = matrixB[0].Length;
      if (aRows != bRows || aCols != bCols)
        throw new Exception("Non-conformable matrices");

      for (int i = 0; i less-than aRows; ++i) // each row of A and B
        for (int j = 0; j less-than aCols; ++j) // each col of A and B
          //if (matrixA[i][j] != matrixB[i][j])
          if (Math.Abs(matrixA[i][j] - matrixB[i][j]) greater-than epsilon)
            return false;
      return true;
    }

    // --------------------------------------------------

    static double[][] MatrixProduct(double[][] matrixA, double[][] matrixB)
    {
      int aRows = matrixA.Length; int aCols = matrixA[0].Length;
      int bRows = matrixB.Length; int bCols = matrixB[0].Length;
      if (aCols != bRows)
        throw new Exception("Non-conformable matrices in MatrixProduct");

      double[][] result = MatrixCreate(aRows, bCols);

      for (int i = 0; i less-than aRows; ++i) // each row of A
        for (int j = 0; j less-than bCols; ++j) // each col of B
          for (int k = 0; k less-than aCols; ++k) // could use k less-than bRows
            result[i][j] += matrixA[i][k] * matrixB[k][j];

      //Parallel.For(0, aRows, i =greater-than
      //  {
      //    for (int j = 0; j less-than bCols; ++j) // each col of B
      //      for (int k = 0; k less-than aCols; ++k) // could use k less-than bRows
      //        result[i][j] += matrixA[i][k] * matrixB[k][j];
      //  }
      //);

      return result;
    }

    // --------------------------------------------------

    static double[] MatrixVectorProduct(double[][] matrix,
      double[] vector)
    {
      // result of multiplying an n x m matrix by a m x 1 
      // column vector (yielding an n x 1 column vector)
      int mRows = matrix.Length; int mCols = matrix[0].Length;
      int vRows = vector.Length;
      if (mCols != vRows)
        throw new Exception("Non-conformable matrix and vector");
      double[] result = new double[mRows]; 
      for (int i = 0; i less-than mRows; ++i)
        for (int j = 0; j less-than mCols; ++j)
          result[i] += matrix[i][j] * vector[j];
      return result;
    }

    // --------------------------------------------------

    static double[][] MatrixDecompose(double[][] matrix, out int[] perm,
      out int toggle)
    {
      // Doolittle LUP decomposition with partial pivoting.
      // rerturns: result is L (with 1s on diagonal) and U;
      // perm holds row permutations; toggle is +1 or -1 (even or odd)
      int rows = matrix.Length;
      int cols = matrix[0].Length; // assume square
      if (rows != cols)
        throw new Exception("Attempt to decompose a non-square m");

      int n = rows; // convenience

      double[][] result = MatrixDuplicate(matrix); 

      perm = new int[n]; // set up row permutation result
      for (int i = 0; i less-than n; ++i) { perm[i] = i; }

      toggle = 1; // toggle tracks row swaps.
      // +1 -greater-than even, -1 -greater-than odd. used by MatrixDeterminant

      for (int j = 0; j less-than n - 1; ++j) // each column
      {
        double colMax = Math.Abs(result[j][j]); // find largest val in col
        int pRow = j;
        //for (int i = j + 1; i less-than n; ++i)
        //{
        //  if (result[i][j] greater-than colMax)
        //  {
        //    colMax = result[i][j];
        //    pRow = i;
        //  }
        //}

        // reader Matt V needed this:
        for (int i = j + 1; i less-than n; ++i) 
        {
          if (Math.Abs(result[i][j]) greater-than colMax)
          {
            colMax = Math.Abs(result[i][j]);
            pRow = i;
          }
        }
        // Not sure if this approach is needed always, or not.

        if (pRow != j) // if largest value not on pivot, swap rows
        {
          double[] rowPtr = result[pRow];
          result[pRow] = result[j];
          result[j] = rowPtr;

          int tmp = perm[pRow]; // and swap perm info
          perm[pRow] = perm[j];
          perm[j] = tmp;

          toggle = -toggle; // adjust the row-swap toggle
        }

        // --------------------------------------------------
        // This part added later (not in original)
        // and replaces the 'return null' below.
        // if there is a 0 on the diagonal, find a good row
        // from i = j+1 down that doesn't have
        // a 0 in column j, and swap that good row with row j
        // --------------------------------------------------

        if (result[j][j] == 0.0)
        {
          // find a good row to swap
          int goodRow = -1;
          for (int row = j + 1; row less-than n; ++row)
          {
            if (result[row][j] != 0.0)
              goodRow = row;
          }

          if (goodRow == -1)
            throw new Exception("Cannot use Doolittle's method");

          // swap rows so 0.0 no longer on diagonal
          double[] rowPtr = result[goodRow];
          result[goodRow] = result[j];
          result[j] = rowPtr;

          int tmp = perm[goodRow]; // and swap perm info
          perm[goodRow] = perm[j];
          perm[j] = tmp;

          toggle = -toggle; // adjust the row-swap toggle
        }
        // --------------------------------------------------
        // if diagonal after swap is zero . .
        //if (Math.Abs(result[j][j]) less-than 1.0E-20) 
        //  return null; // consider a throw

        for (int i = j + 1; i less-than n; ++i)
        {
          result[i][j] /= result[j][j];
          for (int k = j + 1; k less-than n; ++k)
          {
            result[i][k] -= result[i][j] * result[j][k];
          }
        }


      } // main j column loop

      return result;
    } // MatrixDecompose

    // --------------------------------------------------

    static double[][] MatrixInverse(double[][] matrix)
    {
      int n = matrix.Length;
      double[][] result = MatrixDuplicate(matrix);

      int[] perm;
      int toggle;
      double[][] lum = MatrixDecompose(matrix, out perm,
        out toggle);
      if (lum == null)
        throw new Exception("Unable to compute inverse");

      double[] b = new double[n];
      for (int i = 0; i less-than n; ++i)
      {
        for (int j = 0; j less-than n; ++j)
        {
          if (i == perm[j])
            b[j] = 1.0;
          else
            b[j] = 0.0;
        }

        double[] x = HelperSolve(lum, b); // 

        for (int j = 0; j less-than n; ++j)
          result[j][i] = x[j];
      }
      return result;
    }

    // --------------------------------------------------

    static double MatrixDeterminant(double[][] matrix)
    {
      int[] perm;
      int toggle;
      double[][] lum = MatrixDecompose(matrix, out perm, out toggle);
      if (lum == null)
        throw new Exception("Unable to compute MatrixDeterminant");
      double result = toggle;
      for (int i = 0; i less-than lum.Length; ++i)
        result *= lum[i][i];
      return result;
    }

    // --------------------------------------------------

    static double[] HelperSolve(double[][] luMatrix, double[] b)
    {
      // before calling this helper, permute b using the perm array
      // from MatrixDecompose that generated luMatrix
      int n = luMatrix.Length;
      double[] x = new double[n];
      b.CopyTo(x, 0);

      for (int i = 1; i less-than n; ++i)
      {
        double sum = x[i];
        for (int j = 0; j less-than i; ++j)
          sum -= luMatrix[i][j] * x[j];
        x[i] = sum;
      }

      x[n - 1] /= luMatrix[n - 1][n - 1];
      for (int i = n - 2; i greater-than-equal 0; --i)
      {
        double sum = x[i];
        for (int j = i + 1; j less-than n; ++j)
          sum -= luMatrix[i][j] * x[j];
        x[i] = sum / luMatrix[i][i];
      }

      return x;
    }

    // --------------------------------------------------

    static double[] SystemSolve(double[][] A, double[] b)
    {
      // Solve Ax = b
      int n = A.Length;

      // 1. decompose A
      int[] perm;
      int toggle;
      double[][] luMatrix = MatrixDecompose(A, out perm,
        out toggle);
      if (luMatrix == null)
        return null;

      // 2. permute b according to perm[] into bp
      double[] bp = new double[b.Length];
      for (int i = 0; i less-than n; ++i)
        bp[i] = b[perm[i]];

      // 3. call helper
      double[] x = HelperSolve(luMatrix, bp);
      return x;
    } // SystemSolve

    // --------------------------------------------------

    static double[][] MatrixDuplicate(double[][] matrix)
    {
      // allocates/creates a duplicate of a matrix.
      double[][] result = MatrixCreate(matrix.Length, matrix[0].Length);
      for (int i = 0; i less-than matrix.Length; ++i) // copy the values
        for (int j = 0; j less-than matrix[i].Length; ++j)
          result[i][j] = matrix[i][j];
      return result;
    }

    // --------------------------------------------------
Posted in Machine Learning | 2 Comments

Why I Don’t Like the F# Language

I do research and create software systems. My primary programming languages are C# and Python but I frequently use Perl, Java, JavaScript, and many others. A few of my colleagues, whose opinions I respect, are big fans of the F# programming language. But I am not, and I use F# only when I have to. I was a very early adopter of F# and I really, really wanted to like the language but it just didn’t happen.

FSharpAndCSharp

There are several reasons why I don’t like F# very much. Originally, I tried to categorize these reasons objectively and logically, but the reasons are subjective and overlap, and are all related to each other. But here they are, as best as I can explain, along with some comments.

1. F# just doesn’t feel right.

This is purely subjective, but it’s a huge factor and is an absolute deal-killer for F# for me. Some programming languages, like C# and C, just feel right to me and working with them comes naturally. Other languages, like R and classic Visual Basic, feel neutral to me. But some languages, including F#, Prolog, LISP, and SQL, seem to fight me rather than assist me.

2. F# has a tiny user base.

I did a quick search on the day I wrote this post at a job aggregation site and found 109 job listings that mentioned F#. There were over 34,000 job listings that mentioned C#. And at MSDN Magazine, where I’m the Senior Contributing Editor, our F# articles get very few reads. The tiny user base means there is relatively weak community technical support on sites like Stack Overflow, compared to mainstream languages. Additionally, unlike other languages with relatively few users (such as R), there’s no real motivation for me to adopt F# from a career point of view, because of the very limited job opportunities.

3. F# has no compelling technical advantage.

Some overly zealous F# fans claim that F# can do things that general purpose languages like Perl and C# can’t. This all depends on how you define features of languages. But more reasonable F# proponents usually state, correctly, that F# isn’t intended to replace languages like C# and that F# doesn’t have any unique, magic capabilities. So, there’s no technical reason for me to use F#, and the cost of context switching between the primarily procedural C# and the primarily functional F# is a huge price to pay.

Additional Comments

Obviously, one reason why I’m more comfortable working with C# than F# is that I use C# more often. I learned programming using K&K BASIC and C. F# works at a higher level of abstraction than C#, meaning F# is further away from the machine language instructions that all languages ultimately translate to. When I write C-family code, I have a very good idea of how that code will be translated down to machine language. For me, not so with F#.

I don’t like the terseness of F#. Much like Perl, I find myself writing slick, concise F# code, then I look at that code a month later and have difficulty understanding my own code. In a related issue, I am absolutely not in favor of type inference. When I code, I want to specify exactly my data types.

I understand the advantages of functional programming in general, but I can write C# or Java code using a functional approach if I choose to do so. I strongly suspect that the preference for functional programming vs. procedural programming is an inherent personality characteristic in some way, sort of like left handedness and right handedness. This could explain why programmers who like F# really like F#, and vice versa.

There’s a psychological “don’t be a COBOL programmer” issue. Technology is constantly advancing but there are a few — very few — engineers and researchers who have an attitude along the lines of, “Technology X works great and I don’t need to know anything new.” So, there’s pressure to always learn the latest and greatest new programming language, because nobody wants to be labeled as the old guy who can’t change with the times. Well, it’s true that it’s important to stay current, but just because a language or technology is new, doesn’t automatically make it better. Sometimes a new technology is just a solution in search of a problem. A few irritatingly vocal people in the F# community implicitly try to guilt non-believers into F# adoption. This false you-must-learn-F# argument may persuade inexperienced programmers but that tactic just annoys experienced engineers and researchers.

I have no issue with any developer or researcher who likes F# but in my opinion F# will always be a niche programming language like Prolog or LISP.

Posted in Machine Learning, Miscellaneous | 1 Comment