Neural Networks are the Foundation of Machine Learning

One of my jobs at work is providing training about Machine Learning (ML), Artificial Intelligence (AI), and Data Science (DS). Here’s how I distinguish these, in my own mind anyway, from simplest to most complex:

shorttalkinbuilding43

1. Data Science – Analyzing data using classical statistics techniques. Example: a hypothesis test to infer if the average salary of men is the same as women, when you only have sample data.

2. Machine Learning – Making predictions using data, often using a basic neural network. Example: predicting the outcome of a football game.

3. Deep Learning – Making predictions using data using a complex (deep) neural network. Example: predicting if an image of a digit is a ‘1’ or a ‘7’.

4. Artificial Intelligence – Doing something with a computer that is typically associated with a human. Often use a deep neural network. Examples: sight (identifying a picture), hearing (Siri and Cortana), speech (Siri and Cortana again), touch (robotics), and reasoning (asking a program a question and getting an intelligent answer).

I believe that an understanding of basic neural networks is a key to understanding machine learning and artificial intelligence. I recently gave a talk about some of these topics.

Posted in Machine Learning | Leave a comment

Running TensorFlow in Bash on Windows

TensorFlow is a library of machine learning code. TensorFlow is intended for use on Linux machines. In principle, it’s possible to compile TensorFlow source code on a Windows machine, but the details are very tricky. So, instead, I decided to see if TensorFlow would work inside a Linux Bash shell running on Window 10. It does.

I think of Bash on Windows as a miniature Linux (Ubuntu to be exact) environment that runs on a Windows machine. Installing Bash on Windows isn’t difficult and there are plenty of reference on the Internet, but briefly: Control Panel | Programs | Windows Features | “Windows Subsystem for Linux (Beta)” | OK. After a reboot, Run | Bash -> Do you want to install? -> Y. Then after the install completes, you must create a Linux User (I used “jamccaff”) and a password for the user.

Under the covers, TensorFlow is written in C++ but you use the Python programming language to interact with Tensorflow, so I needed Python in my Bash shell. Luckily, Bash on Windows comes with Python 2.7 built-in. But to install TensorFlow I needed the PIP (“Pip installs packages”) tool so from the Bash ‘$’ prompt:

$ sudo apt-get install python-pip python-dev
$ pip --version

Next I used pip to install TensorFlow:

$ sudo pip install --upgrade https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.8.0-cp27-none-linux_x86_64.whl

Somewhat miraculously, the package built and installed without too many problems. The main issues I ran into were related to an incompatible version of TensorFlow with my Linux version. Next, I tested the TensorFlow installation:

$ python
>>> import tensorflow as tf
>>> m = tf.constant("This is a TF message")
>>> s = tf.Session()
>>> print(s.run(m))
This is a TF message

testingtensorflowinstallationinbashonwindows

And that worked too. I got ambitious and searched the Internet for a simple logistic regression example and found one that used the MNIST digit image data set, copied that script into the Notepad++ editor, saved it as logreg.py in a Windows directory and successfully ran it too (Bash on Windows can access files in the Windows file system as well as files in the Linux file system. Cool!) There were a few glitches because TensorFlow is still under development and many of the code examples I’ve found required small changes. But I was expecting much more trouble. Bottom line: If you want to run TensorFlow on Windows, use can definitely use Bash for Windows instead of trying to build TensorFlow directly.

Note: I believe TensorFlow in Bash on Windows runs in CPU mode only, not in GPU mode. This could be an important limitation for some scenarios.

logisticregressionwithtensorflowonwindows

Posted in Machine Learning | Leave a comment

NFL 2016 Season – Zoltar Predictions – Final Results

Zoltar is my NFL prediction computer program. The 2016 season ended a couple of weeks ago when the New England Patriots beat the Atlanta Falcons 34-28 in overtime in Super Bowl 51.

Zoltar predicted that the Patriots were 3.0 points better than the Falcons so the prediction was fairly accurate. The Las Vegas point spread also had the Patriots as 3.0 point favorites.

Zoltar theoretically suggests betting when the Vegas line is more than 3.0 points different from Zoltar’s prediction. For the Super Bowl, Zoltar had no suggestion because his prediction and the Vegas point spread line were exactly the same.

zoltar_predicts_nfl

For the 2016 regular season plus the 11 postseason games, Zoltar was 45-29 against the Vegas spread, for 61% accuracy. Historically, Zoltar is usually between 62% and 72% accuracy against the Vegas spread over the course of an entire season, so for 2016 Zoltar was OK but not great.

Theoretically, if you must bet $110 to win $100 (typical) then you’ll make money if you predict at 53% accuracy or better. But realistically, you need to predict at 60% accuracy or better.

Just for fun, I track how well Zoltar and Cortana/Bing Predictions do when just trying to predict just which team will win a game. This isn’t useful except for parlay betting.

For the final game of the season, just predicting the winner, Zoltar was 1-0. Cortana/Bing was also 1-0. (And so was the Las Vegas point spread).

For the 2016 season, just predicting winners, Zoltar finished 183-82 (69% accuracy). Cortana/Bing finished 170-95 (64% accuracy). There were two tie games in the season, which I didn’t include.

Well, that’s a wrap for the 2016 season. I hope I can find some time to work on Zoltar – there are many improvements that I’d like to explore.

Note: Zoltar sometimes predicts a 0-point margin of victory. In those situations, to pick a winner so I can compare against Cortana/Bing, in the first four weeks of the season, Zoltar picks the home team to win. After week 4, Zoltar uses historical data for the current season (which usually ends up in a prediction that the home team will win).

zoltarweek6predictions

Posted in Machine Learning

Compiling and Running a C Program in Bash on Windows 10

Perhaps the best known deep neural network tool is TensorFlow. I use Windows machines the vast majority of the time, but TensorFlow is designed for Linux systems. So, instead of trying to get TensorFlow to work directly on Windows (which is possible, but very tricky), I thought I’d try to run TensorFlow inside a Bash shell running on Windows. You can think of Bash on Windows as a little Linux environment (Ubuntu Linux to be more specific).

I knew that I was going to run into many problems, so I figured I’d experiment a bit first by getting a C language program to compile and run in Bash on Windows. The first step was to install Bash on Windows. There are several good guides you can find on the Internet but briefly: Control Panel | Programs | Windows Features | “Windows Subsystem for Linux (Beta)” | OK. After a reboot, Run | Bash -> Do you want to install? -> Y. Then after the install completes, you must create a Linux User (I used “jamccaff”) and a password for the user.

Next I needed a C compiler. In a Bash shell I entered the command:

$ sudo apt-get install gcc

The ‘$’ is the Bash prompt. The sudo means “superuser do” which is sort of a Linux “Run as administrator”. The apt-get is “Advanced Packaging Tool” and “gcc” is the GNU C/C++ compiler. The Linux subsystem stores files in hidden directory C:\Users\(user)\AppData\lxss so I’m assuming that gcc went there.

11_install_gcc

To create my test program, I installed the Notepad++ editor. I probably could have used the built-in Notepad but because of the differences between Windows newlines and Linux newlines I figured Notepad++ might be a safer choice. Installing Notepad++ is quick and easy. I selected the 32-bit version, again because I figured it’d be a safer choice.

Next I launched Notepad++ and typed in a minimal C program and named it aloha.c and then saved it on my Windows filesystem at C:\Data\Junk\C_Programs — the Bash shell can access the Windows filesystem so I didn’t have to save my file in some exotic location.

Next, in the Bash shell, I navigated to the directory containing the aloha.c file by using Linux cd commands. In the Linux system this directory is /mnt/c/Data/Junk/C_Programs. I compiled and ran my C program by entering the commands:

$ gcc -o aloha.o aloha.c
$ ./aloha.o
Aloha World!

31_compiling_a_c_program_bash_on_windows

And it worked. OK, now, when I find some free time, I’ll continue the process of trying to compile and run TensorFlow in Bash on Windows.

01_turnonbashchecked

Posted in Machine Learning

The Worst UI I’ve Seen in Quite Some Time

I’m usually not too critical of user interface designs because I usually just don’t care. But today I saw a UI on a Web page that was pretty hilarious. (Click on the image below to enlarge).

bad_ui

Check out the instructions for the “Order Name – REQUIRED” text box:

maximum 50 characters. DO NOT USE “|” in the order name.

What the heck? First of all, looking at the warning text, I cannot tell what the forbidden character is. Is it lower case letter “l” or is it the digit character “1” or is it the pipe character (above the backslash on the keyboard) “|” or is it lower case “i” that’s just hard to read?

And why on earth is a particular character forbidden?

As a special bonus, I like the text in the Step 1 options: ADD FROM NETWORK DO NOT USE. Uh, OK, I won’t.

Wow. This is a seriously awful UI — but so bad it provided me with a bit of geek amusement.

Posted in Miscellaneous

Data Clustering using R

I wrote an article titled “Data Clustering using R” in the February 2017 issue of Visual Studio Magazine. See https://visualstudiomagazine.com/articles/2017/02/01/data-clustering-using-r.aspx.

dataclusteringusingrtitlebanner

Data clustering is the process of programmatically grouping data so that similar items belong together. For example, suppose you have the height (in inches) and weight (in lbs) of eight people:

(1) 65,220
(2) 73,160
(3) 59,110
(4) 61,120
(5) 75,150
(6) 68,230
(7) 62,130
(8) 66,210

Even with a ridiculously small dataset, it’s not easy to see if there’s any kind of pattern here. In my article I explain how to cluster data using the k-means algorithm and the R language. The result of clustering the data into three groups is:

73  160  
75  150  
=====
65  220  
68  230  
66  210  
=====
59  110  
61  120  
62  130

A clear pattern has emerged. There are tall-height, medium-weight people; medium-height, heavy-weight people; and short-height, light-weight people.

clusteringusingr_demoscreenshot

In my article, I explain how to use the R language built-in kmeans() function which is applicable in most scenarios where you want to cluster data, and also how to write a custom cluster function using raw R. This approach is useful when you need some sort of custom behavior.

Posted in Machine Learning, R Language

The Sign Test using C#

I wrote an article titled “The Sign Test using C#” in the February 2017 issue of Microsoft’s MSDN Magazine. See https://msdn.microsoft.com/en-us/magazine/mt793273.

signtestdemorun

The Sign Test is a statistical test designed to infer if there was some sort of an effect based on before-and-after data. The classic example is a weight loss drug. Suppose you weigh 8 people, and their weights (in kg) are:

70, 80, 75, 85, 70, 75, 50, 60

Then you give the 8 people some sort of weight loss drug and weigh them again a few weeks later and the new weights are:

65, 78, 72, 87, 68, 74, 48, 63

Of the 8 people, 6 lost weight but 2 people gained weight. The idea of the Sign Test is to assume there’s no effect, then you calculate how likely the observed data is. For this example, if there was truly no effect, then the probability of seeing 6 (or more) weight losses is only 0.1445 therefore you can conclude there appears to be an effect of some sort.

binomialdistributionwithequations

My article shows how to do all these calculations in a C# program. Of course there are tools, such as R, that can perform a Sign Test, but such tools can be difficult or impossible to integrate into a system written using a different language. And such tools are a black box so you wouldn’t understand what’s going on.

The Sign Test is designed for simple problems, but the test is surprisingly subtle and somewhat difficult to understand.

Posted in Machine Learning