I remember when the C# language yield-return statement was released as part of C# 2.0 along with Visual Studio 2005. Early documentation for the yield-return mechanism made the statement sound exotic and mysterious.   Recent discussions with some of my developer colleagues suggested that several years later now the yield-return statement isn’t used very often. We suspect this lack of use is not so much because developers don’t know how to use the yield-return mechanism, but rather mostly because developers aren’t quite sure exactly when to use yield-return. In this article I’ll describe three common programming scenarios where you might want to consider using yield-return:    1. Generating a sequence of strings where each string in the sequence depends on the value of the previous string. 2. Processing a text file sequentially where different kinds of lines are treated differently. 3. Sequentially filtering or modifying a large List collection of objects.

You can read the rest of the article online at http://visualstudiomagazine.com/articles/2012/02/01/demystifying-the-c-yield-return-mechanism.aspx

[-1.0 to -0.9) -> 0
[-0.9 to -0.8) -> 1
[-0.8 to -0.7) -> 2
. . .
[-0.1 to 0.0) -> 9
[ 0.0 to 0.1) -> 10
[ 0.1 to 0.2) -> 11
. . .
[ 0.9 to 1.0] -> 19

Notice that all but the last interval are open intervals, and the last is closed to capture the final 1.0 value. If an input is 0.245 then the output should be 12. I could do a sequential search but I wanted to use a binary search because I was going to call the function millions of times. My first attempt was this:

static int LatIndexOf(double latitude) // small experiment
{
// input is a fake latitude in [-1.0 +1.0]
// divide 0.1 degree intervals: [-1.0 -0.9) = 0, etc.

if (latitude == -1.0) return 0;
if (latitude == 1.0) return 19;

int lo = 0; //
int hi = 19;
int mid = (lo + hi) / 2;

bool found = false;
while (found == false)
{
double left = -1.0 + (0.1 * mid); // left part of interval
double right = left + 0.1; // right part of interval
if (latitude >= left && latitude < right)
return mid;
else if (latitude < left)
{
hi = mid – 1; mid = (lo + hi) / 2;
}
else
{
lo = mid + 1; mid = (lo + hi) / 2;
}
}
throw new Exception("LatIndexOf no value");
}

Because I was going to process hundreds of millions of real lat-lon data, I took the time to run pretty thorough tests on my function. As first written I ran into very nasty floating point errors that gave incorrect results for certain inputs. The hacky-fix was to round all floating point values:

static int LatIndexOf(double latitude)
{
latitude = Math.Round(latitude, 8); // round to 8 places
. . .

double left = -1.0 + (0.1 * mid);
left = Math.Round(left, 8);
. . .
double right = left + 0.1;
right = Math.Round(right, 8);
. . .

Moral: watch out for equality comparisons on floating point values.

———–
Red Small True
Red Large False
———–
Blue Medium True
Green Medium True
Green Medium False
———–

Step 1 – Calculate the probability of each cluster.

P(k = 0) = 2/5 = 0.40
P(k = 1) = 3/5 = 0.60

Step 2 – Calculate the unconditional expectation = sum of squared probabilities of all attribute values across all clusters.

Red (2/5)^2 = 0.16
Blue (1/5)^2 = 0.04
Green (2/5)^2 = 0.16
Yellow (0/5)^2 = 0.00
—
Small (1/5)^2 = 0.04
Medium (3/5)^2 = 0.36
Large (1/5)^2 = 0.04
—
False (2/5)^2 = 0.16
True (3/5)^2 = 0.36
—-
1.32

Step 3 – Calculate conditional expectations for each cluster.

A. k = 0

Red (2/2)^2 = 1.00
Blue (0/2)^2 = 0.00
Green (0/2)^2 = 0.00
Yellow (0/2)^2 = 0.00
—
Small (1/2)^2 = 0.25
Medium (0/2)^2 = 0.00
Large (1/2)^2 = 0.25
—
False (1/2)^2 = 0.25
True (1/2)^2 = 0.25
—-
2.00
B. k = 1

Red (0/3)^2 = 0.00
Blue (1/3)^2 = 0.11
Green (2/3)^2 = 0.44
Yellow (0/3)^2 = 0.00
—
Small (0/3)^2 = 0.00
Medium (3/3)^2 = 1.00
Large (0/3)^2 = 0.00
—
False (1/3)^2 = 0.11
True (2/3)^2 = 0.44
—-
2.11

Step 4 – Put it all together.

CU = (0.40 * (2.00 – 1.32)) + (0.60 * (2.11 – 1.32)) / 2
= 0.3733

The equation for the last step is too tricky for me to type out. If you search the Internet for Category Utility, you’ll find the equation in a Wikipedia entry. Coding up a routine to compute category utility is surprisingly tricky. See image below for a demo example.

Launch an instance of Visual Studio being sure to be using “Run as administrator”. I’m using VS 2010 but most recent versions of VS work the same to create a COM component. Click File | New | Project. Select the C++ | ATL project template, name it MathCOMLib (the name of the project will become the name of the resulting DLL), specify any convenient Location, clear the “Create directory for solution” and “Add to source control” check boxes, and click OK.

In the Solution Explorer window, right click on the bolded MathCOMLib project (be careful to get the project, not the solution entry), and select Add | Class from the context menu. Select the ATL Simple Object item and click Add.

In the ATL Simple Object Wizard, enter MathClass into the Short Name field. In VS 2008 and earlier, all the remaining fields (including ProgID) will be filled in automatically. But, immensely annoyingly, with VS 2010 you need to manually type in the ProgID field: MathCOMLib.MathClass (the name of the Project + “.” + the name of the Class). The ProgID field is used if you want to allow your COM object to be callable from a scripting language like JavaScript. Click Next. Leave all the File Type Handler Options blank, and click Next. Leave the Options default selections alone, and click Finish.

Now in the Solution Explorer window, select the Class View tab. You should see an interface entry named IMathClass. Right click on that interface (not the class as you might expect) and select Add | Add Method. In the Add Method Wizard, enter Sum for the Method Name. Select the “in” Parameter Attribute, select LONG for the Parameter Type, enter x for the Parameter Name, and then click the Add button. Repeat for y (“in”, “LONG”, “y”, Add). Now, select the *LONG Parameter Type which enables the retval Parameter Attribute, and check retval, and enter pRes (which stands for pointer to result) as the Parameter Name, and click Add. Click Next.

Leave the IDL Attribute alone, and click Finish. Now go back to the main Solution Explorer window. You should see a MathClass.cpp entry and then double click on it. You will be transferred to auto-generated code that looks like:

STDMETHODIMP CMathClass::Sum(LONG x, LONG y, LONG* pRes)

{

// TODO: Add your implementation code here

return S_OK;

}

Replace the comment line with:

*pRes = x + y;

Notice that return values are passed using a pointer because the regular return is reserved for an HRESULT. Now, on a 32-bit machine you can just Build the library COM object DLL by clicking Build | Build Solution. But on a 64-bit machine you need to set the Build Configuration to 64-bit by Build | Configuration Manager -> Active Solution Platform -> x64. The Build process also registers the COM object on your current machine so you don’t have to do it by using the regsvr32.exe or regsvr64.exe utility. At this point you can use Windows Explorer and verify that you just created a file named MathCOMLib.dll at wherever you told VS to save it, in the .\MathCOMLib\Debug subdirectory (along with a bazillion other files).

Now you can call your Sum method to test it in several ways. The easiest is from a C# console application. After creating the application you would first Add Reference to the MathCOMLib.dll file. Then calling code would look like this:

MathLib.MathClass mc = new MathLib.MathClass();

int x = 2;

int y = 3;

int sum = mc.Sum(x, y);

Console.WriteLine(sum);

From JavaScript you could call the method like this:

var mc = new ActiveXObject(“MathCOMLib.MathClass”);

var x = 2;

var y = 3;

var sum = mc.Sum(x, y);

WScript.Echo(sum);

And of course you could call the Sum method by using C++ if you’re a real glutton for punishment.

Now do Ctrl-W and resize the image size in percentage without “Maintain aspect ratio” to Vertical = 1% (leave the Horizontal at 100%). Now expand by Ctrl-W, Resize, Vertical = 500%. Now shrink by Ctrl-W, Resize, Vertical = 1%.

At this point expand the image by repeating Ctrl-W, Resize, Vertical = 500% until the image is close to its original size (either 3 or 4 times). Finish with a final resize (without maintaining aspect ratio). Very cool; see result below.

5.0 7.0 2.0 3.0 red

9.0 8.0 4.0 5.0 blue

2.0 2.0 4.0 7.0 red

6.0 5.0 2.0 8.0 green

8.0 7.0 8.0 7.0 blue

. . .

There are four numeric inputs and one categorical value (red, green, blue) to classify. In this situation it’s recommended that you use 1-of-N encoding for the categorical data like so:

5.0 7.0 2.0 3.0  1  0  0

9.0 8.0 4.0 5.0  0  0  1

2.0 2.0 4.0 7.0  1  0  0

6.0 5.0 2.0 8.0  0  1  0

8.0 7.0 8.0 7.0  0  0  1

. . .

There’s a long explanation behind this but just assume this encoding is a good thing. By the way, if there are only two categorical values being predicted you’d use 1-of-(N-1) encoding, and if you have categorical input you use 1-of-(N-1) encoding on it. Now with this output encoding you want the neural network output layer to have three neurons. In order to compare an output with an input like (0 1 0) instead of using a normal sigmoid or step function, you want the output to be three values between 0.0 and 1.0 that sum to 1.0. Why? Suppose an output is (1.0 1.0 1.0) — it’s not entirely clear how to evaluate that against a categorical value of (1 0 0). To get output that sums to 1.0 you can use the softmax function, where you allow arbitrary output values but then compute the sum of their Math.Exp() values and then divide each output by that sum. For example if some output from the neural net is (2.0 -3.0 0.0) their Math.Exp values are (7.39 0.05 1.00) which sum to 8.44 and dividing each by that sum gives a softmax activation output of (0.87 0.01 0.12). By the way, this computation is tricky and you have to guard against numeric overflow.

You can think of softmax outputs as probabilities. But now comparing a softmax output with a training output becomes somewhat of a problem if you use a standard sum of squared deviations (SSD) approach. For example suppose the softmax output is (0.87 0.01 0.12) as above and the training value is (1 0 0). The SSD would be computed as (1 – 0.87)^2 + (0 – 0.05)^2 + (0 – 1.00)^2. You’d compute all these values and sum them to get the total SSD error. Again there is a huge story here but the net conclusion is that it’s recommended to use cross entropy error instead of sum of squared deviations error. Cross entropy error, in principle, looks like this:

(1 * Ln(0.87)) + (0 * Ln(0.01)) + (0 * Ln(1.00))

= -0.06 + 0 + 0 = -0.06

You would add all the cross entropies for each training vector up and then multiply by -1. There’s a lot to understand here but in essence we only look at the 1 term in the training data (multiplying by the 0s has no effect on the sum). If the output is close to 1.0 the Ln is close to zero. To summarize, when using a neural network to classify categorical data, encode the output categorical data using 1-of-N encoding (except if there are only two categories), use the softmax activation function to generate output, and use cross entropy to measure error.

f(x,y) = (-x * sin(sqrt(abs(x)))) + (-y * sin(sqrt(abs(y))))

The function has a minimum value of 2 * -418.9829 = -837.9658 when both x = 420.9687 and y = 420.9687. The minimum value is hard to find because it is relatively far from the second best solution.

Here are commands to visualize Schwefel’s function using SciLab, a free open source program similar to MatLab:

x1 = linspace(-500,500,100);

x2 = linspace(-500,500,100);

[X1,X2] = meshgrid(x1,x2);

Z = (-1 * X1. * sin(sqrt(abs(X1)))) + (-1 * X2. * sin(sqrt(abs(X2))));

f = scf();

f.color_map = jetcolormap(64);

plot3d1(x1,x2,Z’);

SciLab code is close to MatLab code and both are a bit obscure. I generate two vectors of 100 values using linspace, create a cross product of those values using meshgrid (SciLab is case sensitive so X1 is different from x1), defined a function Z (where the dot operator means do a value by value multiplication rather than matrix multiplication), and plot the graph (where the ‘ operator is the transpose operator). The result is shown below. The minimum value is in the far left corner.