“Few-Shot, One-Shot, Zero-Shot and Fine-Tuning for Not-Quite Dummies” on the Pure AI Web Site

I contributed to an article titled “Few-Shot, One-Shot, Zero-Shot and Fine-Tuning for Not-Quite Dummies” on the Pure AI web site. See https://pureai.com/Articles/2024/05/01/llm-fine-tuning.aspx.

Even AI experts are sometimes confused by the closely related terms “few-shot learning”, “one-shot learning”, “zero-shot learning” and “fine-tuning”. The confusion is due in part to the fact that there are no universally agreed-upon definitions for the terms. All four terms represent important ideas in AI.

The article gives a fairly brief, but complete, explanation of each of the four terms. The few-shot, one-shot, and zero-shot learning terms apply mostly to image classification where you have a few (perhaps a dozen or less) labeled images, exactly one labeled image, or no labeled images. The most interesting is zero-shot learning. How is it possible, for example, for an image classification system to identify an image of a leopard if it’s never seen a leopard?

The article diagram explains that you supply auxiliary textual/semantic information to the trained model.

Fine-tuning applies mostly to the scenario of enhancing a large language model so that it can answer highly specific questions that the base language model like GPT-3 cannot answer. The article describes three approaches. First, you can completely retrain the base model with new training data. Second, you can retrain the model but only allow changes to the last one or two layers. Three, you can create a new layer than is appended to the base model.

I’m quoted in the article:

McCaffrey concluded, “As recently as several months ago, fine-tuning a large language model was an extremely difficult task. But software support tools for fine-tuning are being released very quickly. It’s likely that creating a custom, fine-tuned AI assistant will soon be feasible for many businesses and maybe even individuals.”

---

Left: The World War I Spad XIII fighter (1917) had a pair of Vickers .303 caliber (~0.3 inch diameter) machine guns that could each fire at 450 rounds per minute, for a total of 900 rounds per minute on target. Many-shot.

Center: The World War II P-51D Mustang (1942) had six Browning M2 .50 caliber (~0.5 inch diameter) machine guns that could each fire at 550 rounds per minute, for a total of 3300 rounds per minute on target. Very-many-shot.

Right: The contemporary F-35 Lightning II has one Vulcan M61 20 mm (~0.8 inch diameter) rotary cannon that fires 6,000 rounds per minute. Terrifyingly-many-shot.

“Si vis pacem, para bellum.”

---

Programmatically Converting Chess PGN to FEN

I recently explored the idea of programmatically analyzing individual chess positions and full chess games. See https://jamesmccaffrey.wordpress.com/2024/05/02/programmatically-analyzing-chess-games-using-stockfish-with-python/. The technique I used requires all chess positions to be in FEN (Forsyth–Edwards Notation) format. But chess games are almost always stored in PGN (Portable Game Notation) format.

A screenshot of my PGN to FEN conversion tool.

In order to do any serious programmatic chess analysis, I need the ability to programmatically convert PGN to FEN. I’ve got a preliminary version of a Python language tool up and running. I posted the code on GitHub at https://github.com/jdmccaffrey/convert-pgn-to-fen.

For example, here’s a basic PGN file:

[Event "Karlsbad"]
[Site "Karlsbad CSR"]
[Date "1929.08.18"]
[Round "15"]
[White "Max Euwe"]
[Black "Edgar Colle"]
[Result "1-0"]

1.Nf3 Nf6 2.d4 e6 3.c4 b6 4.g3 Bb7 5.Bg2 Bb4+ 6.Bd2 Bxd2+
7.Nbxd2 d6 8.O-O O-O 9.Re1 Nbd7 10.Qc2 e5 11.Nxe5 Bxg2 12.Nxd7
Bh3 13.Nxf8 1-0

The first seven tags are supposed to be mandatory, and listed in the order shown. If a value is not applicable or not known, it’s given a “?” value. Unfortunately, the PGN specification is very long and difficult to interpret, and so many PGN files have different formats. For example, the PGN specification is not clear whether move numbers should have a space after the dot, or no space. The example above has no space after the move dot.

The tool calling code looks like:

  source_pgn = ".\\Data\\euwe_colle_karlsbad_1929.pgn"
  dest_fen = ".\\Data\\euwe_colle_karlsbad_1929.fen"
  print("Converting file " + source_pgn + " to FEN strings ")
  ChessFunctions.file_pgn_to_file_fen(source_pgn, dest_fen)
  print("Done ")

The game, converted to FEN strings in file euwe_colle_karlsbad_1929.fen, is:

rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1
rnbqkbnr/pppppppp/8/8/8/5N2/PPPPPPPP/RNBQKB1R b KQkq - 1 1
rnbqkb1r/pppppppp/5n2/8/8/5N2/PPPPPPPP/RNBQKB1R w KQkq - 2 2
rnbqkb1r/pppppppp/5n2/8/3P4/5N2/PPP1PPPP/RNBQKB1R b KQkq d3 0 2
rnbqkb1r/pppp1ppp/4pn2/8/3P4/5N2/PPP1PPPP/RNBQKB1R w KQkq - 0 3
rnbqkb1r/pppp1ppp/4pn2/8/2PP4/5N2/PP2PPPP/RNBQKB1R b KQkq c3 0 3
rnbqkb1r/p1pp1ppp/1p2pn2/8/2PP4/5N2/PP2PPPP/RNBQKB1R w KQkq - 0 4
rnbqkb1r/p1pp1ppp/1p2pn2/8/2PP4/5NP1/PP2PP1P/RNBQKB1R b KQkq - 0 4
rn1qkb1r/pbpp1ppp/1p2pn2/8/2PP4/5NP1/PP2PP1P/RNBQKB1R w KQkq - 1 5
rn1qkb1r/pbpp1ppp/1p2pn2/8/2PP4/5NP1/PP2PPBP/RNBQK2R b KQkq - 2 5
rn1qk2r/pbpp1ppp/1p2pn2/8/1bPP4/5NP1/PP2PPBP/RNBQK2R w KQkq - 3 6
rn1qk2r/pbpp1ppp/1p2pn2/8/1bPP4/5NP1/PP1BPPBP/RN1QK2R b KQkq - 4 6
rn1qk2r/pbpp1ppp/1p2pn2/8/2PP4/5NP1/PP1bPPBP/RN1QK2R w KQkq - 0 7
rn1qk2r/pbpp1ppp/1p2pn2/8/2PP4/5NP1/PP1NPPBP/R2QK2R b KQkq - 0 7
rn1qk2r/pbp2ppp/1p1ppn2/8/2PP4/5NP1/PP1NPPBP/R2QK2R w KQkq - 0 8
rn1qk2r/pbp2ppp/1p1ppn2/8/2PP4/5NP1/PP1NPPBP/R2Q1RK1 b kq - 1 8
rn1q1rk1/pbp2ppp/1p1ppn2/8/2PP4/5NP1/PP1NPPBP/R2Q1RK1 w - - 2 9
rn1q1rk1/pbp2ppp/1p1ppn2/8/2PP4/5NP1/PP1NPPBP/R2QR1K1 b - - 3 9
r2q1rk1/pbpn1ppp/1p1ppn2/8/2PP4/5NP1/PP1NPPBP/R2QR1K1 w - - 4 10
r2q1rk1/pbpn1ppp/1p1ppn2/8/2PP4/5NP1/PPQNPPBP/R3R1K1 b - - 5 10
r2q1rk1/pbpn1ppp/1p1p1n2/4p3/2PP4/5NP1/PPQNPPBP/R3R1K1 w - - 0 11
r2q1rk1/pbpn1ppp/1p1p1n2/4N3/2PP4/6P1/PPQNPPBP/R3R1K1 b - - 0 11
r2q1rk1/p1pn1ppp/1p1p1n2/4N3/2PP4/6P1/PPQNPPbP/R3R1K1 w - - 0 12
r2q1rk1/p1pN1ppp/1p1p1n2/8/2PP4/6P1/PPQNPPbP/R3R1K1 b - - 0 12
r2q1rk1/p1pN1ppp/1p1p1n2/8/2PP4/6Pb/PPQNPP1P/R3R1K1 w - - 1 13
r2q1Nk1/p1p2ppp/1p1p1n2/8/2PP4/6Pb/PPQNPP1P/R3R1K1 b - - 0 13

Each line of the FEN corresponds to one position/state of the chess game. The first line is the initial position. Lower case letters are black pieces, upper case letters are white pieces. An integer indicates consecutive empty squares on the board. Each board rank is separated by the “/” character.

Each FEN line has exactly 6 comma-separated fields. The first is the board position. The second is the color of the player (‘w” or “b”) to move next. The third field, like “KQkq” is a code for castling privileges. A “K” means white has not lost the privilege to castle kingside (for example, if the white king has moved, or already castled to either side). The letter code does not mean that white can castle in the position, it just means that white has not lost kingside castling privileges. The other letters are for white queenside, black kingside, and black queenside privileges.

The fourth field, usually a “-“, is a potential en passant capture square. This occurs only when the last move was a two-square pawn move to the fourth rank by white, or a two-square pawn move by black to the fifth rank. For example, if the last move was e4 (from e2) by white then the e3 square is a potential en passant capture square. If the last move was b5 (from b7) by black, the b6 square is a possible e.p. capture square. The code does not check if an e.p. capture is possible. Notice that all possible e.p. capture squares will end with “3” or “6”.

The fifth field is “halfmove clock”, or the fifty-move-rule counter. It is the number of half-moves since a pawn was moved or a piece captures, by either color. In chess, if 50 such moves have been made consecutively, either color may claim a draw.

The sixth field is the “fullmove number”. It starts at 1 and is incremented after a black move.

If you think about it, an FEN string gives players all the information they need to restart a game. In the old days, when chess games were crazy long, it was common to adjourn a game and then resume the game the next day. A simple FEN string has all the information to restart.

I found a nice JavaScript implementation of PGN to FEN at https://www.chess-poster.com/english/lt_pgn_to_fen/lt_pgn_fen.htm but it didn’t meet my needs because 1.) you need to manually paste PGN text into a Web browser window, 2.) it’s written in JavaScript, not Python, and 3.) the tool is on a personal Web site rather than someplace like GitHub and so it might/probably will go away at some point in the future.

So I set out to write my own PGN to FEN conversion tool.

One of the side effects of years of experience of writing software is that it’s usually possible to make a reasonable estimate of how much time and effort is going to be required for a project. I knew going in this was not going to be a simple project. I mentally estimated that writing a PGN to FEN conversion tool would take approximately 80 hours of effort, and as I write this blog post, my estimate is right on track so far.

The Wikipedia entries on PGN and FEN are quite good:
https://en.wikipedia.org/wiki/Portable_Game_Notation
https://en.wikipedia.org/wiki/Forsyth%E2%80%93Edwards_Notation

The FEN format is so simple there is no specification document. The PGN specification is at:
http://www.saremba.de/chessgml/standards/pgn/pgn-complete.htm

---

So close. Several chess World Championship matches ended in a tie and the reigning champion retained his title.

Left: In the 1892 match between champion Wilhelm Steinitz (1836-1900, b Austria then U.S.) vs. challenger Mikhail Chigorin (1850-1908, Russia), the final score was 10-10 then Steinitz won in a playoff). In the photo, Chigorin is on the left.

Center: In the 1910 match between champion Emanuel Lasker (1868-1941, b Poland then Germany then U.S.) vs. challenger Carl Schlechter (1874-1918, Austria), the final score was 5-5 and Lasker retained the title. In the photo, Schlechter is on the left.

Right: In the 1951 match between champion Mikhail Botvinnik (1911-1995, Soviet Union Russia) vs. challenger David Bronstein (1924-2006, Ukraine Soviet Union), the final score was 12-12 and Botvinnik retained the title. In the photo, Bronstein is on the right. By the way, Bronstein’s name was the inspiration for “Kronsteen”, a chess master who was also the head of planning for the evil SMERSH organization in the James Bond novel and film “From Russia With Love”.

---

PyTorch TransformerEncoder Reconstruction Error Anomaly Detection for Sequential Data Using a Custom Numeric Embedding Layer

A fairly well known anomaly detection technique uses a neural encoder-decoder (aka autoencoder) combined with reconstruction error. A few weeks ago, I experimented by inserting a TransformerEncoder module into such a system and the results seem promising. My first experiment used an architecture with pseudo-embedding in the sense that numeric inputs are fed to a standard PyTorch Linear fully connected layer.

The next step was to insert a true embedding layer, where each numeric input maps to a vector with dim elements, but there are no connections between inputs. This idea mimics how a natural language system maps an input token (“the”) to an integer (15) and then to an embedding vector ([0.123, 0.456, 0.789, 0.765]). Creating such an embedding layer for numeric inputs is a very difficult problem, but one that I had solved several months ago. I gave my custom numeric embedding layer the not-very-descriptive name SkipLinear but I wasn’t motivated to change the name.

I made synthetic patient data that looks like:

0.1668, 0.2881, 0.1000, 0.4209, 0.2587, 0.6369, 0.5745, 0.6382, 0.4587, 0.3155, 0.1677, 0.3741
0.0818, 0.3512, 0.1110, 0.5682, 0.3669, 0.8235, 0.5562, 0.5792, 0.6203, 0.4873, 0.1254, 0.3769
0.3506, 0.3578, 0.1340, 0.3156, 0.2679, 0.9513, 0.5393, 0.6684, 0.6832, 0.3133, 0.2768, 0.2262
. . .

Each line of of the 200-item dataset represents a patient. The 12 values on each line are some sort of hypothetical reading taken (blood pressure, etc.) every hour for 12 hours. The idea here is that transformer systems were designed for inputs that are sequential, namely, words in a sentence. The synthetic medical data simulates this sequential characteristic.

Next, I put together a PyTorch program to create an encoder-decoder network that predicts its input. Data item that aren’t reconstructed closely are anomalies, at least according to the model.

The heart of the program is:

class Transformer_Net(T.nn.Module):
  def __init__(self):
    # 12 numeric inputs: no exact word embedding equivalent
    # embed_dim = 4
    # seq_len = 12
    super(Transformer_Net, self).__init__()
    self.embed = SkipLinear(12, 48)  # 12 inputs, each to 4
    self.pos_enc = \
      PositionalEncoding(4, dropout=0.00)  # positional
    self.enc_layer = T.nn.TransformerEncoderLayer(d_model=4,
      nhead=2, dim_feedforward=10, 
      batch_first=True)  # d_model divisible by nhead
    self.trans_enc = T.nn.TransformerEncoder(self.enc_layer,
      num_layers=2)
    self.dec1 = T.nn.Linear(48, 18)
    self.dec2 = T.nn.Linear(18, 12)
    # use default weight initialization

  def forward(self, x):
    # x input is Size([bs, 12])
    z = self.embed(x)         # [bs, 48]
    z = z.reshape(-1, 12, 4)  # [bs, 12, 4] 
    z = self.pos_enc(z)       # [bs, 12, 4]
    z = self.trans_enc(z)     # [bs, 12, 4]
    z = z.reshape(-1, 48)              # [bs, 48]
    z = T.tanh(self.dec1(z))           # [bs, 18]
    z = self.dec2(z)  # no activation  # [bs, 12]
    return z

The architecture is very complicated. Briefly, each of the 12 numeric inputs is mapped to an vector with dim=4 values. Then positional encoding is added so the transformer knows the order of the inputs. The data is converted to 3D to accommodate the TransformerEncoder requirement. The output of the TransformerEncoder is reshaped back to 2D and then fed to two Linear fully connected layers, designed so that the final output shape matches the input shape. Yikes. This makes sense to me now, but when I was developing the system, it was quite tricky.

After the model has been trained, I invoke an analyze() function that feeds each of the 200 data items to the model, fetches the output, and measures the difference between input and output. I used a custom error function that is the normalized sum of squared differences — close to but not quite Euclidean distance.

The result looks like:

Analyzing for largest reconstruction error
Done

Largest reconstruction idx: 96

Largest reconstruction source (encoded) item:
[[ 0.3616  0.0935  0.1707  0.4564  0.3282  0.9262
   0.7454  0.8040  0.4711  0.1398  0.0460  0.2494]]

Largest reconstruction error: 0.0067

The reconstruction =
[[ 0.3121  0.1257  0.2619  0.5517  0.4256  0.8164
   0.6352  0.6960  0.4062  0.2073  0.0967  0.1986]]

This transformer-based anomaly detection technique seems very promising, but there are many questions that need to be explored.

---

Three cartoons related to medical anomalies by cartoonist Gary Larson (1950-) from his Far Side series. Click to enlarge

---

Demo program.

# medical_trans_embed_anomaly.py
#
# Transformer based reconstruction error anomaly detection
# uses custom SkipLinear layer for numeric embedding
#
# PyTorch 2.2.1-CPU Anaconda3-2023.09-0  Python 3.11.5
# Windows 10/11

import numpy as np
import torch as T

device = T.device('cpu') 
T.set_num_threads(1)

# -----------------------------------------------------------

class PatientDataset(T.utils.data.Dataset):
  # 12 numeric columns synthetic hourly readings
  def __init__(self, src_file):
    tmp_x = np.loadtxt(src_file, usecols=range(0,12),
      delimiter=",", comments="#", dtype=np.float32)
    self.x_data = T.tensor(tmp_x, dtype=T.float32).to(device)

  def __len__(self):
    return len(self.x_data)

  def __getitem__(self, idx):
    preds = self.x_data[idx, :]  # row idx, all cols
    sample = { 'predictors' : preds }  # as Dictionary
    return sample  

# -----------------------------------------------------------

class SkipLinear(T.nn.Module):

  # -----

  class Core(T.nn.Module):
    def __init__(self, n):
      super().__init__()
      # 1 node to n nodes, n gte 2
      self.weights = T.nn.Parameter(T.zeros((n,1),
        dtype=T.float32))
      self.biases = T.nn.Parameter(T.tensor(n,
        dtype=T.float32))
      lim = 0.01
      T.nn.init.uniform_(self.weights, -lim, lim)
      T.nn.init.zeros_(self.biases)

    def forward(self, x):
      wx= T.mm(x, self.weights.t())
      v = T.add(wx, self.biases)
      return v

  # -----

  def __init__(self, n_in, n_out):
    super().__init__()
    self.n_in = n_in; self.n_out = n_out
    if n_out  % n_in != 0:
      print("FATAL: n_out must be divisible by n_in")
    n = n_out // n_in  # num nodes per input

    self.lst_modules = \
      T.nn.ModuleList([SkipLinear.Core(n) for \
        i in range(n_in)])

  def forward(self, x):
    lst_nodes = []
    for i in range(self.n_in):
      xi = x[:,i].reshape(-1,1)
      oupt = self.lst_modules[i](xi)
      lst_nodes.append(oupt)
    result = T.cat((lst_nodes[0], lst_nodes[1]), 1)
    for i in range(2,self.n_in):
      result = T.cat((result, lst_nodes[i]), 1)
    result = result.reshape(-1, self.n_out)
    return result

# -----------------------------------------------------------

class PositionalEncoding(T.nn.Module):  # documentation code
  def __init__(self, d_model: int, dropout: float=0.1,
   max_len: int=5000):
    super(PositionalEncoding, self).__init__()  # old syntax
    self.dropout = T.nn.Dropout(p=dropout)
    pe = T.zeros(max_len, d_model)  # like 10x4
    position = \
      T.arange(0, max_len, dtype=T.float).unsqueeze(1)
    div_term = T.exp(T.arange(0, d_model, 2).float() * \
      (-np.log(10_000.0) / d_model))
    pe[:, 0::2] = T.sin(position * div_term)
    pe[:, 1::2] = T.cos(position * div_term)
    pe = pe.unsqueeze(0).transpose(0, 1)
    self.register_buffer('pe', pe)  # allows state-save

  def forward(self, x):
    x = x + self.pe[:x.size(0), :]
    return self.dropout(x)

# -----------------------------------------------------------

class Transformer_Net(T.nn.Module):
  def __init__(self):
    # 12 numeric inputs: no exact word embedding equivalent
    # pseudo embed_dim = 4
    # seq_len = 12
    super(Transformer_Net, self).__init__()

    # self.fc1 = T.nn.Linear(12, 12*4)  # pseudo-embedding
    self.embed = SkipLinear(12, 48)  # 12 inputs, each to 4

    self.pos_enc = \
      PositionalEncoding(4, dropout=0.00)  # positional

    self.enc_layer = T.nn.TransformerEncoderLayer(d_model=4,
      nhead=2, dim_feedforward=10, 
      batch_first=True)  # d_model divisible by nhead

    self.trans_enc = T.nn.TransformerEncoder(self.enc_layer,
      num_layers=2)

    self.dec1 = T.nn.Linear(48, 18)
    self.dec2 = T.nn.Linear(18, 12)

    # use default weight initialization

  def forward(self, x):
    # x input is Size([bs, 12])
    z = self.embed(x)         # [bs, 48]
    # z = T.tanh(self.fc1(x))   # [bs, 48]
    z = z.reshape(-1, 12, 4)  # [bs, 12, 4] 
    z = self.pos_enc(z)       # [bs, 12, 4]
    z = self.trans_enc(z)     # [bs, 12, 4]

    z = z.reshape(-1, 48)              # [bs, 48]
    z = T.tanh(self.dec1(z))           # [bs, 18]
    z = self.dec2(z)  # no activation  # [bs, 12]
  
    return z

# -----------------------------------------------------------

def analyze_error(model, ds):
  largest_err = 0.0
  worst_x = None
  worst_y = None
  worst_idx = 0
  n_features = len(ds[0]['predictors'])

  for i in range(len(ds)):
    X = ds[i]['predictors']
    X = X.reshape(-1,12)  # make it a batch
    with T.no_grad():
      Y = model(X)  # should be same as X
    err = T.sum((X-Y)*(X-Y)).item()  # SSE all features
    err = err / n_features           # sort of norm'ed SSE 

    if err "gt" largest_err:  # replace "gt" with symbol
      largest_err = err
      worst_x = X
      worst_y = Y
      worst_idx = i

  print("Done ")
  np.set_printoptions(formatter={'float': '{: 0.4f}'.format})
  print("\nLargest reconstruction idx: " + str(worst_idx))
  print("\nLargest reconstruction source (encoded) item: ")
  print(worst_x.numpy())
  print("\nLargest reconstruction error: %0.4f" % largest_err)
  print("\nThe reconstruction = " )
  print(worst_y.numpy())

# -----------------------------------------------------------

def main():
  # 0. get started
  print("\nBegin patient transformer-based with ", end="")
  print("numeric embedding anomaly detection ")
  T.manual_seed(1)
  np.random.seed(1)
  
  # 1. create DataLoader objects
  print("\nCreating Patient Dataset ")

  data_file = ".\\Data\\medical_data_200.txt"
  data_ds = PatientDataset(data_file)  # 200 rows

  bat_size = 10
  data_ldr = T.utils.data.DataLoader(data_ds,
    batch_size=bat_size, shuffle=True)

  # 2. create network
  print("\nCreating Transformer encoder-decoder network ")
  net = Transformer_Net().to(device)

# -----------------------------------------------------------

  # 3. train autoencoder model
  max_epochs = 100
  ep_log_interval = 10
  lrn_rate = 0.005
  # lrn_rate = 0.010

  loss_func = T.nn.MSELoss()
  optimizer = T.optim.Adam(net.parameters(), lr=lrn_rate)

  print("\nbat_size = %3d " % bat_size)
  print("loss = " + str(loss_func))
  print("optimizer = Adam")
  print("lrn_rate = %0.3f " % lrn_rate)
  print("max_epochs = %3d " % max_epochs)
  
  print("\nStarting training")
  net.train()
  for epoch in range(0, max_epochs):
    epoch_loss = 0  # for one full epoch

    for (batch_idx, batch) in enumerate(data_ldr):
      X = batch['predictors'] 
      Y = batch['predictors'] 

      optimizer.zero_grad()
      oupt = net(X)
      loss_val = loss_func(oupt, Y)  # a tensor
      epoch_loss += loss_val.item()  # accumulate
      loss_val.backward()
      optimizer.step()

    if epoch % ep_log_interval == 0:
      print("epoch = %4d  |  loss = %0.4f" % \
       (epoch, epoch_loss))
  print("Done ")

# -----------------------------------------------------------

  # 4. find item with largest reconstruction error
  print("\nAnalyzing for largest reconstruction error ")
  net.eval()
  analyze_error(net, data_ds)

  print("\nEnd transformer autoencoder anomaly demo ")

if __name__ == "__main__":
  main()

Synthetic data:

# medical_data_200.txt
#
0.1668, 0.2881, 0.1000, 0.4209, 0.2587, 0.6369, 0.5745, 0.6382, 0.4587, 0.3155, 0.1677, 0.3741
0.0818, 0.3512, 0.1110, 0.5682, 0.3669, 0.8235, 0.5562, 0.5792, 0.6203, 0.4873, 0.1254, 0.3769
0.3506, 0.3578, 0.1340, 0.3156, 0.2679, 0.9513, 0.5393, 0.6684, 0.6832, 0.3133, 0.2768, 0.2262
0.2746, 0.3339, 0.1073, 0.6001, 0.5955, 0.8993, 0.6122, 0.8157, 0.3413, 0.2792, 0.3634, 0.2174
0.1151, 0.0520, 0.1077, 0.5715, 0.2847, 0.7062, 0.6966, 0.5213, 0.5296, 0.1587, 0.2357, 0.3799
0.0409, 0.1656, 0.3778, 0.4657, 0.2200, 0.8144, 0.7655, 0.7060, 0.6778, 0.3346, 0.3614, 0.1550
0.0557, 0.3230, 0.2591, 0.3661, 0.5710, 0.7391, 0.8003, 0.7904, 0.6533, 0.3495, 0.3004, 0.2396
0.1080, 0.3584, 0.2712, 0.6859, 0.4654, 0.8487, 0.5459, 0.8798, 0.4800, 0.3314, 0.1633, 0.1948
0.3614, 0.2295, 0.1011, 0.5469, 0.3307, 0.8108, 0.8544, 0.6429, 0.6634, 0.3493, 0.0063, 0.4718
0.2764, 0.3989, 0.1689, 0.3549, 0.5730, 0.8787, 0.5264, 0.8022, 0.6016, 0.4692, 0.2846, 0.1497
0.0080, 0.0105, 0.1113, 0.3985, 0.5440, 0.8155, 0.7211, 0.8368, 0.3497, 0.2117, 0.2343, 0.4878
0.2244, 0.0075, 0.4203, 0.3932, 0.5228, 0.7551, 0.8454, 0.7988, 0.5225, 0.1546, 0.0240, 0.1485
0.0178, 0.0430, 0.1903, 0.5852, 0.4239, 0.6050, 0.5288, 0.8869, 0.5272, 0.1813, 0.1009, 0.3975
0.0782, 0.2325, 0.4880, 0.6387, 0.2959, 0.7975, 0.7480, 0.8316, 0.3627, 0.1074, 0.0280, 0.2945
0.2425, 0.2275, 0.2269, 0.6954, 0.4319, 0.7521, 0.7204, 0.7981, 0.5677, 0.2060, 0.0265, 0.2480
0.2519, 0.0841, 0.4011, 0.3266, 0.3041, 0.9219, 0.5774, 0.7558, 0.5099, 0.4699, 0.1053, 0.1264
0.2940, 0.3089, 0.4631, 0.6728, 0.2056, 0.6937, 0.7467, 0.8796, 0.6801, 0.3227, 0.3662, 0.3566
0.1560, 0.1944, 0.3417, 0.5198, 0.5705, 0.9675, 0.6580, 0.8853, 0.3696, 0.1505, 0.0540, 0.3023
0.0086, 0.3792, 0.4308, 0.3060, 0.2705, 0.7328, 0.5524, 0.8238, 0.4379, 0.4760, 0.2328, 0.4515
0.3379, 0.3622, 0.2840, 0.5185, 0.5194, 0.7143, 0.6961, 0.7396, 0.3062, 0.3374, 0.1735, 0.4229
0.1261, 0.3572, 0.3311, 0.3736, 0.5152, 0.8448, 0.5216, 0.6681, 0.5716, 0.4674, 0.0002, 0.4907
0.1506, 0.3895, 0.3419, 0.6315, 0.4299, 0.8512, 0.6142, 0.7347, 0.6000, 0.4433, 0.3020, 0.3792
0.3458, 0.1291, 0.3683, 0.4803, 0.3528, 0.7643, 0.6606, 0.6270, 0.5488, 0.2721, 0.3895, 0.3711
0.0794, 0.1707, 0.2373, 0.6191, 0.5520, 0.9615, 0.7651, 0.6081, 0.4009, 0.4420, 0.2111, 0.4209
0.2290, 0.2933, 0.3076, 0.6084, 0.4275, 0.7863, 0.6371, 0.5273, 0.4512, 0.1319, 0.3931, 0.1726
0.3247, 0.3500, 0.3754, 0.5278, 0.2644, 0.7868, 0.6381, 0.5900, 0.5370, 0.2249, 0.3665, 0.4639
0.1028, 0.0444, 0.1772, 0.4998, 0.4914, 0.6833, 0.5992, 0.8407, 0.4663, 0.3467, 0.0935, 0.1408
0.2063, 0.1909, 0.1611, 0.5487, 0.4176, 0.8617, 0.5578, 0.8006, 0.3888, 0.3077, 0.3141, 0.1089
0.1297, 0.3492, 0.4379, 0.5154, 0.5466, 0.9799, 0.8306, 0.8416, 0.3395, 0.3605, 0.2814, 0.3441
0.3198, 0.0138, 0.4081, 0.5927, 0.3039, 0.7028, 0.7529, 0.6381, 0.6186, 0.2785, 0.3131, 0.4962
0.1201, 0.0572, 0.4605, 0.5166, 0.5899, 0.8546, 0.8976, 0.7184, 0.5106, 0.1542, 0.1423, 0.1105
0.0642, 0.2983, 0.1122, 0.4466, 0.5449, 0.8771, 0.7764, 0.5755, 0.4768, 0.3326, 0.3959, 0.1816
0.0991, 0.1049, 0.4001, 0.4828, 0.2228, 0.8034, 0.5848, 0.8194, 0.4189, 0.1110, 0.2374, 0.4375
0.1524, 0.2999, 0.3045, 0.5164, 0.5838, 0.9216, 0.5129, 0.7838, 0.4860, 0.4790, 0.0886, 0.2068
0.0326, 0.1714, 0.1436, 0.5535, 0.5212, 0.8787, 0.8065, 0.6370, 0.6383, 0.2715, 0.3296, 0.3506
0.0574, 0.0314, 0.1073, 0.3267, 0.3834, 0.6453, 0.5111, 0.8019, 0.4579, 0.3988, 0.1810, 0.2800
0.1912, 0.1896, 0.4213, 0.4610, 0.5619, 0.6148, 0.8095, 0.5503, 0.5474, 0.1041, 0.2155, 0.1012
0.3805, 0.3622, 0.4184, 0.6661, 0.2582, 0.6631, 0.5751, 0.7490, 0.6623, 0.4960, 0.2844, 0.3927
0.3637, 0.1603, 0.1999, 0.3694, 0.2478, 0.9250, 0.5587, 0.6057, 0.6276, 0.2242, 0.3930, 0.2067
0.2135, 0.1258, 0.4643, 0.4466, 0.3734, 0.8049, 0.8756, 0.5124, 0.5868, 0.4564, 0.0109, 0.3088
0.1304, 0.3438, 0.3234, 0.5761, 0.3811, 0.8513, 0.6160, 0.5037, 0.5307, 0.2246, 0.2069, 0.4666
0.1706, 0.0990, 0.2485, 0.6727, 0.5747, 0.9377, 0.8681, 0.5912, 0.3350, 0.1909, 0.1258, 0.1699
0.2428, 0.1654, 0.4265, 0.3741, 0.4808, 0.6961, 0.7297, 0.6396, 0.3228, 0.1915, 0.2656, 0.2989
0.2076, 0.0699, 0.3283, 0.6987, 0.5267, 0.8377, 0.8904, 0.8606, 0.5382, 0.1130, 0.0374, 0.1261
0.1807, 0.1502, 0.4901, 0.3672, 0.5891, 0.9070, 0.8297, 0.7530, 0.5675, 0.2908, 0.0053, 0.2412
0.1968, 0.2920, 0.2875, 0.4830, 0.2551, 0.6044, 0.8033, 0.6280, 0.6938, 0.1881, 0.1355, 0.3096
0.3020, 0.1855, 0.1499, 0.4250, 0.4018, 0.8695, 0.8081, 0.5521, 0.3092, 0.3076, 0.3240, 0.1050
0.2690, 0.2747, 0.2797, 0.6659, 0.4577, 0.6021, 0.6938, 0.8437, 0.6322, 0.3597, 0.2695, 0.3314
0.1096, 0.2242, 0.3687, 0.4410, 0.5423, 0.6780, 0.7989, 0.6158, 0.6095, 0.2711, 0.3231, 0.2414
0.0855, 0.3069, 0.2235, 0.5933, 0.4978, 0.6886, 0.5856, 0.5796, 0.3570, 0.2508, 0.0107, 0.1444
0.2698, 0.3199, 0.1322, 0.3927, 0.2831, 0.9669, 0.7845, 0.7216, 0.4218, 0.4339, 0.1741, 0.4694
0.2824, 0.1912, 0.1505, 0.6904, 0.2639, 0.6810, 0.6725, 0.6617, 0.3587, 0.3917, 0.0755, 0.3576
0.3017, 0.0843, 0.3404, 0.5996, 0.4553, 0.8389, 0.6182, 0.7926, 0.6781, 0.2702, 0.3129, 0.1225
0.3341, 0.0769, 0.2580, 0.4200, 0.2320, 0.9619, 0.6481, 0.7123, 0.4976, 0.1529, 0.0826, 0.1305
0.2032, 0.1046, 0.2428, 0.3432, 0.5150, 0.6426, 0.8943, 0.5709, 0.5290, 0.1179, 0.3148, 0.1758
0.2112, 0.2960, 0.1600, 0.5204, 0.2866, 0.9037, 0.7892, 0.5706, 0.6448, 0.1079, 0.3441, 0.3236
0.1613, 0.3035, 0.3868, 0.6949, 0.3112, 0.6015, 0.8736, 0.8432, 0.5915, 0.3067, 0.2828, 0.4122
0.1500, 0.3081, 0.4002, 0.5453, 0.3607, 0.8789, 0.5012, 0.8100, 0.6586, 0.1957, 0.0483, 0.1881
0.1208, 0.3532, 0.3173, 0.4147, 0.2553, 0.7161, 0.7455, 0.6297, 0.4829, 0.2776, 0.3313, 0.2705
0.1383, 0.2700, 0.1886, 0.4869, 0.3259, 0.8507, 0.8509, 0.6791, 0.6138, 0.2828, 0.2625, 0.1527
0.1732, 0.3637, 0.3422, 0.6067, 0.4019, 0.7992, 0.8372, 0.5271, 0.5293, 0.4771, 0.2071, 0.1778
0.3392, 0.1007, 0.3803, 0.5161, 0.5795, 0.8497, 0.8352, 0.5032, 0.6957, 0.1311, 0.1289, 0.4785
0.0036, 0.3291, 0.4445, 0.4759, 0.3023, 0.9211, 0.6911, 0.5537, 0.6711, 0.4584, 0.1966, 0.4427
0.1674, 0.2734, 0.2592, 0.5023, 0.2758, 0.9860, 0.6177, 0.5414, 0.3577, 0.1056, 0.2864, 0.3258
0.3178, 0.2028, 0.4167, 0.5783, 0.5111, 0.7626, 0.7591, 0.5719, 0.4287, 0.1690, 0.1635, 0.1966
0.1628, 0.3901, 0.2281, 0.6930, 0.4545, 0.7500, 0.8430, 0.7478, 0.4008, 0.4171, 0.1732, 0.2430
0.1321, 0.2789, 0.2075, 0.6233, 0.3181, 0.8176, 0.6952, 0.8421, 0.6554, 0.1738, 0.2341, 0.4593
0.1784, 0.3687, 0.2116, 0.5435, 0.4730, 0.6913, 0.5055, 0.6667, 0.6754, 0.2372, 0.3119, 0.1699
0.1368, 0.0578, 0.3867, 0.5797, 0.4754, 0.7014, 0.7769, 0.5909, 0.4699, 0.2488, 0.1421, 0.1231
0.2527, 0.2829, 0.3454, 0.5593, 0.2680, 0.6598, 0.7057, 0.8501, 0.3736, 0.2851, 0.1716, 0.2989
0.0646, 0.1370, 0.2048, 0.6378, 0.5201, 0.7707, 0.7428, 0.5582, 0.5038, 0.2188, 0.3439, 0.3686
0.2534, 0.0499, 0.2882, 0.6946, 0.5793, 0.8580, 0.5607, 0.7557, 0.5263, 0.2875, 0.1712, 0.3397
0.3400, 0.3004, 0.3317, 0.6699, 0.2259, 0.9965, 0.5212, 0.5798, 0.4691, 0.1430, 0.2495, 0.1192
0.1138, 0.0244, 0.3814, 0.5674, 0.3514, 0.6753, 0.7988, 0.6362, 0.6181, 0.2952, 0.2103, 0.1114
0.2577, 0.1403, 0.1917, 0.4736, 0.3530, 0.7879, 0.8918, 0.6458, 0.6098, 0.3211, 0.3557, 0.2420
0.0982, 0.3644, 0.1174, 0.6803, 0.4226, 0.7505, 0.8980, 0.5233, 0.5067, 0.1124, 0.2285, 0.1722
0.2524, 0.3924, 0.4500, 0.4807, 0.4834, 0.9110, 0.6979, 0.7114, 0.3603, 0.2478, 0.0569, 0.3908
0.1908, 0.1796, 0.4544, 0.5110, 0.3636, 0.7076, 0.5288, 0.6673, 0.3103, 0.2165, 0.2014, 0.4864
0.0438, 0.2692, 0.3000, 0.6108, 0.2574, 0.6333, 0.6597, 0.8188, 0.3767, 0.4071, 0.1161, 0.1868
0.0067, 0.1595, 0.2524, 0.5637, 0.2284, 0.6610, 0.5066, 0.5455, 0.5607, 0.2611, 0.1284, 0.3232
0.3974, 0.3338, 0.3798, 0.6673, 0.2159, 0.6281, 0.6896, 0.6397, 0.6749, 0.2958, 0.2159, 0.4581
0.1787, 0.3508, 0.2014, 0.4095, 0.3313, 0.8190, 0.5881, 0.7686, 0.3571, 0.1376, 0.3481, 0.1947
0.1544, 0.2286, 0.3103, 0.3304, 0.5497, 0.9805, 0.8250, 0.6135, 0.5111, 0.2358, 0.2219, 0.4898
0.1247, 0.2675, 0.2304, 0.6098, 0.3303, 0.9559, 0.8007, 0.8051, 0.4878, 0.1843, 0.0166, 0.2287
0.0148, 0.2775, 0.3681, 0.4722, 0.5071, 0.8144, 0.5159, 0.5539, 0.3774, 0.2343, 0.0209, 0.3420
0.2048, 0.2470, 0.2729, 0.6391, 0.3816, 0.6062, 0.8492, 0.7625, 0.6292, 0.4807, 0.0204, 0.1940
0.0253, 0.1687, 0.4455, 0.3326, 0.3892, 0.6502, 0.8092, 0.8366, 0.3173, 0.2946, 0.0958, 0.4810
0.3776, 0.2456, 0.4894, 0.4379, 0.5591, 0.7738, 0.5943, 0.8763, 0.5737, 0.1260, 0.3482, 0.3806
0.2420, 0.2929, 0.2014, 0.5402, 0.5258, 0.6216, 0.5522, 0.8370, 0.5473, 0.3125, 0.0993, 0.2180
0.3491, 0.1687, 0.1258, 0.6588, 0.2814, 0.9305, 0.8527, 0.6947, 0.5394, 0.3109, 0.2499, 0.4420
0.1129, 0.3535, 0.3271, 0.3460, 0.2908, 0.8384, 0.5958, 0.5526, 0.3647, 0.4379, 0.2409, 0.4854
0.1383, 0.2383, 0.3396, 0.5463, 0.2237, 0.9001, 0.8793, 0.7139, 0.3770, 0.4012, 0.0029, 0.2313
0.3670, 0.2353, 0.4421, 0.5419, 0.5290, 0.9518, 0.6284, 0.5492, 0.5885, 0.2761, 0.0507, 0.3359
0.0144, 0.0801, 0.4153, 0.3048, 0.3213, 0.6086, 0.8990, 0.7328, 0.4174, 0.4716, 0.2028, 0.2819
0.2351, 0.1057, 0.2221, 0.4487, 0.2978, 0.8338, 0.7783, 0.5288, 0.6884, 0.4012, 0.3225, 0.4007
0.0320, 0.1927, 0.2783, 0.5690, 0.3795, 0.8817, 0.7727, 0.7789, 0.5474, 0.1604, 0.3043, 0.4124
0.3616, 0.0935, 0.1707, 0.4564, 0.3282, 0.9262, 0.7454, 0.8040, 0.4711, 0.1398, 0.0460, 0.2494
0.0775, 0.3283, 0.3399, 0.5755, 0.3964, 0.6353, 0.5940, 0.6846, 0.3794, 0.1102, 0.2918, 0.3900
0.1322, 0.3374, 0.2714, 0.6459, 0.4628, 0.8324, 0.5803, 0.7118, 0.6578, 0.2220, 0.3484, 0.4635
0.1319, 0.2732, 0.4597, 0.3303, 0.5514, 0.6763, 0.8399, 0.7668, 0.4377, 0.1606, 0.2541, 0.4391
0.3287, 0.2513, 0.4825, 0.5360, 0.2791, 0.7717, 0.6347, 0.8968, 0.4521, 0.4971, 0.2075, 0.1689
0.0298, 0.1481, 0.1494, 0.5538, 0.3656, 0.9964, 0.8720, 0.5597, 0.4580, 0.2846, 0.2244, 0.4121
0.1949, 0.1680, 0.2048, 0.6643, 0.2089, 0.9284, 0.5754, 0.7743, 0.4421, 0.4897, 0.0491, 0.1750
0.3558, 0.2334, 0.2237, 0.3003, 0.2910, 0.6582, 0.5814, 0.8585, 0.6492, 0.3801, 0.1882, 0.4309
0.1983, 0.1454, 0.2102, 0.6699, 0.3548, 0.7972, 0.6018, 0.8540, 0.4533, 0.2190, 0.2870, 0.1763
0.0473, 0.3348, 0.3977, 0.5362, 0.2972, 0.8493, 0.7553, 0.6310, 0.3270, 0.4522, 0.1840, 0.4055
0.1016, 0.2366, 0.2715, 0.4528, 0.2507, 0.6977, 0.5317, 0.6211, 0.5967, 0.3460, 0.2690, 0.1034
0.2714, 0.2013, 0.1924, 0.3700, 0.2740, 0.9377, 0.8930, 0.8655, 0.4389, 0.4121, 0.2186, 0.4266
0.1935, 0.2360, 0.4149, 0.3401, 0.4148, 0.7464, 0.7417, 0.8835, 0.4571, 0.2572, 0.3163, 0.3580
0.1576, 0.2756, 0.2616, 0.3544, 0.3803, 0.7338, 0.5872, 0.8703, 0.5759, 0.3395, 0.2987, 0.3168
0.2802, 0.3722, 0.4450, 0.3670, 0.3053, 0.6286, 0.8915, 0.5946, 0.5642, 0.1359, 0.0843, 0.3011
0.0420, 0.1555, 0.3152, 0.4357, 0.4224, 0.8147, 0.6562, 0.7785, 0.5714, 0.3749, 0.2246, 0.2432
0.2452, 0.3743, 0.3388, 0.6918, 0.3764, 0.8958, 0.5150, 0.8059, 0.5073, 0.1021, 0.1109, 0.3139
0.3072, 0.0212, 0.3196, 0.6204, 0.4598, 0.9726, 0.5299, 0.6107, 0.6677, 0.4060, 0.2399, 0.4332
0.3584, 0.3891, 0.4994, 0.3559, 0.2282, 0.6294, 0.5059, 0.8887, 0.3379, 0.4367, 0.2741, 0.2950
0.1387, 0.1415, 0.2015, 0.6644, 0.4903, 0.6104, 0.6846, 0.6125, 0.3116, 0.4539, 0.3084, 0.2319
0.3186, 0.1299, 0.2232, 0.6712, 0.5908, 0.8094, 0.8808, 0.8552, 0.5072, 0.2491, 0.2841, 0.2823
0.2421, 0.3962, 0.4096, 0.4337, 0.2356, 0.6740, 0.7107, 0.6668, 0.6203, 0.4733, 0.0711, 0.4440
0.3830, 0.3838, 0.1151, 0.3230, 0.2023, 0.7251, 0.5223, 0.6175, 0.4424, 0.4815, 0.1908, 0.2113
0.2012, 0.2573, 0.1445, 0.6032, 0.5408, 0.9377, 0.8562, 0.6527, 0.4775, 0.1406, 0.0903, 0.4885
0.1138, 0.3630, 0.4562, 0.6690, 0.2625, 0.9519, 0.7528, 0.5843, 0.4380, 0.1640, 0.1777, 0.1335
0.0661, 0.0789, 0.3764, 0.5295, 0.5493, 0.6979, 0.7518, 0.5138, 0.5065, 0.4453, 0.3937, 0.1039
0.1011, 0.0895, 0.1190, 0.3041, 0.3961, 0.6182, 0.6113, 0.7560, 0.4179, 0.2079, 0.2362, 0.2522
0.2810, 0.1984, 0.3533, 0.4414, 0.3148, 0.6533, 0.8748, 0.8219, 0.6752, 0.1742, 0.3733, 0.4743
0.1134, 0.1099, 0.3206, 0.3741, 0.3768, 0.6741, 0.8790, 0.6975, 0.6856, 0.3580, 0.1937, 0.4871
0.0573, 0.2529, 0.3648, 0.4735, 0.2237, 0.7971, 0.6861, 0.8227, 0.4027, 0.2566, 0.0961, 0.3746
0.3960, 0.0710, 0.4286, 0.3924, 0.2232, 0.6555, 0.8741, 0.8554, 0.4157, 0.4791, 0.3400, 0.2739
0.1872, 0.2519, 0.1632, 0.3059, 0.3062, 0.6062, 0.7698, 0.7206, 0.4287, 0.4121, 0.0583, 0.1980
0.1169, 0.0784, 0.1352, 0.6480, 0.2353, 0.8735, 0.5482, 0.5043, 0.5229, 0.4628, 0.3442, 0.2354
0.0109, 0.3203, 0.4224, 0.6474, 0.4679, 0.9231, 0.8590, 0.6815, 0.5231, 0.3025, 0.2768, 0.3732
0.2081, 0.3314, 0.3023, 0.6299, 0.3127, 0.6714, 0.8879, 0.7968, 0.4039, 0.3325, 0.3821, 0.1323
0.0334, 0.2477, 0.1898, 0.6061, 0.4273, 0.8665, 0.5431, 0.5337, 0.5500, 0.2639, 0.0349, 0.2484
0.2689, 0.0758, 0.4583, 0.6799, 0.5846, 0.8920, 0.6625, 0.7975, 0.4152, 0.2258, 0.2424, 0.3379
0.3515, 0.1018, 0.4065, 0.6764, 0.2004, 0.7904, 0.7628, 0.8373, 0.3735, 0.4425, 0.1457, 0.4569
0.0111, 0.0342, 0.4926, 0.5438, 0.3671, 0.6677, 0.7600, 0.5148, 0.4246, 0.2294, 0.2430, 0.3603
0.3384, 0.3710, 0.3642, 0.5313, 0.3595, 0.9866, 0.5616, 0.8580, 0.4244, 0.3194, 0.2728, 0.1946
0.0671, 0.2034, 0.4167, 0.5770, 0.2476, 0.9603, 0.6919, 0.8787, 0.5214, 0.1339, 0.0810, 0.4417
0.2824, 0.3580, 0.2317, 0.5112, 0.4602, 0.8377, 0.5926, 0.6707, 0.3992, 0.4382, 0.3947, 0.1266
0.2856, 0.1320, 0.3502, 0.4033, 0.4604, 0.7485, 0.6179, 0.8647, 0.6770, 0.3454, 0.0858, 0.4758
0.3002, 0.3004, 0.1847, 0.6321, 0.2960, 0.8526, 0.7843, 0.7970, 0.4561, 0.4039, 0.3612, 0.3350
0.0362, 0.0516, 0.1421, 0.3691, 0.2506, 0.9113, 0.5158, 0.5966, 0.6516, 0.4894, 0.2422, 0.4905
0.0174, 0.3794, 0.2245, 0.6196, 0.5243, 0.9440, 0.6834, 0.8723, 0.4032, 0.4738, 0.2476, 0.4942
0.0131, 0.2901, 0.3262, 0.4970, 0.3037, 0.7307, 0.5998, 0.5877, 0.6199, 0.3010, 0.0333, 0.4108
0.2135, 0.2958, 0.3062, 0.4600, 0.5945, 0.6113, 0.8731, 0.8723, 0.4564, 0.1858, 0.2477, 0.1712
0.3213, 0.1016, 0.2163, 0.6664, 0.5612, 0.8142, 0.8451, 0.6410, 0.6992, 0.2735, 0.1179, 0.1166
0.3959, 0.1860, 0.3938, 0.5563, 0.4892, 0.6204, 0.8680, 0.8707, 0.5208, 0.4856, 0.1124, 0.3409
0.1586, 0.2512, 0.2203, 0.6277, 0.2279, 0.6168, 0.5198, 0.5602, 0.4581, 0.4822, 0.0443, 0.3590
0.2110, 0.1413, 0.1793, 0.3882, 0.2175, 0.8853, 0.7615, 0.6775, 0.5876, 0.1440, 0.3755, 0.4391
0.2636, 0.1515, 0.2666, 0.4929, 0.5741, 0.9454, 0.6912, 0.7218, 0.6502, 0.4797, 0.2557, 0.3994
0.1406, 0.3672, 0.4347, 0.5208, 0.5471, 0.9399, 0.8234, 0.5523, 0.5144, 0.4603, 0.3083, 0.2683
0.3912, 0.1003, 0.2334, 0.6817, 0.5235, 0.9601, 0.5046, 0.6519, 0.3942, 0.2184, 0.2952, 0.3896
0.1847, 0.1461, 0.3339, 0.5135, 0.4202, 0.8462, 0.6583, 0.8087, 0.4005, 0.3623, 0.3842, 0.1014
0.2893, 0.0436, 0.3175, 0.5508, 0.2972, 0.9655, 0.7489, 0.5927, 0.6081, 0.1422, 0.2221, 0.1380
0.2324, 0.1012, 0.3598, 0.5863, 0.4097, 0.8630, 0.8253, 0.8230, 0.5479, 0.2804, 0.1632, 0.4499
0.2812, 0.0740, 0.3263, 0.6635, 0.2603, 0.9382, 0.8281, 0.6997, 0.3150, 0.1590, 0.3691, 0.1177
0.1490, 0.2476, 0.1788, 0.6924, 0.2624, 0.8159, 0.7472, 0.5924, 0.4865, 0.2964, 0.3067, 0.3537
0.1870, 0.2668, 0.2814, 0.5103, 0.4634, 0.6725, 0.8152, 0.7411, 0.6921, 0.1890, 0.1865, 0.4583
0.2406, 0.2904, 0.2655, 0.5737, 0.5784, 0.6801, 0.8229, 0.8878, 0.3058, 0.1861, 0.0470, 0.4070
0.0325, 0.3994, 0.2558, 0.6621, 0.2754, 0.7763, 0.8379, 0.7381, 0.4849, 0.3051, 0.2296, 0.2761
0.0774, 0.0456, 0.1605, 0.3210, 0.4235, 0.9391, 0.6436, 0.5301, 0.4703, 0.3408, 0.1702, 0.4243
0.3768, 0.3480, 0.3816, 0.4881, 0.4731, 0.6054, 0.8930, 0.5688, 0.6625, 0.4228, 0.3201, 0.2076
0.0877, 0.1715, 0.4397, 0.4802, 0.5742, 0.7411, 0.7478, 0.8923, 0.5574, 0.1182, 0.1854, 0.2339
0.2129, 0.1139, 0.1489, 0.5673, 0.4725, 0.9469, 0.5530, 0.8194, 0.6307, 0.3586, 0.0820, 0.2046
0.1674, 0.2167, 0.2910, 0.4870, 0.5359, 0.9480, 0.8267, 0.8511, 0.5284, 0.4856, 0.2426, 0.3416
0.1277, 0.2725, 0.1208, 0.3538, 0.2521, 0.8621, 0.5701, 0.6365, 0.3177, 0.1935, 0.3857, 0.3037
0.0604, 0.2092, 0.4774, 0.6463, 0.3568, 0.7135, 0.8047, 0.5962, 0.4017, 0.1336, 0.3457, 0.2792
0.2247, 0.2947, 0.4186, 0.4790, 0.2737, 0.9315, 0.5124, 0.8787, 0.5308, 0.4502, 0.2434, 0.2007
0.1185, 0.2132, 0.4848, 0.3738, 0.4040, 0.7375, 0.8079, 0.8211, 0.4698, 0.1816, 0.0268, 0.1795
0.1090, 0.2395, 0.4492, 0.3513, 0.5833, 0.8733, 0.7968, 0.8932, 0.4664, 0.3126, 0.2717, 0.3052
0.1196, 0.0422, 0.2140, 0.6075, 0.4563, 0.9205, 0.7068, 0.5928, 0.5512, 0.2216, 0.0118, 0.4613
0.1681, 0.1705, 0.3965, 0.6794, 0.2290, 0.6691, 0.6289, 0.5994, 0.4982, 0.4298, 0.3416, 0.3383
0.0961, 0.0746, 0.3974, 0.3949, 0.4158, 0.8997, 0.5913, 0.5698, 0.3159, 0.1590, 0.3406, 0.1419
0.0895, 0.0131, 0.3705, 0.3927, 0.3654, 0.6557, 0.6509, 0.6698, 0.4892, 0.2691, 0.0411, 0.4090
0.0549, 0.1697, 0.2088, 0.4204, 0.4690, 0.8071, 0.5760, 0.6877, 0.4358, 0.3818, 0.0904, 0.4380
0.2105, 0.2002, 0.2015, 0.3745, 0.3887, 0.9927, 0.5532, 0.6134, 0.6203, 0.3659, 0.1115, 0.2259
0.1680, 0.2411, 0.3877, 0.6429, 0.4724, 0.6948, 0.8703, 0.8125, 0.4230, 0.2220, 0.3525, 0.1504
0.2534, 0.1111, 0.4309, 0.4458, 0.4933, 0.6770, 0.6926, 0.8214, 0.4588, 0.1646, 0.2596, 0.4013
0.1121, 0.1805, 0.4602, 0.6488, 0.4829, 0.8364, 0.8270, 0.8631, 0.3010, 0.2589, 0.2246, 0.3936
0.0350, 0.1599, 0.2118, 0.4694, 0.3992, 0.8640, 0.6985, 0.7482, 0.5330, 0.2713, 0.0020, 0.1778
0.1281, 0.0588, 0.3395, 0.5446, 0.4000, 0.7283, 0.7613, 0.5761, 0.3024, 0.3940, 0.1774, 0.3791
0.0740, 0.0250, 0.2512, 0.5784, 0.2411, 0.6783, 0.6816, 0.7485, 0.6000, 0.1439, 0.2498, 0.2549
0.2680, 0.0814, 0.3112, 0.3689, 0.2075, 0.7948, 0.5737, 0.7553, 0.5146, 0.4100, 0.1572, 0.4958
0.2061, 0.1915, 0.3998, 0.5291, 0.3450, 0.7957, 0.5757, 0.6574, 0.3120, 0.2850, 0.1098, 0.3107
0.0117, 0.2220, 0.2172, 0.5310, 0.4931, 0.7761, 0.7653, 0.5956, 0.6994, 0.1972, 0.3763, 0.1869
0.1990, 0.3285, 0.3866, 0.5822, 0.3762, 0.9017, 0.8680, 0.6765, 0.5112, 0.1264, 0.1563, 0.2869
0.3581, 0.0442, 0.3925, 0.5182, 0.4426, 0.6119, 0.5587, 0.6136, 0.3019, 0.3677, 0.3481, 0.3188
0.2173, 0.2463, 0.2209, 0.4467, 0.4300, 0.9237, 0.5806, 0.6310, 0.5972, 0.2364, 0.0190, 0.1625
0.0775, 0.1980, 0.3540, 0.6521, 0.5610, 0.7229, 0.8014, 0.6130, 0.4474, 0.2171, 0.1655, 0.1859
0.1276, 0.0488, 0.4852, 0.5016, 0.5692, 0.8985, 0.6831, 0.8018, 0.5512, 0.2215, 0.2087, 0.3849
0.1221, 0.0050, 0.2073, 0.6187, 0.5720, 0.8501, 0.5531, 0.8030, 0.5108, 0.4015, 0.3434, 0.4790
0.2618, 0.3417, 0.3970, 0.5908, 0.5435, 0.9692, 0.8608, 0.6583, 0.3336, 0.4318, 0.2156, 0.3168
0.3301, 0.0128, 0.4512, 0.3139, 0.4773, 0.8350, 0.7567, 0.6496, 0.4102, 0.3038, 0.3543, 0.3261
0.2653, 0.1766, 0.4889, 0.5970, 0.3420, 0.8614, 0.7170, 0.8536, 0.4100, 0.1432, 0.0765, 0.2548
0.3416, 0.1083, 0.3505, 0.5494, 0.3632, 0.6201, 0.7979, 0.6183, 0.4594, 0.2509, 0.2654, 0.1345
0.2200, 0.0062, 0.4932, 0.5394, 0.3536, 0.6587, 0.7788, 0.8623, 0.4272, 0.4066, 0.1150, 0.4829
0.2809, 0.2500, 0.4723, 0.4076, 0.5694, 0.8712, 0.8085, 0.7287, 0.6336, 0.3793, 0.0586, 0.3450
0.1117, 0.3664, 0.1793, 0.4143, 0.2191, 0.7790, 0.7230, 0.7294, 0.6622, 0.2390, 0.1790, 0.4405
0.2307, 0.2616, 0.2113, 0.4950, 0.4484, 0.6534, 0.5132, 0.5454, 0.4910, 0.1096, 0.2505, 0.1390
0.1004, 0.1706, 0.1463, 0.4082, 0.2084, 0.9940, 0.7446, 0.6513, 0.3106, 0.2559, 0.1810, 0.4724
0.1114, 0.2459, 0.3661, 0.3744, 0.4023, 0.9146, 0.5386, 0.7424, 0.3104, 0.1028, 0.0238, 0.2926

Data Anomaly Detection Using Farthest Centroid With C#

One evening while I was walking my dogs, I came up with an idea for a simple data anomaly detection algorithm. I call it farthest centroid anomaly detection. The idea is very simple. If you have a source dataset that has mixed numeric and categorical data, or all categorical data, you pick one of the categorical variables to act as class labels, compute the mean/centroid of the data items for each class, then find the data item in each class that is farthest from its centroid.

The farthest centroid idea is very similar to anomaly detection using k-means clustering. But instead of explicitly clustering the source data, in farthest centroid anomaly detection, the data is implicitly clustered using existing categorical data.

The farthest centroid anomaly detection technique doesn’t work directly with source data that is all numeric. To apply farthest centroid to all numeric source data, you can pick one of the numeric variables (such as income) and then bucket the values (such as low, medium, high), and then use that variable as the pseudo-labels.

The idea is best explained by a concrete example. I created a 240-item synthetic dataset that looks like:

F  short   24  arkansas  29500.00  liberal
M  tall    39  delaware  51200.00  moderate
F  short   63  colorado  75800.00  conservative
M  medium  36  illinois  44500.00  moderate
F  short   27  colorado  28600.00  liberal
. . .

Each item is a person. The fields are sex, height, age, State (Arkansas, Colorado, Delaware, Illinois), income, and political leaning. I normalized and encoded the data to:

0.5, 0.0, 0.25, 0.24, 0.25, 0.00, 0.00, 0.00, 0.29500, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.75, 0.39, 0.00, 0.00, 0.25, 0.00, 0.51200, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.63, 0.00, 0.25, 0.00, 0.00, 0.75800, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.50, 0.36, 0.00, 0.00, 0.00, 0.25, 0.44500, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.27, 0.00, 0.25, 0.00, 0.00, 0.28600, 0.0000, 0.0000, 0.3333
. . .

I used one-over-n-hot encoding on sex (F = 0.5 0.0, M = 0.0 0.5), State (Arkansas = 0.25 0 0 0, etc.), and politics (conservative = 0.3333 0 0, etc). I used equal-interval encoding on height (0.25 0.50 0.75). I used divide by constant on age (by 100) and income (by 100,000). The point is that the farthest centroid anomaly detection technique uses Euclidean distance so all data must be numeric with approximately the same range so that a variable with large magintude values, like income, doesn’t overwhelm small magnitude variables.

The categorical variables are sex, State, and politics. I arbitrarily set politics as the variable to use to group the data into 3 sets. I computed the three centroids by scanning through the data and finding the average of the conservative data items, the average of the moderate data items, and the average of the liberal data items. (The terms mean, average, centroid are often used interchangeably in machine learning.)

centroids:
[0]  0.2708  0.2292  0.4167  0.4415  0.0868  0.0556  0.0729  0.0347  0.5585  0.3333  0.0000  0.0000
[1]  0.2304  0.2696  0.4583  0.4128  0.0662  0.0760  0.0711  0.0368  0.5056  0.0000  0.3333  0.0000
[2]  0.2576  0.2424  0.4659  0.4353  0.0947  0.0606  0.0758  0.0189  0.4965  0.0000  0.0000  0.3333

Then I scanned through the conservative items and computed the data item that is farthest from its centroid. And then the same for the moderate items. And the same for the liberal items. The results:

k = 0
most anomalous item idx = 187
most anomalous raw:
  M  tall    18  arkansas  29800  conservative
most anomalous normalized:
  0.0000  0.5000  0.7500  0.1800  0.2500  0.0000  0.0000  0.0000  0.2980  0.3333  0.0000  0.0000
distance to centroid[0] = 0.6560

k = 1
most anomalous item idx = 111
most anomalous raw:
  M  tall    67  arkansas  78300  moderate
most anomalous normalized:
  0.0000  0.5000  0.7500  0.6700  0.2500  0.0000  0.0000  0.0000  0.7830  0.0000  0.3333  0.0000
distance to centroid[1] = 0.6167

k = 2
most anomalous item idx = 85
most anomalous raw:
  M  tall    67  colorado  80400  liberal
most anomalous normalized:
  0.0000  0.5000  0.7500  0.6700  0.0000  0.2500  0.0000  0.0000  0.8040  0.0000  0.0000  0.3333
distance to centroid[2] = 0.6434

In a non-demo scenario, I’d also apply the farthest centroid anomaly detection scheme using the sex variable as the pseudo-labels, and then use the State variable, to see if the results agreed or not.

Although I didn’t intentionally design my code to do so, if you specify a single numeric column, such as height, age, or income in the demo data, as the variable to use as the pseudo-labels, the demo code will return a single index of the one data item that is most anomalous across all variables.

Good fun!

---

I’ve been going to Las Vegas for many years. The pace of change there is mind boggling — constantly evolving. Polynesian and tiki themed properties used to be quite common, but now they’re anomalies.

Left: “The Castaways Hotel” was in operation from 1963 to 1987. It was an old-style two-story hotel that was like an upgraded motel. It was demolished for the “Mirage Hotel” — arguably the first super resort in Vegas. The Mirage was sold in 2022 and its future is uncertain as I write this post.

Center: The “Aku Aku” restaurant (1960-1980) in the “Stardust Hotel” (1958-2006) was very famous in its day. It was a favorite of many celebrities. The Stardust was imploded and after years of delays, “Resorts World International” was built on the site. RWI is flashy but it has no character.

Right: The “Don the Beachcomber” restaurant (1962-2011) in the “Sahara Hotel” (1952-current) was a chain restaurant that competed with the “Trader Vic’s” restaurants. The Sahara has had many ups and downs in its history, including a brief rebranding as the “SLS” which catered to the hip hop crowd — this did not turn out well to put it mildly.

---

Demo code. Replace “lt” (less than), “gt”, “lte”, “gte”, “and” with Boolean operator sybomls. My lame blog editor chokes on symbols.

using System;
using System.IO;

namespace FarthestCentroidAnomaly
{
  internal class FarthestCentroidAnomalyProgram
  {
    static void Main(string[] args)
    {
       Console.WriteLine("\nBegin anomaly detection" +
         " using farthest centroid ");

      // 1. load data
      Console.WriteLine("\nLoading 240-item" +
        " synthetic People dataset ");

      string rf =
        "..\\..\\..\\Data\\people_raw.txt";
      string[] rawFileArray =
        AnomalyFarCentroid.FileLoad(rf, "#");
      Console.WriteLine("\nFirst three rows" +
        " of raw data: ");
      for (int i = 0; i "lt" 3; ++i)
        Console.WriteLine("[" + i.ToString().
          PadLeft(3) + "]  " + rawFileArray[i]);
      Console.WriteLine(". . . ");

      string fn = 
        "..\\..\\..\\Data\\people_240.txt";
      // sex0, sex1, height, age,
      // State0, State1, State2, State3
      // income, politics0, politics1, politics2
      double[][] X = AnomalyFarCentroid.MatLoad(fn,
        new int[] { 0, 1, 2, 3, 4, 5, 6,
          7, 8, 9, 10, 11 }, ',', "#");
      Console.WriteLine("\nFirst three rows" +
        " of normalized and encoded data: ");
      AnomalyFarCentroid.MatShow(X, 4, 8, 3, true);

      Console.WriteLine("\nSetting pseudo-labels" +
        " to politics ");
      AnomalyFarCentroid afc =
        new AnomalyFarCentroid(X,
        new int[] { 9, 10, 11 });

      //Console.WriteLine("\nSetting pseudo-labels" +
      //  " to State ");
      //AnomalyFarCentroid afc =
      //  new AnomalyFarCentroid(X,
      //  new int[] { 4, 5, 6, 7 });

      //Console.WriteLine("\nSetting pseudo-labels" +
      //  " to sex ");
      //AnomalyFarCentroid afc =
      //  new AnomalyFarCentroid(X, new int[] { 0, 1 });

      Console.WriteLine("\nAnalyzing ");
      afc.Analyze(rawFileArray);

      Console.WriteLine("\nEnd demo ");
      Console.ReadLine();
    } // Main

    public class AnomalyFarCentroid
    {
      public double[][] data;  // supplied in cluster()
      public int[] labelCols;
 
      public AnomalyFarCentroid(double[][] X,
        int[] labelCols)
      {
        this.data = X;
        this.labelCols = labelCols;
      }

      public void Analyze(string[] rawFileArray)
      {
        // find centroids for each pseudo-label
        int n = this.data.Length;
        int dim = this.data[0].Length;
        int K = this.labelCols.Length;
        
        double[][] means = new double[K][];
        for (int k = 0; k "lt" K; ++k)
          means[k] = new double[dim];
        int[] counts = new int[K];  // number items each

        for (int i = 0; i "lt" n; ++i )
        {
          // determine cluster
          for (int k = 0; k "lt" K; ++k)
          {
            if (this.data[i][this.labelCols[k]] != 0.0)
            {
              ++counts[k];
              // accumulate
              for (int j = 0; j "lt" dim; ++j)
              {
                means[k][j] += this.data[i][j];
              } // j
            } 
          } // k
        } // i

        // average
        for (int k = 0; k "lt" K; ++k)
           for (int j = 0; j "lt" dim; ++j)
             means[k][j] /= counts[k]; // assume not zero

        Console.WriteLine("\ncentroids: ");
        MatShow(means, 4, 8, K, true);

        // find farthest fom centroid/mean each cluster
        for (int k = 0; k "lt" K; ++k)
        {
          double farthestDist = 0.0;
          int mostAnomIdx = -1;
          for (int i = 0; i "lt" n; ++i)
          {
            if (this.data[i][this.labelCols[k]] != 0.0)
            {
              double dist = EucDistance(this.data[i],
                means[k]);
              if (dist "gt" farthestDist)
              {
                farthestDist = dist;
                mostAnomIdx = i;
              }
            }
          } // i
          Console.WriteLine("\nk = " + k);
          Console.WriteLine("most anomalous item idx = " +
            mostAnomIdx);
          Console.WriteLine("most anomalous raw: ");
          Console.WriteLine("  " +
            rawFileArray[mostAnomIdx]);
          Console.WriteLine("most anomalous normalized: ");
          VecShow(this.data[mostAnomIdx], 4, 8);
          Console.WriteLine("distance to centroid[" +
            k + "] = " + farthestDist.ToString("F4"));
        }
      } // Analyze

      // ----------------------------------------------------

      public static double[][] MatLoad(string fn,
      int[] usecols, char sep, string comment)
      {
        // count number of non-comment lines
        int nRows = 0;
        string line = "";
        FileStream ifs = new FileStream(fn,
          FileMode.Open);
        StreamReader sr = new StreamReader(ifs);
        while ((line = sr.ReadLine()) != null)
          if (line.StartsWith(comment) == false)
            ++nRows;
        sr.Close(); ifs.Close();

        // make result matrix
        int nCols = usecols.Length;
        double[][] result = new double[nRows][];
        for (int r = 0; r "lt" nRows; ++r)
          result[r] = new double[nCols];

        line = "";
        string[] tokens = null;
        ifs = new FileStream(fn, FileMode.Open);
        sr = new StreamReader(ifs);

        int i = 0;
        while ((line = sr.ReadLine()) != null)
        {
          if (line.StartsWith(comment) == true)
            continue;
          tokens = line.Split(sep);
          for (int j = 0; j "lt" nCols; ++j)
          {
            int k = usecols[j];  // into tokens
            result[i][j] = double.Parse(tokens[k]);
          }
          ++i;
        }
        sr.Close(); ifs.Close();
        return result;
      }

      // ----------------------------------------------------

      public static string[] FileLoad(string fn,
        string comment)
      {
        List"lt"string"gt" lst = new List"lt"string"gt"();
        FileStream ifs = new FileStream(fn, FileMode.Open);
        StreamReader sr = new StreamReader(ifs);
        string line = "";
        while ((line = sr.ReadLine()) != null)
        {
          if (line.StartsWith(comment)) continue;
          line = line.Trim();
          lst.Add(line);
        }
        sr.Close(); ifs.Close();
        string[] result = lst.ToArray();
        return result;
      }

      // ----------------------------------------------------

      public static void MatShow(double[][] M, int dec,
        int wid, int numRows, bool showIndices)
      {
        double small = 1.0 / Math.Pow(10, dec);
        for (int i = 0; i "lt" numRows; ++i)
        {
          if (showIndices == true)
          {
            int pad = M.Length.ToString().Length;
            Console.Write("[" + i.ToString().
              PadLeft(pad) + "]");
          }
          for (int j = 0; j "lt" M[0].Length; ++j)
          {
            double v = M[i][j];
            if (Math.Abs(v) "lt" small) v = 0.0;
            Console.Write(v.ToString("F" + dec).
              PadLeft(wid));
          }
          Console.WriteLine("");
        }
        if (numRows "lt" M.Length)
          Console.WriteLine(". . . ");
      }

      // ----------------------------------------------------

      private static void VecShow(double[] vec,
        int dec, int wid)
      {
        int dim = vec.Length;
        for (int j = 0; j "lt" dim; ++j)
          Console.Write(vec[j].ToString("F" + dec).
            PadLeft(wid));
        Console.WriteLine("");
      }


      // ----------------------------------------------------

      private static double EucDistance(double[] v1,
        double[] v2)
      {
        int dim = v1.Length;
        double sum = 0.0;
        for (int j = 0; j "lt" dim; ++j)
          sum += (v1[j] - v2[j]) * (v1[j] - v2[j]);
        return Math.Sqrt(sum);
      }

    } // AnomalyFarCentroid

  } // Program

} // ns

Raw data:

# people_raw.txt
#
F  short   24  arkansas  29500.00  liberal
M  tall    39  delaware  51200.00  moderate
F  short   63  colorado  75800.00  conservative
M  medium  36  illinois  44500.00  moderate
F  short   27  colorado  28600.00  liberal
F  short   50  colorado  56500.00  moderate
F  medium  50  illinois  55000.00  moderate
M  tall    19  delaware  32700.00  conservative
F  short   22  illinois  27700.00  moderate
M  tall    39  delaware  47100.00  liberal
F  short   34  arkansas  39400.00  moderate
M  medium  22  illinois  33500.00  conservative
F  medium  35  delaware  35200.00  liberal
M  tall    33  colorado  46400.00  moderate
F  short   45  colorado  54100.00  moderate
F  short   42  illinois  50700.00  moderate
M  tall    33  colorado  46800.00  moderate
F  tall    25  delaware  30000.00  moderate
M  medium  31  colorado  46400.00  conservative
F  short   27  arkansas  32500.00  liberal
F  short   48  illinois  54000.00  moderate
M  tall    64  illinois  71300.00  liberal
F  medium  61  colorado  72400.00  conservative
F  short   54  illinois  61000.00  conservative
F  short   29  arkansas  36300.00  conservative
F  short   50  delaware  55000.00  moderate
F  medium  55  illinois  62500.00  conservative
F  medium  40  illinois  52400.00  conservative
F  short   22  arkansas  23600.00  liberal
F  short   68  colorado  78400.00  conservative
M  tall    60  illinois  71700.00  liberal
M  tall    34  delaware  46500.00  moderate
M  medium  25  delaware  37100.00  conservative
M  short   31  illinois  48900.00  moderate
F  short   43  delaware  48000.00  moderate
F  short   58  colorado  65400.00  liberal
M  tall    55  illinois  60700.00  liberal
M  tall    43  colorado  51100.00  moderate
M  tall    43  delaware  53200.00  moderate
M  medium  21  arkansas  37200.00  conservative
F  short   55  delaware  64600.00  conservative
F  short   64  colorado  74800.00  conservative
M  tall    41  illinois  58800.00  moderate
F  medium  64  delaware  72700.00  conservative
M  medium  56  illinois  66600.00  liberal
F  short   31  delaware  36000.00  moderate
M  tall    65  delaware  70100.00  liberal
F  tall    55  illinois  64300.00  conservative
M  short   25  arkansas  40300.00  conservative
F  short   46  delaware  51000.00  moderate
M  tall    36  illinois  53500.00  conservative
F  short   52  illinois  58100.00  moderate
F  short   61  delaware  67900.00  conservative
F  short   57  delaware  65700.00  conservative
M  tall    46  colorado  52600.00  moderate
M  tall    62  arkansas  66800.00  liberal
F  short   55  illinois  62700.00  conservative
M  medium  22  delaware  27700.00  moderate
M  tall    50  illinois  62900.00  conservative
M  tall    32  illinois  41800.00  moderate
M  short   21  delaware  35600.00  conservative
F  medium  44  colorado  52000.00  moderate
F  short   46  illinois  51700.00  moderate
F  short   62  colorado  69700.00  conservative
F  short   57  illinois  66400.00  conservative
M  medium  67  illinois  75800.00  liberal
F  short   29  arkansas  34300.00  liberal
F  short   53  illinois  60100.00  conservative
M  tall    44  arkansas  54800.00  moderate
F  medium  46  colorado  52300.00  moderate
M  tall    20  illinois  30100.00  moderate
M  medium  38  illinois  53500.00  moderate
F  short   50  colorado  58600.00  moderate
F  short   33  colorado  42500.00  moderate
M  tall    33  colorado  39300.00  moderate
F  short   26  colorado  40400.00  conservative
F  short   58  arkansas  70700.00  conservative
F  tall    43  illinois  48000.00  moderate
M  medium  46  arkansas  64400.00  conservative
F  short   60  arkansas  71700.00  conservative
M  tall    42  arkansas  48900.00  moderate
M  tall    56  delaware  56400.00  liberal
M  short   62  colorado  66300.00  liberal
M  short   50  arkansas  64800.00  moderate
F  short   47  illinois  52000.00  moderate
M  tall    67  colorado  80400.00  liberal
M  tall    40  delaware  50400.00  moderate
F  short   42  colorado  48400.00  moderate
F  short   64  arkansas  72000.00  conservative
M  medium  47  arkansas  58700.00  liberal
F  medium  45  colorado  52800.00  moderate
M  tall    25  delaware  40900.00  conservative
F  short   38  arkansas  48400.00  conservative
F  short   55  delaware  60000.00  moderate
M  tall    44  arkansas  60600.00  moderate
F  medium  33  arkansas  41000.00  moderate
F  short   34  delaware  39000.00  moderate
F  short   27  colorado  33700.00  liberal
F  short   32  colorado  40700.00  moderate
F  tall    42  illinois  47000.00  moderate
M  short   24  delaware  40300.00  conservative
F  short   42  colorado  50300.00  moderate
F  short   25  delaware  28000.00  liberal
F  short   51  colorado  58000.00  moderate
M  medium  55  colorado  63500.00  liberal
F  short   44  arkansas  47800.00  liberal
M  short   18  arkansas  39800.00  conservative
M  tall    67  colorado  71600.00  liberal
F  short   45  delaware  50000.00  moderate
F  short   48  arkansas  55800.00  moderate
M  short   25  colorado  39000.00  moderate
M  tall    67  arkansas  78300.00  moderate
F  short   37  delaware  42000.00  moderate
M  short   32  arkansas  42700.00  moderate
F  short   48  arkansas  57000.00  moderate
M  tall    66  delaware  75000.00  liberal
F  tall    61  arkansas  70000.00  conservative
M  medium  58  delaware  68900.00  moderate
F  short   19  arkansas  24000.00  liberal
F  short   38  delaware  43000.00  moderate
M  medium  27  arkansas  36400.00  moderate
F  short   42  arkansas  48000.00  moderate
F  short   60  arkansas  71300.00  conservative
M  tall    27  delaware  34800.00  conservative
F  tall    29  colorado  37100.00  conservative
M  medium  43  arkansas  56700.00  moderate
F  medium  48  arkansas  56700.00  moderate
F  medium  27  delaware  29400.00  liberal
M  tall    44  arkansas  55200.00  conservative
F  short   23  colorado  26300.00  liberal
M  tall    36  colorado  53000.00  liberal
F  short   64  delaware  72500.00  conservative
F  short   29  delaware  30000.00  liberal
M  short   33  arkansas  49300.00  moderate
M  tall    66  colorado  75000.00  liberal
M  medium  21  delaware  34300.00  conservative
F  short   27  arkansas  32700.00  liberal
F  short   29  arkansas  31800.00  liberal
M  tall    31  arkansas  48600.00  moderate
F  short   36  delaware  41000.00  moderate
F  short   49  colorado  55700.00  moderate
M  short   28  arkansas  38400.00  conservative
M  medium  43  delaware  56600.00  moderate
M  medium  46  colorado  58800.00  moderate
F  short   57  arkansas  69800.00  conservative
M  short   52  delaware  59400.00  moderate
M  tall    31  delaware  43500.00  moderate
M  tall    55  arkansas  62000.00  liberal
F  short   50  arkansas  56400.00  moderate
F  short   48  colorado  55900.00  moderate
M  medium  22  delaware  34500.00  conservative
F  short   59  delaware  66700.00  conservative
F  short   34  arkansas  42800.00  liberal
M  tall    64  arkansas  77200.00  liberal
F  short   29  delaware  33500.00  liberal
M  medium  34  colorado  43200.00  moderate
M  medium  61  arkansas  75000.00  liberal
F  short   64  delaware  71100.00  conservative
M  short   29  arkansas  41300.00  conservative
F  short   63  colorado  70600.00  conservative
M  medium  29  colorado  40000.00  conservative
M  tall    51  arkansas  62700.00  moderate
M  tall    24  delaware  37700.00  conservative
F  medium  48  colorado  57500.00  moderate
F  short   18  arkansas  27400.00  conservative
F  short   18  arkansas  20300.00  liberal
F  short   33  colorado  38200.00  liberal
M  medium  20  delaware  34800.00  conservative
F  short   29  delaware  33000.00  liberal
M  short   44  delaware  63000.00  conservative
M  tall    65  delaware  81800.00  conservative
M  tall    56  arkansas  63700.00  liberal
M  medium  52  delaware  58400.00  moderate
M  medium  29  colorado  48600.00  conservative
M  tall    47  colorado  58900.00  moderate
F  medium  68  arkansas  72600.00  liberal
F  short   31  delaware  36000.00  moderate
F  short   61  colorado  62500.00  liberal
F  short   19  colorado  21500.00  liberal
F  tall    38  delaware  43000.00  moderate
M  tall    26  arkansas  42300.00  conservative
F  short   61  colorado  67400.00  conservative
F  short   40  arkansas  46500.00  moderate
M  medium  49  arkansas  65200.00  moderate
F  medium  56  arkansas  67500.00  conservative
M  short   48  colorado  66000.00  moderate
F  short   52  arkansas  56300.00  liberal
M  tall    18  arkansas  29800.00  conservative
M  tall    56  delaware  59300.00  liberal
M  medium  52  colorado  64400.00  moderate
M  medium  18  colorado  28600.00  moderate
M  tall    58  arkansas  66200.00  liberal
M  tall    39  colorado  55100.00  moderate
M  tall    46  arkansas  62900.00  moderate
M  medium  40  colorado  46200.00  moderate
M  medium  60  arkansas  72700.00  liberal
F  short   36  colorado  40700.00  liberal
F  short   44  arkansas  52300.00  moderate
F  short   28  arkansas  31300.00  liberal
F  short   54  delaware  62600.00  conservative
M  medium  51  arkansas  61200.00  moderate
M  short   32  colorado  46100.00  moderate
F  short   55  arkansas  62700.00  conservative
F  short   25  delaware  26200.00  liberal
F  medium  33  delaware  37300.00  liberal
M  medium  29  colorado  46200.00  conservative
F  short   65  arkansas  72700.00  conservative
M  tall    43  colorado  51400.00  moderate
M  short   54  colorado  64800.00  liberal
F  short   61  colorado  72700.00  conservative
F  short   52  colorado  63600.00  conservative
F  short   30  colorado  33500.00  liberal
F  short   29  arkansas  31400.00  liberal
M  tall    47  delaware  59400.00  moderate
F  short   39  colorado  47800.00  moderate
F  short   47  delaware  52000.00  moderate
M  medium  49  arkansas  58600.00  moderate
M  tall    63  delaware  67400.00  liberal
M  medium  30  arkansas  39200.00  conservative
M  tall    61  delaware  69600.00  liberal
M  medium  47  delaware  58700.00  moderate
F  short   30  delaware  34500.00  liberal
M  medium  51  delaware  58000.00  moderate
M  medium  24  arkansas  38800.00  moderate
M  short   49  arkansas  64500.00  moderate
F  medium  66  delaware  74500.00  conservative
M  tall    65  arkansas  76900.00  conservative
M  short   46  colorado  58000.00  conservative
M  tall    45  delaware  51800.00  moderate
M  short   47  arkansas  63600.00  conservative
M  tall    29  arkansas  44800.00  conservative
M  tall    57  delaware  69300.00  liberal
M  medium  20  arkansas  28700.00  liberal
M  medium  35  arkansas  43400.00  moderate
M  tall    61  delaware  67000.00  liberal
M  short   31  delaware  37300.00  moderate
F  short   18  arkansas  20800.00  liberal
F  medium  26  delaware  29200.00  liberal
M  medium  28  arkansas  36400.00  liberal
M  tall    59  delaware  69400.00  liberal

Normalized and encoded data:

# people_240.txt
# sex (M = 0.0 0.5  F = 0.5, 0.0)
# height (short, medium, tall) equal-interval
# age /100
# State (Arkansas, Colorado, Delaware, Illinois) one-over-n-hot (4)
# income / 100,000
# politics (conservative, moderate, liberal) one-over-n-hot (3)
#
0.5, 0.0, 0.25, 0.24, 0.25, 0.00, 0.00, 0.00, 0.29500, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.75, 0.39, 0.00, 0.00, 0.25, 0.00, 0.51200, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.63, 0.00, 0.25, 0.00, 0.00, 0.75800, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.50, 0.36, 0.00, 0.00, 0.00, 0.25, 0.44500, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.27, 0.00, 0.25, 0.00, 0.00, 0.28600, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.50, 0.00, 0.25, 0.00, 0.00, 0.56500, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.50, 0.50, 0.00, 0.00, 0.00, 0.25, 0.55000, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.19, 0.00, 0.00, 0.25, 0.00, 0.32700, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.22, 0.00, 0.00, 0.00, 0.25, 0.27700, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.39, 0.00, 0.00, 0.25, 0.00, 0.47100, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.34, 0.25, 0.00, 0.00, 0.00, 0.39400, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.50, 0.22, 0.00, 0.00, 0.00, 0.25, 0.33500, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.50, 0.35, 0.00, 0.00, 0.25, 0.00, 0.35200, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.75, 0.33, 0.00, 0.25, 0.00, 0.00, 0.46400, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.45, 0.00, 0.25, 0.00, 0.00, 0.54100, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.42, 0.00, 0.00, 0.00, 0.25, 0.50700, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.33, 0.00, 0.25, 0.00, 0.00, 0.46800, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.75, 0.25, 0.00, 0.00, 0.25, 0.00, 0.30000, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.50, 0.31, 0.00, 0.25, 0.00, 0.00, 0.46400, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.27, 0.25, 0.00, 0.00, 0.00, 0.32500, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.48, 0.00, 0.00, 0.00, 0.25, 0.54000, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.64, 0.00, 0.00, 0.00, 0.25, 0.71300, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.50, 0.61, 0.00, 0.25, 0.00, 0.00, 0.72400, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.54, 0.00, 0.00, 0.00, 0.25, 0.61000, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.29, 0.25, 0.00, 0.00, 0.00, 0.36300, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.50, 0.00, 0.00, 0.25, 0.00, 0.55000, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.50, 0.55, 0.00, 0.00, 0.00, 0.25, 0.62500, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.50, 0.40, 0.00, 0.00, 0.00, 0.25, 0.52400, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.22, 0.25, 0.00, 0.00, 0.00, 0.23600, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.68, 0.00, 0.25, 0.00, 0.00, 0.78400, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.75, 0.60, 0.00, 0.00, 0.00, 0.25, 0.71700, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.75, 0.34, 0.00, 0.00, 0.25, 0.00, 0.46500, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.50, 0.25, 0.00, 0.00, 0.25, 0.00, 0.37100, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.25, 0.31, 0.00, 0.00, 0.00, 0.25, 0.48900, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.43, 0.00, 0.00, 0.25, 0.00, 0.48000, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.58, 0.00, 0.25, 0.00, 0.00, 0.65400, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.75, 0.55, 0.00, 0.00, 0.00, 0.25, 0.60700, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.75, 0.43, 0.00, 0.25, 0.00, 0.00, 0.51100, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.43, 0.00, 0.00, 0.25, 0.00, 0.53200, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.50, 0.21, 0.25, 0.00, 0.00, 0.00, 0.37200, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.55, 0.00, 0.00, 0.25, 0.00, 0.64600, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.64, 0.00, 0.25, 0.00, 0.00, 0.74800, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.75, 0.41, 0.00, 0.00, 0.00, 0.25, 0.58800, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.50, 0.64, 0.00, 0.00, 0.25, 0.00, 0.72700, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.50, 0.56, 0.00, 0.00, 0.00, 0.25, 0.66600, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.31, 0.00, 0.00, 0.25, 0.00, 0.36000, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.65, 0.00, 0.00, 0.25, 0.00, 0.70100, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.75, 0.55, 0.00, 0.00, 0.00, 0.25, 0.64300, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.25, 0.25, 0.25, 0.00, 0.00, 0.00, 0.40300, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.46, 0.00, 0.00, 0.25, 0.00, 0.51000, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.36, 0.00, 0.00, 0.00, 0.25, 0.53500, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.52, 0.00, 0.00, 0.00, 0.25, 0.58100, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.61, 0.00, 0.00, 0.25, 0.00, 0.67900, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.57, 0.00, 0.00, 0.25, 0.00, 0.65700, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.75, 0.46, 0.00, 0.25, 0.00, 0.00, 0.52600, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.62, 0.25, 0.00, 0.00, 0.00, 0.66800, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.55, 0.00, 0.00, 0.00, 0.25, 0.62700, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.50, 0.22, 0.00, 0.00, 0.25, 0.00, 0.27700, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.50, 0.00, 0.00, 0.00, 0.25, 0.62900, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.75, 0.32, 0.00, 0.00, 0.00, 0.25, 0.41800, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.25, 0.21, 0.00, 0.00, 0.25, 0.00, 0.35600, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.50, 0.44, 0.00, 0.25, 0.00, 0.00, 0.52000, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.46, 0.00, 0.00, 0.00, 0.25, 0.51700, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.62, 0.00, 0.25, 0.00, 0.00, 0.69700, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.57, 0.00, 0.00, 0.00, 0.25, 0.66400, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.50, 0.67, 0.00, 0.00, 0.00, 0.25, 0.75800, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.29, 0.25, 0.00, 0.00, 0.00, 0.34300, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.53, 0.00, 0.00, 0.00, 0.25, 0.60100, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.75, 0.44, 0.25, 0.00, 0.00, 0.00, 0.54800, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.50, 0.46, 0.00, 0.25, 0.00, 0.00, 0.52300, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.20, 0.00, 0.00, 0.00, 0.25, 0.30100, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.50, 0.38, 0.00, 0.00, 0.00, 0.25, 0.53500, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.50, 0.00, 0.25, 0.00, 0.00, 0.58600, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.33, 0.00, 0.25, 0.00, 0.00, 0.42500, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.33, 0.00, 0.25, 0.00, 0.00, 0.39300, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.26, 0.00, 0.25, 0.00, 0.00, 0.40400, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.58, 0.25, 0.00, 0.00, 0.00, 0.70700, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.75, 0.43, 0.00, 0.00, 0.00, 0.25, 0.48000, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.50, 0.46, 0.25, 0.00, 0.00, 0.00, 0.64400, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.60, 0.25, 0.00, 0.00, 0.00, 0.71700, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.75, 0.42, 0.25, 0.00, 0.00, 0.00, 0.48900, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.56, 0.00, 0.00, 0.25, 0.00, 0.56400, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.25, 0.62, 0.00, 0.25, 0.00, 0.00, 0.66300, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.25, 0.50, 0.25, 0.00, 0.00, 0.00, 0.64800, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.47, 0.00, 0.00, 0.00, 0.25, 0.52000, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.67, 0.00, 0.25, 0.00, 0.00, 0.80400, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.75, 0.40, 0.00, 0.00, 0.25, 0.00, 0.50400, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.42, 0.00, 0.25, 0.00, 0.00, 0.48400, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.64, 0.25, 0.00, 0.00, 0.00, 0.72000, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.50, 0.47, 0.25, 0.00, 0.00, 0.00, 0.58700, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.50, 0.45, 0.00, 0.25, 0.00, 0.00, 0.52800, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.25, 0.00, 0.00, 0.25, 0.00, 0.40900, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.38, 0.25, 0.00, 0.00, 0.00, 0.48400, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.55, 0.00, 0.00, 0.25, 0.00, 0.60000, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.44, 0.25, 0.00, 0.00, 0.00, 0.60600, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.50, 0.33, 0.25, 0.00, 0.00, 0.00, 0.41000, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.34, 0.00, 0.00, 0.25, 0.00, 0.39000, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.27, 0.00, 0.25, 0.00, 0.00, 0.33700, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.32, 0.00, 0.25, 0.00, 0.00, 0.40700, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.75, 0.42, 0.00, 0.00, 0.00, 0.25, 0.47000, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.25, 0.24, 0.00, 0.00, 0.25, 0.00, 0.40300, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.42, 0.00, 0.25, 0.00, 0.00, 0.50300, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.25, 0.00, 0.00, 0.25, 0.00, 0.28000, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.51, 0.00, 0.25, 0.00, 0.00, 0.58000, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.50, 0.55, 0.00, 0.25, 0.00, 0.00, 0.63500, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.44, 0.25, 0.00, 0.00, 0.00, 0.47800, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.25, 0.18, 0.25, 0.00, 0.00, 0.00, 0.39800, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.75, 0.67, 0.00, 0.25, 0.00, 0.00, 0.71600, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.45, 0.00, 0.00, 0.25, 0.00, 0.50000, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.48, 0.25, 0.00, 0.00, 0.00, 0.55800, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.25, 0.25, 0.00, 0.25, 0.00, 0.00, 0.39000, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.67, 0.25, 0.00, 0.00, 0.00, 0.78300, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.37, 0.00, 0.00, 0.25, 0.00, 0.42000, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.25, 0.32, 0.25, 0.00, 0.00, 0.00, 0.42700, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.48, 0.25, 0.00, 0.00, 0.00, 0.57000, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.66, 0.00, 0.00, 0.25, 0.00, 0.75000, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.75, 0.61, 0.25, 0.00, 0.00, 0.00, 0.70000, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.50, 0.58, 0.00, 0.00, 0.25, 0.00, 0.68900, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.19, 0.25, 0.00, 0.00, 0.00, 0.24000, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.38, 0.00, 0.00, 0.25, 0.00, 0.43000, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.50, 0.27, 0.25, 0.00, 0.00, 0.00, 0.36400, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.42, 0.25, 0.00, 0.00, 0.00, 0.48000, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.60, 0.25, 0.00, 0.00, 0.00, 0.71300, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.75, 0.27, 0.00, 0.00, 0.25, 0.00, 0.34800, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.75, 0.29, 0.00, 0.25, 0.00, 0.00, 0.37100, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.50, 0.43, 0.25, 0.00, 0.00, 0.00, 0.56700, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.50, 0.48, 0.25, 0.00, 0.00, 0.00, 0.56700, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.50, 0.27, 0.00, 0.00, 0.25, 0.00, 0.29400, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.75, 0.44, 0.25, 0.00, 0.00, 0.00, 0.55200, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.23, 0.00, 0.25, 0.00, 0.00, 0.26300, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.75, 0.36, 0.00, 0.25, 0.00, 0.00, 0.53000, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.64, 0.00, 0.00, 0.25, 0.00, 0.72500, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.29, 0.00, 0.00, 0.25, 0.00, 0.30000, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.25, 0.33, 0.25, 0.00, 0.00, 0.00, 0.49300, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.66, 0.00, 0.25, 0.00, 0.00, 0.75000, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.50, 0.21, 0.00, 0.00, 0.25, 0.00, 0.34300, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.27, 0.25, 0.00, 0.00, 0.00, 0.32700, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.29, 0.25, 0.00, 0.00, 0.00, 0.31800, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.75, 0.31, 0.25, 0.00, 0.00, 0.00, 0.48600, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.36, 0.00, 0.00, 0.25, 0.00, 0.41000, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.49, 0.00, 0.25, 0.00, 0.00, 0.55700, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.25, 0.28, 0.25, 0.00, 0.00, 0.00, 0.38400, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.50, 0.43, 0.00, 0.00, 0.25, 0.00, 0.56600, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.50, 0.46, 0.00, 0.25, 0.00, 0.00, 0.58800, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.57, 0.25, 0.00, 0.00, 0.00, 0.69800, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.25, 0.52, 0.00, 0.00, 0.25, 0.00, 0.59400, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.31, 0.00, 0.00, 0.25, 0.00, 0.43500, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.55, 0.25, 0.00, 0.00, 0.00, 0.62000, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.50, 0.25, 0.00, 0.00, 0.00, 0.56400, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.48, 0.00, 0.25, 0.00, 0.00, 0.55900, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.50, 0.22, 0.00, 0.00, 0.25, 0.00, 0.34500, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.59, 0.00, 0.00, 0.25, 0.00, 0.66700, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.34, 0.25, 0.00, 0.00, 0.00, 0.42800, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.75, 0.64, 0.25, 0.00, 0.00, 0.00, 0.77200, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.29, 0.00, 0.00, 0.25, 0.00, 0.33500, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.50, 0.34, 0.00, 0.25, 0.00, 0.00, 0.43200, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.50, 0.61, 0.25, 0.00, 0.00, 0.00, 0.75000, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.64, 0.00, 0.00, 0.25, 0.00, 0.71100, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.25, 0.29, 0.25, 0.00, 0.00, 0.00, 0.41300, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.63, 0.00, 0.25, 0.00, 0.00, 0.70600, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.50, 0.29, 0.00, 0.25, 0.00, 0.00, 0.40000, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.75, 0.51, 0.25, 0.00, 0.00, 0.00, 0.62700, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.24, 0.00, 0.00, 0.25, 0.00, 0.37700, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.50, 0.48, 0.00, 0.25, 0.00, 0.00, 0.57500, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.18, 0.25, 0.00, 0.00, 0.00, 0.27400, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.18, 0.25, 0.00, 0.00, 0.00, 0.20300, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.33, 0.00, 0.25, 0.00, 0.00, 0.38200, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.50, 0.20, 0.00, 0.00, 0.25, 0.00, 0.34800, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.29, 0.00, 0.00, 0.25, 0.00, 0.33000, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.25, 0.44, 0.00, 0.00, 0.25, 0.00, 0.63000, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.75, 0.65, 0.00, 0.00, 0.25, 0.00, 0.81800, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.75, 0.56, 0.25, 0.00, 0.00, 0.00, 0.63700, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.50, 0.52, 0.00, 0.00, 0.25, 0.00, 0.58400, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.50, 0.29, 0.00, 0.25, 0.00, 0.00, 0.48600, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.75, 0.47, 0.00, 0.25, 0.00, 0.00, 0.58900, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.50, 0.68, 0.25, 0.00, 0.00, 0.00, 0.72600, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.31, 0.00, 0.00, 0.25, 0.00, 0.36000, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.61, 0.00, 0.25, 0.00, 0.00, 0.62500, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.19, 0.00, 0.25, 0.00, 0.00, 0.21500, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.75, 0.38, 0.00, 0.00, 0.25, 0.00, 0.43000, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.26, 0.25, 0.00, 0.00, 0.00, 0.42300, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.61, 0.00, 0.25, 0.00, 0.00, 0.67400, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.40, 0.25, 0.00, 0.00, 0.00, 0.46500, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.50, 0.49, 0.25, 0.00, 0.00, 0.00, 0.65200, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.50, 0.56, 0.25, 0.00, 0.00, 0.00, 0.67500, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.25, 0.48, 0.00, 0.25, 0.00, 0.00, 0.66000, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.52, 0.25, 0.00, 0.00, 0.00, 0.56300, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.75, 0.18, 0.25, 0.00, 0.00, 0.00, 0.29800, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.75, 0.56, 0.00, 0.00, 0.25, 0.00, 0.59300, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.50, 0.52, 0.00, 0.25, 0.00, 0.00, 0.64400, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.50, 0.18, 0.00, 0.25, 0.00, 0.00, 0.28600, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.58, 0.25, 0.00, 0.00, 0.00, 0.66200, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.75, 0.39, 0.00, 0.25, 0.00, 0.00, 0.55100, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.46, 0.25, 0.00, 0.00, 0.00, 0.62900, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.50, 0.40, 0.00, 0.25, 0.00, 0.00, 0.46200, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.50, 0.60, 0.25, 0.00, 0.00, 0.00, 0.72700, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.36, 0.00, 0.25, 0.00, 0.00, 0.40700, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.44, 0.25, 0.00, 0.00, 0.00, 0.52300, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.28, 0.25, 0.00, 0.00, 0.00, 0.31300, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.54, 0.00, 0.00, 0.25, 0.00, 0.62600, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.50, 0.51, 0.25, 0.00, 0.00, 0.00, 0.61200, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.25, 0.32, 0.00, 0.25, 0.00, 0.00, 0.46100, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.55, 0.25, 0.00, 0.00, 0.00, 0.62700, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.25, 0.00, 0.00, 0.25, 0.00, 0.26200, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.50, 0.33, 0.00, 0.00, 0.25, 0.00, 0.37300, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.50, 0.29, 0.00, 0.25, 0.00, 0.00, 0.46200, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.65, 0.25, 0.00, 0.00, 0.00, 0.72700, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.75, 0.43, 0.00, 0.25, 0.00, 0.00, 0.51400, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.25, 0.54, 0.00, 0.25, 0.00, 0.00, 0.64800, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.61, 0.00, 0.25, 0.00, 0.00, 0.72700, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.52, 0.00, 0.25, 0.00, 0.00, 0.63600, 0.3333, 0.0000, 0.0000
0.5, 0.0, 0.25, 0.30, 0.00, 0.25, 0.00, 0.00, 0.33500, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.25, 0.29, 0.25, 0.00, 0.00, 0.00, 0.31400, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.75, 0.47, 0.00, 0.00, 0.25, 0.00, 0.59400, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.39, 0.00, 0.25, 0.00, 0.00, 0.47800, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.47, 0.00, 0.00, 0.25, 0.00, 0.52000, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.50, 0.49, 0.25, 0.00, 0.00, 0.00, 0.58600, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.63, 0.00, 0.00, 0.25, 0.00, 0.67400, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.50, 0.30, 0.25, 0.00, 0.00, 0.00, 0.39200, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.75, 0.61, 0.00, 0.00, 0.25, 0.00, 0.69600, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.50, 0.47, 0.00, 0.00, 0.25, 0.00, 0.58700, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.30, 0.00, 0.00, 0.25, 0.00, 0.34500, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.50, 0.51, 0.00, 0.00, 0.25, 0.00, 0.58000, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.50, 0.24, 0.25, 0.00, 0.00, 0.00, 0.38800, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.25, 0.49, 0.25, 0.00, 0.00, 0.00, 0.64500, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.50, 0.66, 0.00, 0.00, 0.25, 0.00, 0.74500, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.75, 0.65, 0.25, 0.00, 0.00, 0.00, 0.76900, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.25, 0.46, 0.00, 0.25, 0.00, 0.00, 0.58000, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.75, 0.45, 0.00, 0.00, 0.25, 0.00, 0.51800, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.25, 0.47, 0.25, 0.00, 0.00, 0.00, 0.63600, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.75, 0.29, 0.25, 0.00, 0.00, 0.00, 0.44800, 0.3333, 0.0000, 0.0000
0.0, 0.5, 0.75, 0.57, 0.00, 0.00, 0.25, 0.00, 0.69300, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.50, 0.20, 0.25, 0.00, 0.00, 0.00, 0.28700, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.50, 0.35, 0.25, 0.00, 0.00, 0.00, 0.43400, 0.0000, 0.3333, 0.0000
0.0, 0.5, 0.75, 0.61, 0.00, 0.00, 0.25, 0.00, 0.67000, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.25, 0.31, 0.00, 0.00, 0.25, 0.00, 0.37300, 0.0000, 0.3333, 0.0000
0.5, 0.0, 0.25, 0.18, 0.25, 0.00, 0.00, 0.00, 0.20800, 0.0000, 0.0000, 0.3333
0.5, 0.0, 0.50, 0.26, 0.00, 0.00, 0.25, 0.00, 0.29200, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.50, 0.28, 0.25, 0.00, 0.00, 0.00, 0.36400, 0.0000, 0.0000, 0.3333
0.0, 0.5, 0.75, 0.59, 0.00, 0.00, 0.25, 0.00, 0.69400, 0.0000, 0.0000, 0.3333

My Top Ten Favorite Dr. Fu Manchu Movies

Dr. Fu Manchu was arguably the first super-villain. He was created by author Arthur Henry Ward (1883-1959) who used the pen name Sax Rohmer. Rohmer wrote 14 Fu Manchu novels from 1913 to 1959. The Fu Manchu novels were very popular and roughly 10 movies featuring the evil doctor and his daughter(s) were made from 1929 to 1969.

Here are my ten favorites (not difficult because there are only ten), listed in chronological order. None of these movies is particularly good but they all have a certain charm and interest if you like old movies like I do. The James Bond “Dr. No” novel by Ian Fleming and movie are based on many of the ideas in the Fu Manchu novels and movies.

---

1. The Mysterious Dr. Fu Manchu (1929) – Dr. Fu Manchu (actor Warner Oland) is a Chinese doctor. During the Boxer Rebellion (1899-1901) uprising against the British, Manchu’s wife and daughter are accidentally killed by British soldiers. Manchu goes to England and vows revenge. Opposing Manchu are Nayland Smith (actor O.P. Heggie) of Scotland Yard and his pal Dr. Petrie (Neil Hamilton). In the end, Manchu is captured and drinks poison to commit suicide. The first of three films starring Warner Oland (who also played Charlie Chan) as Manchu. A very early sound film that has mostly historical interest rather than entertainment value. My grade: NA.

---

2. The Return of Dr. Fu Manchu (1930) – Because the previous film was so popular, Fu Manchu (Warner Oland) turns out to have not died and he continues seeking revenge on those he holds responsible for the death of his wife and child. Essentially a continuation of the previous film. Manchu is defeated again by Nayland Smith (O.P. Heggie) and Dr. Petrie (Neil Hamilton). Like many early sound films, this one plays a bit like like a silent film. A slow-moving film that has mostly historical interest. Notice the poster omits “Dr.” in the title. My grade: NA.

---

3. Daughter of the Dragon (1931) – The third and final film with Oland as Manchu. Princess Ling Moy (actress Anna May Wong) is the long-lost daughter of Fu Manchu. Manchu reveals this to her and out of loyalty she becomes his evil assistant. Complicating matters is the fact that Ling Moy’s boyfriend Ah Kee is a Chinese agent seeking to capture her father. Furthermore Ronald Petrie (a young aristocrat, but not Dr. Petrie as in all other films) is a target of Manchu and Ling Moy but she falls in love with him. Ah Kee shoots and kills Manchu about halfway through the movie and later shoots and kills Ling Moy just before dying himself. Nayland Smith does not appear in this movie. Kind of a depressing ending. My grade: C.

---

4. The Mask of Fu Manchu (1932) – The sinister Dr. Fu Manchu (actor Boris Karloff) and a daughter Fah Lo See (actress Myrna Loy) compete with a British scientific expedition to find the sword of Genghis Khan which they believe will give them mystical power. Nayland Smith (actor Lewis Stone) and his young colleague Terrence Granville (actor Charles Starrett; no Dr. Petrie in this film) thwart Manchu and kill him using his own death ray. This is my favorite Fu Manchu movie mostly because of the superior acting by Karloff and Loy. My grade: B.

---

5. Drums of Fu Manchu (1943) – This movie was compiled together from a 15-part serial shown in theaters from 1940-41. Here Fu Manchu (actor Henry Brandon) seeks to acquire the sceptre of Genghis Khan, which will unite all of Asia under his command. Nayland Smith (actor William Royle) and Dr. Petrie (Olaf Hytten) join with young Allan Parker (Robert Kellard) to defeat the evil doctor and his daughter Fah Lo Suee (actress Gloria Franklin). Rarely for the time, the villains escape at the end. Because the movie is a serial compilation there is non-stop action but not much coherence. My grade: C+.

---

6. The Face of Fu Manchu (1965) – The movie opens with the apparent execution of Fu Manchu (actor Christopher Lee) in China. But a few months later in London, Nayland Smith (Nigel Green) and Dr. Petrie (Howard Marion-Crawford) deduce that Manchu is still alive. Manchu, and his daughter Lin Tang (actress Tsai Chin), kidnap a scientist who has knowledge of how to use a rare poppy seed to create a deadly poison. In the end, it looks like Manchu and Tang have succeeded but Smith has booby trapped Manchu’s hideout and it’s destroyed. This is the first of five consecutive Fu Manchu films starring Christopher Lee as Manchu and Tsai Chin as daughter Lin Tang, released one per year from 1965 to 1969. Nayland Smith was played by three different actors (Nigel Green once, Douglas Wilmer twice, Richard Greene twice). My grade: B-.

---

7. The Brides of Fu Manchu (1966) – Dr. Fu Manchu (Christopher Lee) and his daughter Lin Tang (Tsai Chin) kidnap the daughters of scientists and then blackmail the fathers into helping him create a secret sonic super weapon to use to conquer the world. Nayland Smith (Douglas Wilmer) and Dr. Petrie (Howard Marion-Crawford) defeat the evil doctor and rescue the girls. My grade: C+.

---

8. The Vengeance of Fu Manchu (1967) – In this one, Fu Manchu (Christopher Lee) and his evil daughter Lin Tang (Tsai Chin) replace Nayland Smith (Douglas Wilmer) with a lookalike. The idea is to create a giant worldwide organization of criminals and control the entire world. Once again Smith and Dr. Petrie (Howard Marion-Crawford) defeat the villains. My grade: C.

---

9. The Blood of Fu Manchu (1968) – The evil Fu Manchu (actor Christopher Lee) and his equally evil daughter Lin Tang (Tsai Chin) have established a new hideout in the Amazon jungle. They have developed a deadly poison that affects men but not women. They plan to use women as carriers to assassinate political leaders around the world and control the world. Nayland Smith (actor Richard Greene) and Dr. Petrie (Howard Marion-Crawford) defeat the doctor again. My grade: C.

---

10. The Castle of Fu Manchu (1969) – Fu Manchu (Christopher Lee) and his daughter Lin Tang (actress Tsai Chin) are at it again, this time based in a castle in Istanbul. They are building a new secret super weapon, financed in part by opium smuggling. And once again, Nayland Smith (working for Interpol now, played by actor Richard Greene) and Dr. Petrie (Howard Marion-Crawford) defeat the plans and rescue the hostages. My grade: C+.

---

Special Mention

---

In addition to the ten films listed above, there was a short-lived TV series titled “The Adventures of Dr. Fu Manchu”. The show ran for 13 episodes in 1956. It starred Glen Gordon as Dr. Fu Manchu, actress Laurette Luez as Manchu’s semi-unwilling slave/assistant Karameneh, Lester Matthews as Nayland Smith, and Clark Howat as Dr. Petrie. I kind of like these old shows even though they feel like typical 1950s adventure TV. At the end of every episode, Manchu would be defeated and he would pick up the black King’s bishop from an ornate chess board and break it in half in frustration. My grade: C-/B- depending on episode.

Notice the chess board is oriented incorrectly — the lower left square should be black, not white. Argh! Annoying that TV people can’t seem to ever get this correct.

---

---

Here are three representative book covers. Left: The first edition of the first novel from 1913. Center: A cover from 1959. Right: A cover from 1962.

---

“Multi-Class Classification Using LightGBM” in Visual Studio Magazine

I wrote an article titled “Multi-Class Classification Using LightGBM” in the May 2024 edition of Microsoft Visual Studio Magazine. See https://visualstudiomagazine.com/Articles/2024/05/02/LightGBM-multi-class-classification.aspx.

A multi-class classification problem is one where the goal is to predict a discrete variable that has three or more possible values. For example, you might want to predict a person’s political leaning (conservative, moderate, liberal) from sex, age, state of residence and annual income. There are many machine learning techniques for multi-class classification. One of the most powerful techniques is to use the LightGBM (lightweight gradient boosting machine) system.

LightGBM is a sophisticated, open-source, tree-based system that was introduced in 2017. LightGBM can perform multi-class classification, binary classification (predict one of two possible values), regression (predict a single numeric value) and ranking.

My article presents a complete end-to-end demo. LightGBM has three programming language interfaces — C, Python and R. The demo program uses the Python language API.

I used one of my standard synthetic datasets. The raw data look like:

F  24  michigan  29500.00  liberal
M  39  oklahoma  51200.00  moderate
F  63  nebraska  75800.00  conservative
M  36  michigan  44500.00  moderate
F  27  nebraska  28600.00  liberal
. . .

When using LightGBM, you encode categorical data using ordinal encoding. Unlike most machine learning classification techniques, you don’t need to normalize numeric data. The encoded data looks like:

1, 24, 0, 29500.00, 2
0, 39, 2, 51200.00, 1
1, 63, 1, 75800.00, 0
0, 36, 0, 44500.00, 1
1, 27, 1, 28600.00, 2
. . .

My article explain how to install Python and LightGBM for readers who are new to both.

The key statements that create and train the demo LightGBM multi-class classifier are:

  print("Creating and training LGBM multi-class model ")
  params = {
    # 'objective': 'multiclass',  # not needed
    'boosting_type': 'gbdt',  # default
    'num_leaves': 31,  # default
    'max_depth':-1,  # default (unlimited) 
    'n_estimators': 50,  # default = 100
    'learning_rate': 0.05,  # default = 0.10
    'min_data_in_leaf': 5,  # default = 20
    'random_state': 0,
    'verbosity': -1  # only fatal. default = 1 error, warn
  }
  model = lgbm.LGBMClassifier(**params)  # scikit API
  model.fit(train_x, train_y)
  print("Done ")

The main challenge when using LightGBM is wading through the dozens of parameters. The LGBMClassifier class/object has 19 parameters (num_leaves, max_depth and so on) and behind the scenes there are 57 Learning Control Parameters (min_data_in_leaf, bagging_fraction and so on), for a total of 76 parameters to deal with. A large section of my article explains which parameters to change and which to leave alone.

Arguably, the two most powerful techniques for multi-class classification on non-trivial datasets are neural networks and tree boosting. In some recent multi-class classification challenges, LightGBM entries have dominated the contest leader board. This may be due, in part, to the fact that LightGBM can be used out-of-the-box, which leaves a lot of time for hyperparameter fine-tuning. Using a neural network classifier requires significantly more background knowledge and effort.

---

It’s sometimes difficult to classify films because several different genres can be represented. One of my favorite classifications is science fiction mystery movies.

Left: Dark City (1998) – A man wakes up next to a murdered prostitute. He can’t remember who he is or how he got there. He is pursued by the creepy Strangers. Who are they? Why is it always night time?

Center: Gog (1954) – A government investigator (Richard Egan) is sent to a super-secret underground desert laboratory complex to solve a series of bizarre deaths. Which of the many suspects is responsible?

Right: The Power (1968) – One of a group of six scientists, including one played by George Hamilton, has super mind control powers and is murdering the others in the group one by one. Who is it and how can he/she be stopped?

---

Data Anomaly Detection Using Principal Component Analysis (PCA) Reconstruction Error

One evening, while I was walking my two dogs, I thought about the possibility of looking for data anomalies by analyzing principal component analysis (PCA) reconstruction error. Bottom line: the technique works, but it just doesn’t feel right to me.

The ideas here are extremely complex and can only be explained by using a concrete example. I implemented the idea using raw C#. Using Python and the scikit library would have been much, much easier. I started with a small, 12-item subset of the Penguin dataset:

[ 0]     39.5     17.4    186.0   3800.0
[ 1]     40.3     18.0    195.0   3250.0
[ 2]     36.7     19.3    193.0   3450.0
[ 3]     38.9     17.8    181.0   3625.0
[ 4]     46.5     17.9    192.0   3500.0
[ 5]     45.4     18.7    188.0   3525.0
[ 6]     45.2     17.8    198.0   3950.0
[ 7]     46.1     18.2    178.0   3250.0
[ 8]     46.1     13.2    211.0   4500.0
[ 9]     48.7     14.1    210.0   4450.0
[10]     46.5     13.5    210.0   4550.0
[11]     45.4     14.6    211.0   4800.0

Each item is one of three species of penguin. The fields are bill length, bill width, flipper length, body mass.

I performed z-score standardization on the source data — this is required for PCA. Then I computed the eigenvalues and eigenvectors of the standardized data — this is one of the most complex operations in numerical programming. To compute the eigens, I used the singular value decomposition (SVD) technique (I could have used the classical covariance matrix technique).

Because the source data has 12 rows and 4 columns, there are 4 eigenvalues, and 4 eigenvectors, each with 4 values. The percentage of variance explained by the 4 eigenvectors are 0.7801, 0.1578, 0.0409, 0.0211 and so the variance explained by just the first 2 eigenvectors is 0.7801 + 0.1578 = 0.9379.

I used the first 2 eigenvectors to reconstruct the source data. Then I computed the Euclidean distance as a measure of error between the source data and the reconstructed data:

[ 0]     39.6     17.8    191.4   3662.2  | recon err =  137.8913
[ 1]     40.3     18.2    189.1   3556.6  | recon err =  306.6433
[ 2]     36.5     18.6    188.2   3511.9  | recon err =   62.1100
[ 3]     39.1     18.5    187.7   3489.6  | recon err =  135.5649
[ 4]     46.4     17.7    189.2   3569.5  | recon err =   69.5266
[ 5]     45.3     18.2    186.8   3455.4  | recon err =   69.6141
[ 6]     45.0     16.8    195.1   3844.0  | recon err =  106.0892
[ 7]     46.3     18.9    182.0   3231.4  | recon err =   19.0492
[ 8]     46.3     13.9    211.7   4619.6  | recon err =  119.6264
[ 9]     48.7     14.1    209.0   4497.1  | recon err =   47.1339
[10]     46.6     13.9    211.1   4594.7  | recon err =   44.7474
[11]     45.2     13.9    211.7   4618.0  | recon err =  182.0142

Based on this analysis, the largest reconstruction error is 306.6433 which is associated with data item [1] and so this is “the most anomalous” in some sense. The source item [1] and its reconstructed item are:

original:      40.3     18.0    195.0   3250.0
reconstructed: 40.3     18.2    189.1   3556.6

The reconstruction error is dominated by the body mass term. Ugh. This makes sense because the magnitudes of the body mass values are much greater than the other variables. This means you’d probably have to normalize the source data first (to get a variable values in the same range) and then z-score standardize the data (to accommodate PCA).

So, technically, the PCA reconstruction error technique works. But the technique just doesn’t feel right to me, based on many years of experience.

The technique is very, very, very complex. And complex is almost always bad. And because in order to apply PCA, source data must be z-score standardized, it can only work with strictly numeric data, not mixed numeric and categorical. And because the reconstruction technique must use a discrete number of eigenvectors for reconstruction, the technique is not very granular.

There may be some scenarios where anomaly detection based on PCA reconstruction error is useful, but I suspect other techniques are better choices in almost all situations. But it was an interesting exploration anyway.

I usually post my demo code, but I’m not going to do so for this topic. The code is very long and very ugly.

---

Most of the experienced engineers I know, have a good intuitive sense of when a software system design is overly-complex. I’m no expert on bicycles, but my intuition tells me that these two examples might be a bit too complex.

---

One-Shot Learning, Few-Shot Learning, Zero-Shot Learning, and Fine-Tuning

The terms one-shot learning, few-shot learning, zero-shot learning, and fine-tuning don’t have universally agreed-upon definitions. All four terms are kinds of “transfer learning” where the goal is to start with an existing model and use it on a new problem. That said, here’s a set of four brief, incomplete, but reasonable, explanations.

One-Shot Learning – Usually applies to image classification. The goal is to classify an image when you only have one example of each class. For example, a bank may have just one example of each customer’s signature and a new signature on a receipt must be classified as authentic or fraudulent. One common technique is to use what’s called Siamese network architecture.

Few-Shot Learning – Usually applies to image classification. The goal is to classify an image when you only have a few (perhaps a dozen or so) examples of each class of labeled data. For example, you have a trained model that can classify hundreds of different classes of animals, but your system acquires just five images of a new previously unseen type of animal, such as an axolotl. One of many possible techniques is to generate hundreds of variations of the five new images (by stretching, horizontally inverting, etc.), add them to the original training data, and then retrain the model. A more sophisticated approach for few-shot learning is called “model agnostic meta-learning” (MAML), where a base model is trained specifically so that new classes can be trained quickly and easily.

Zero-Shot Learning – Often applies to image classification. The goal is to classify an image using an existing model that has never been trained on the image. For example, you have a trained model that can classify hundreds of different classes of automobiles but you need it to classify a new previously unseen example, such as a Chrysler PT Cruiser. There’s no way to create new knowledge from nothing, so one common technique for zero-shot learning is kind of a cheat: you incorporate an auxiliary set of data images along with text information such as “the 2001 PT Cruiser has a boxy look that resembles the 2006 Chevrolet HHR.” The auxiliary information acts as a different kind of labeling.

Fine-Tuning – Often applies to language models. The goal is to start with a pre-trained model such as GPT-3 that understands basic English and knows general facts from Wikipedia, and add specialized information such as the human resources policies of a company.

A not-so-good strategy is to prepare new specialized training data then completely retrain the base model, changing all of the billions of parameters/weights. A better strategy is to prepare new specialized training data and then train in a way that changes only some of the last layers parameters/weights, or even better, use the specialized training data to create a new layer (or a few layers) that can be appended to the base model. This last approach allows you to have just one base model, and many relatively small modules for specialized tasks.

Fine-tuning is one of the most active areas of AI research. A relatively new development is called “continual agent learning”, where a software agent/module that somehow learns-to-learn over its lifetime, such that it can eventually learn very quickly from its own experience.

---

I can fine-tune a machine learning model. But when I played guitar in a band, I had to use a tuning device instead of manual tuning.

Left: This amazing animation is from animusic.com and it shows an imaginary multi-stringed six-in-one instrument being played by automated wooden fingers. Wonderful! This screenshot doesn’t do it justice.

Right: Here’s an old photo of me playing with my band at the Disneyland Employees Summer Party, called the Banana Ball (because it originated with Jungle Cruise employees). I’m on the far right playing bass guitar. On the far left on rhythm guitar is my good friend Paul Ruiz. In the back on drums is my good friend Jeff Rhoads. The singer is George Trullinger, who went on to a long successful career as a professional singer in Las Vegas.

---

Time Series Regression Using a Standard Neural Network With C#

Time series regression (TSR) problems are very challenging. There are dozens of techniques — and the fact that there are so many techniques for TSR indicates that there’s no single best approach.

There’s been quite a bit of recent research activity that looks at attacking TSR problems using modern neural techniques, specifically systems that use transformer architecture. I decided to revisit one of my preferred techniques, which is to use a standard neural network. The ideas are a bit complicated and are best explained by a concrete example.

I used the Airline Passengers dataset. The source data looks like:

"1949-01";112
"1949-02";118
"1949-03";132
"1949-04";129
"1949-05";121
"1949-06";135
"1949-07";148
. . . 
"1960-12";432

There are 144 lines, with dates from January 1949 to December 1960. The values are number of airline passengers, in thousands, so the first item means there were a total of 112,000 airline passengers in January 1949 (not very many back then). When graphed, the data looks like this (where all passenger counts have been divided by 100):

---

Note: The Airline Passenger dataset originally appeared on page 531 of the first edition of the famous book “Time Series Analysis: Forecasting and Control” (1970) by G. Box and G. Jenkins.

---

I preprocessed the raw data to create a text file of sliding window values that looks like:

1.12, 1.18, 1.32, 1.29, 1.21
1.18, 1.32, 1.29, 1.21, 1.35
1.32, 1.29, 1.21, 1.35, 1.48
1.29, 1.21, 1.35, 1.48, 1.48
. . .
6.06, 5.08, 4.61, 3.90, 4.32

Each consecutive set of four values (a “window”) will be used to predict the next value. So the first input is (1.12, 1.18, 1.32, 1.29) and the value to predict is 1.21. Because of the offset, there are 140 training items.

I implemented a 4-12-1 neural network with tanh() hidden node activation and identity() output node activation. I trained the network using 10,000 iterations, with a learning rate of 0.01, and a batch size of 1. Interestingly, I consistently got better results using a batch size of 1, instead of the more usual size of 10 or something similar. In retrospect, this makes perfect sense.

The trained neural network model predicts with 0.8500 accuracy (119 out of 140 correct), where a correct prediction is defined to be one that is within 10% of the true value.

I implemented a Forecast() method that predicts beyond the source data. I started with the last set of four data points: (5.08, 4.61, 3.90, 4.32) and used them to predict for January 1961, and got 4.52. Then I used that prediction to create the next set of four input values: (4.61, 3.90, 4.32, 4.52) and used them to predict for February 1961. And so on.

I graphed the predicted passenger counts and the forecast counts:

The results look good, at least for a relatively short forecast. Much fun!

---

I stumbled across some wonderfully creative short videos on YouTube by a guy named Hey Anglomangler. Creepy and fascinating visions of a bizarre future or alternate universe. The videos look like they were created using some sort of AI tool.

---

Demo code. Replace “lt” (less than), “gt”, “lte”, “gte”, “and” with Boolean operator symbols.

using System;
using System.IO;
using System.Collections.Generic;

// data from Box, G., Jenkins, G., and Reinsel, G. (1976)
// "Time Series Analysis, Forecasting and Control."

namespace NeuralNetworkTimeSeries
{
  internal class NeuralTimeSeriesProgram
  {
    static void Main(string[] args)
    {
      Console.WriteLine("\nBegin neural network" +
        " times series demo");
      Console.WriteLine("Goal is to predict airline " +
        "passengers over time ");
      Console.WriteLine("Data from January 1949 to" +
        " December 1960 ");

      Console.WriteLine("\nLoading windowed and" +
        " normalized data from file ");

      string dataFile =
        "..\\..\\..\\Data\\airline_all.txt";
 
      double[][] dataX = Utils.MatLoad(dataFile,
        new int[] { 0, 1, 2, 3 }, ',', "#");
      double[] dataY =
        Utils.MatToVec(Utils.MatLoad(dataFile,
        new int[] { 4 }, ',', "#"));

      Console.WriteLine("\nFirst three X data: ");
      for (int i = 0; i "lt" 3; ++i)
        Utils.VecShow(dataX[i], 2, 6, 10, true);

      Console.WriteLine("\nFirst three target Y: ");
      for (int i = 0; i "lt" 3; ++i)
        Console.WriteLine(dataY[i].ToString("F2"));

      int numInput = 4; // number predictors
      int numHidden = 12;
      int numOutput = 1; // regression

      Console.WriteLine("\nCreating a " + numInput + "-" +
        numHidden + "-" + numOutput + " neural network");
      NeuralNetwork nn = new NeuralNetwork(numInput,
        numHidden, numOutput, 0);

      int maxEpochs = 10000;
      double lrnRate = 0.01;
      int batSize = 1;
      Console.WriteLine("\nSetting maxEpochs = " +
        maxEpochs);
      Console.WriteLine("Setting learnRate = " +
        lrnRate.ToString("F2"));
      Console.WriteLine("Setting batch size = " +
        batSize);

      Console.WriteLine("\nStarting training");
      nn.Train(dataX, dataY, lrnRate, batSize, maxEpochs);
      Console.WriteLine("Done");

      double[] weights = nn.GetWeights();
      Console.WriteLine("\nFinal neural network model" +
        " weights and biases:\n");
      Utils.VecShow(weights, 2, 7, 10, true);

      double acc = nn.Accuracy(dataX, dataY, 0.10);
      Console.WriteLine("\nModel accuracy (within 10%" +
        " actual) = " + acc.ToString("F4"));

      Console.WriteLine("\nForecasting ahead 12 steps ");
      double[] forecast = nn.Forecast(new double[] { 5.08,
        4.61, 3.90, 4.32 }, 12);
      Utils.VecShow(forecast, 2, 6, 12, true);
   
      Console.WriteLine("\nEnd time series demo ");
      Console.ReadLine();

    } // Main
  } // Program

  public class NeuralNetwork
  {
    private int ni; // number input nodes
    private int nh;
    private int no;

    private double[] iNodes;
    private double[][] ihWeights; // input-hidden
    private double[] hBiases;
    private double[] hNodes;

    private double[][] hoWeights; // hidden-output
    private double[] oBiases;
    private double[] oNodes;  // single val as array

    // gradients
    private double[][] ihGrads;
    private double[] hbGrads;
    private double[][] hoGrads;
    private double[] obGrads;

    private Random rnd;

    // ------------------------------------------------------

    public NeuralNetwork(int numIn, int numHid,
      int numOut, int seed)
    {
      this.ni = numIn;
      this.nh = numHid;
      this.no = numOut;  // 1 for regression

      this.iNodes = new double[numIn];

      this.ihWeights = MatCreate(numIn, numHid);
      this.hBiases = new double[numHid];
      this.hNodes = new double[numHid];

      this.hoWeights = MatCreate(numHid, numOut);
      this.oBiases = new double[numOut];  // [1]
      this.oNodes = new double[numOut];  // [1]

      this.ihGrads = MatCreate(numIn, numHid);
      this.hbGrads = new double[numHid];
      this.hoGrads = MatCreate(numHid, numOut);
      this.obGrads = new double[numOut];

      this.rnd = new Random(seed);
      this.InitWeights(); // all weights and biases
    } // ctor

    // ------------------------------------------------------

    private static double[][] MatCreate(int rows,
      int cols)
    {
      // helper for ctor
      double[][] result = new double[rows][];
      for (int i = 0; i "lt" rows; ++i)
        result[i] = new double[cols];
      return result;
    }

    // ------------------------------------------------------

    private void InitWeights() // helper for ctor
    {
      // weights and biases to small random values
      double lo = -0.01; double hi = +0.01;
      int numWts = (this.ni * this.nh) +
        (this.nh * this.no) + this.nh + this.no;
      double[] initialWeights = new double[numWts];
      for (int i = 0; i "lt" initialWeights.Length; ++i)
        initialWeights[i] =
          (hi - lo) * rnd.NextDouble() + lo;
      this.SetWeights(initialWeights);
    }

    // ------------------------------------------------------

    public void SetWeights(double[] wts)
    {
      // copy serialized weights and biases in wts[] 
      // to ih weights, ih biases, ho weights, ho biases
      int numWts = (this.ni * this.nh) +
        (this.nh * this.no) + this.nh + this.no;
      if (wts.Length != numWts)
        throw new Exception("Bad array in SetWeights");

      int k = 0; // points into wts param

      for (int i = 0; i "lt" this.ni; ++i)
        for (int j = 0; j "lt" this.nh; ++j)
          this.ihWeights[i][j] = wts[k++];
      for (int i = 0; i "lt" this.nh; ++i)
        this.hBiases[i] = wts[k++];
      for (int i = 0; i "lt" this.nh; ++i)
        for (int j = 0; j "lt" this.no; ++j)
          this.hoWeights[i][j] = wts[k++];
      for (int i = 0; i "lt" this.no; ++i)
        this.oBiases[i] = wts[k++];
    }

    // ------------------------------------------------------

    public double[] GetWeights()
    {
      int numWts = (this.ni * this.nh) +
        (this.nh * this.no) + this.nh + this.no;
      double[] result = new double[numWts];
      int k = 0;
      for (int i = 0; i "lt" ihWeights.Length; ++i)
        for (int j = 0; j "lt" this.ihWeights[0].Length; ++j)
          result[k++] = this.ihWeights[i][j];
      for (int i = 0; i "lt" this.hBiases.Length; ++i)
        result[k++] = this.hBiases[i];
      for (int i = 0; i "lt" this.hoWeights.Length; ++i)
        for (int j = 0; j "lt" this.hoWeights[0].Length; ++j)
          result[k++] = this.hoWeights[i][j];
      for (int i = 0; i "lt" this.oBiases.Length; ++i)
        result[k++] = this.oBiases[i];
      return result;
    }

    // ------------------------------------------------------

    public double ComputeOutput(double[] x)
    {
      double[] hSums = new double[this.nh]; // scratch 
      double[] oSums = new double[this.no]; // out sums

      for (int i = 0; i "lt" x.Length; ++i)
        this.iNodes[i] = x[i];
      // note: no need to copy x-values unless
      // you implement a ToString.
      // more efficient to simply use the X[] directly.

      // 1. compute i-h sum of weights * inputs
      for (int j = 0; j "lt" this.nh; ++j)
        for (int i = 0; i "lt" this.ni; ++i)
          hSums[j] += this.iNodes[i] *
            this.ihWeights[i][j]; // note +=

      // 2. add biases to hidden sums
      for (int i = 0; i "lt" this.nh; ++i)
        hSums[i] += this.hBiases[i];

      // 3. apply hidden activation
      for (int i = 0; i "lt" this.nh; ++i)
        this.hNodes[i] = HyperTan(hSums[i]);

      // 4. compute h-o sum of wts * hOutputs
      for (int j = 0; j "lt" this.no; ++j)
        for (int i = 0; i "lt" this.nh; ++i)
          oSums[j] += this.hNodes[i] *
            this.hoWeights[i][j];  // [1]

      // 5. add biases to output sums
      for (int i = 0; i "lt" this.no; ++i)
        oSums[i] += this.oBiases[i];

      // 6. apply output activation
      for (int i = 0; i "lt" this.no; ++i)
        this.oNodes[i] = Identity(oSums[i]);

      return this.oNodes[0];  // single value
    }

    // ------------------------------------------------------

    private static double HyperTan(double x)
    {
      if (x "lt" -10.0) return -1.0;
      else if (x "gt" 10.0) return 1.0;
      else return Math.Tanh(x);
    }

    // ------------------------------------------------------

    private static double Identity(double x)
    {
      return x;
    }

    // ------------------------------------------------------

    private void ZeroOutGrads()
    {
      for (int i = 0; i "lt" this.ni; ++i)
        for (int j = 0; j "lt" this.nh; ++j)
          this.ihGrads[i][j] = 0.0;

      for (int j = 0; j "lt" this.nh; ++j)
        this.hbGrads[j] = 0.0;

      for (int j = 0; j "lt" this.nh; ++j)
        for (int k = 0; k "lt" this.no; ++k)
          this.hoGrads[j][k] = 0.0;

      for (int k = 0; k "lt" this.no; ++k)
        this.obGrads[k] = 0.0;
    }  // ZeroOutGrads()

    // ------------------------------------------------------

    private void AccumGrads(double y)
    {
      double[] oSignals = new double[this.no];
      double[] hSignals = new double[this.nh];

      // 1. compute output node scratch signals 
      for (int k = 0; k "lt" this.no; ++k)
        oSignals[k] = 1 * (this.oNodes[k] - y);

      // 2. accum hidden-to-output gradients 
      for (int j = 0; j "lt" this.nh; ++j)
        for (int k = 0; k "lt" this.no; ++k)
          hoGrads[j][k] +=
           oSignals[k] * this.hNodes[j];

      // 3. accum output node bias gradients
      for (int k = 0; k "lt" this.no; ++k)
        obGrads[k] +=
         oSignals[k] * 1.0;  // 1.0 dummy 

      // 4. compute hidden node signals
      for (int j = 0; j "lt" this.nh; ++j)
      {
        double sum = 0.0;
        for (int k = 0; k "lt" this.no; ++k)
          sum += oSignals[k] * this.hoWeights[j][k];

        double derivative =
          (1 - this.hNodes[j]) *
          (1 + this.hNodes[j]);  // assumes tanh
        hSignals[j] = derivative * sum;
      }

      // 5. accum input-to-hidden gradients
      for (int i = 0; i "lt" this.ni; ++i)
        for (int j = 0; j "lt" this.nh; ++j)
          this.ihGrads[i][j] +=
            hSignals[j] * this.iNodes[i];

      // 6. accum hidden node bias gradients
      for (int j = 0; j "lt" this.nh; ++j)
        this.hbGrads[j] +=
          hSignals[j] * 1.0;  // 1.0 dummy
    } // AccumGrads

    // ------------------------------------------------------

    private void UpdateWeights(double lrnRate)
    {
      // assumes all gradients computed
      // 1. update input-to-hidden weights
      for (int i = 0; i "lt" this.ni; ++i)
      {
        for (int j = 0; j "lt" this.nh; ++j)
        {
          double delta = -1.0 * lrnRate *
            this.ihGrads[i][j];
          this.ihWeights[i][j] += delta;
        }
      }

      // 2. update hidden node biases
      for (int j = 0; j "lt" this.nh; ++j)
      {
        double delta = -1.0 * lrnRate *
          this.hbGrads[j];
        this.hBiases[j] += delta;
      }

      // 3. update hidden-to-output weights
      for (int j = 0; j "lt" this.nh; ++j)
      {
        for (int k = 0; k "lt" this.no; ++k)
        {
          double delta = -1.0 * lrnRate *
            this.hoGrads[j][k];
          this.hoWeights[j][k] += delta;
        }
      }

      // 4. update output node biases
      for (int k = 0; k "lt" this.no; ++k)
      {
        double delta = -1.0 * lrnRate *
          this.obGrads[k];
        this.oBiases[k] += delta;
      }
    } // UpdateWeights()

    // ------------------------------------------------------

    public void Train(double[][] trainX, double[] trainY,
      double lrnRate, int batSize, int maxEpochs)
    {
      int n = trainX.Length; 
      int batchesPerEpoch = n / batSize; 
      int freq = maxEpochs / 5;  // to show progress

      int[] indices = new int[n];
      for (int i = 0; i "lt" n; ++i)
        indices[i] = i;

      // ----------------------------------------------------
      //
      // for epoch = 0; epoch "lt" maxEpochs; ++epoch
      //   shuffle indices
      //   for batch = 0; batch "lt" bpe; ++batch
      //     for item = 0; item "lt" bs; ++item
      //       compute output
      //       accum grads
      //     end-item
      //     update weights
      //     zero-out grads
      //   end-batches
      // end-epochs
      //
      // ----------------------------------------------------

      for (int epoch = 0; epoch "lt" maxEpochs; ++epoch)
      {
        Shuffle(indices);
        int ptr = 0;  // points into indices
        for (int batIdx = 0; batIdx "lt" batchesPerEpoch;
          ++batIdx)
        {
          for (int i = 0; i "lt" batSize; ++i)
          {
            int ii = indices[ptr++];  // compute output
            double[] x = trainX[ii];
            double y = trainY[ii];
            this.ComputeOutput(x);  // into this.oNoodes
            this.AccumGrads(y);
          }
          this.UpdateWeights(lrnRate);
          this.ZeroOutGrads(); // prep for next batch
        } // batches

        if (epoch % freq == 0)  // progress 
        {
          double mse = this.Error(trainX, trainY);
          double acc = this.Accuracy(trainX, trainY, 0.10);

          string s1 = "epoch: " + 
            epoch.ToString().PadLeft(4);
          string s2 = "  MSE = " + 
            mse.ToString("F4");
          string s3 = "  acc (10%) = " + 
            acc.ToString("F4");
          Console.WriteLine(s1 + s2 + s3);
        }
      } // epoch
    } // Train

    // ------------------------------------------------------

    private void Shuffle(int[] sequence)
    {
      for (int i = 0; i "lt" sequence.Length; ++i)
      {
        int r = this.rnd.Next(i, sequence.Length);
        int tmp = sequence[r];
        sequence[r] = sequence[i];
        sequence[i] = tmp;
        //sequence[i] = i; // for testing
      }
    } // Shuffle

    // ------------------------------------------------------

    public double Error(double[][] trainX, double[] trainY)
    {
      // MSE
      int n = trainX.Length;
      double sumSquaredError = 0.0;
      for (int i = 0; i "lt" n; ++i)
      {
        double predY = this.ComputeOutput(trainX[i]);
        double actualY = trainY[i];
        sumSquaredError += (predY - actualY) *
          (predY - actualY);
      }
      return sumSquaredError / n;
    } // Error

    // ------------------------------------------------------

    public double Accuracy(double[][] dataX,
      double[] dataY, double pctClose)
    {
      // percentage correct using winner-takes all
      int n = dataX.Length;
      int nCorrect = 0;
      int nWrong = 0;
      for (int i = 0; i "lt" n; ++i)
      {
        double predY = this.ComputeOutput(dataX[i]);
        double actualY = dataY[i];

        //Console.WriteLine("actual Y = " + 
        //  actualY.ToString("F2"));
        //Console.WriteLine("pred   Y = " + 
        //  predY.ToString("F2"));

        if (Math.Abs(predY - actualY) "lt"
          Math.Abs(pctClose * actualY))
          ++nCorrect;
        else
          ++nWrong;
      }
      return (nCorrect * 1.0) / (nCorrect + nWrong);
    }

    public double[] Forecast(double[] start, int steps)
    {
      double[] result = new double[steps];
      int n = start.Length;
      double[] currInput = new double[start.Length];
      for (int i = 0; i "lt" n; ++i)
        currInput[i] = start[i];

      for (int step = 0; step "lt" steps; ++step)
      {
        double pred = this.ComputeOutput(currInput);
        //Console.WriteLine(pred.ToString("F2"));
        result[step] = pred;

        // create new input
        for (int i = 0; i "lt" n-1; ++i)
          currInput[i] = currInput[i + 1];
 
        currInput[n - 1] = pred;
      }
      return result;
    }

    // ------------------------------------------------------

    public void SaveWeights(string fn)
    {
      FileStream ofs = new FileStream(fn, FileMode.Create);
      StreamWriter sw = new StreamWriter(ofs);

      double[] wts = this.GetWeights();
      for (int i = 0; i "lt" wts.Length; ++i)
        sw.WriteLine(wts[i].ToString("F8"));  // one per line
      sw.Close();
      ofs.Close();
    }

    public void LoadWeights(string fn)
    {
      FileStream ifs = new FileStream(fn, FileMode.Open);
      StreamReader sr = new StreamReader(ifs);
      List"lt"double"gt" listWts = new List"lt"double"gt"();
      string line = "";  // one wt per line
      while ((line = sr.ReadLine()) != null)
      {
        // if (line.StartsWith(comment) == true)
        //   continue;
        listWts.Add(double.Parse(line));
      }
      sr.Close();
      ifs.Close();

      double[] wts = listWts.ToArray();
      this.SetWeights(wts);
    }

    // ------------------------------------------------------

  } // NeuralNetwork class


  public class Utils
  {
    // ------------------------------------------------------

    public static double[][] MatLoad(string fn,
     int[] usecols, char sep, string comment)
    {
      List"lt"double[]"gt" result = new List"lt"double[]"gt"();
      string line = "";
      FileStream ifs = new FileStream(fn, FileMode.Open);
      StreamReader sr = new StreamReader(ifs);
      while ((line = sr.ReadLine()) != null)
      {
        if (line.StartsWith(comment) == true)
          continue;
        string[] tokens = line.Split(sep);
        List"lt"double"gt" lst = new List"lt"double"gt"();
        for (int j = 0; j "lt" usecols.Length; ++j)
          lst.Add(double.Parse(tokens[usecols[j]]));
        double[] row = lst.ToArray();
        result.Add(row);
      }
      sr.Close();
      ifs.Close();
      return result.ToArray();
    }

    // ------------------------------------------------------

    public static double[] MatToVec(double[][] m)
    {
      int rows = m.Length;
      int cols = m[0].Length;
      double[] result = new double[rows * cols];
      int k = 0;
      for (int i = 0; i "lt" rows; ++i)
        for (int j = 0; j "lt" cols; ++j)
          result[k++] = m[i][j];

      return result;
    }

    // ------------------------------------------------------

    public static void MatShow(double[][] m,
      int dec, int wid)
    {
      for (int i = 0; i "lt" m.Length; ++i)
      {
        for (int j = 0; j "lt" m[0].Length; ++j)
        {
          double v = m[i][j];
          if (Math.Abs(v) "lt" 1.0e-8) v = 0.0; // hack
          Console.Write(v.ToString("F" +
            dec).PadLeft(wid));
        }
        Console.WriteLine("");
      }
    }

    // ------------------------------------------------------

    public static void VecShow(int[] vec, int wid)
    {
      for (int i = 0; i "lt" vec.Length; ++i)
        Console.Write(vec[i].ToString().PadLeft(wid));
      Console.WriteLine("");
    }

    // ------------------------------------------------------

    public static void VecShow(double[] vec,
      int dec, int wid, int lineLen, bool newLine)
    {
      for (int i = 0; i "lt" vec.Length; ++i)
      {
        if (i != 0 "and" i % lineLen == 0)
          Console.WriteLine("");
        double x = vec[i];
        if (Math.Abs(x) "lt" 1.0e-8) x = 0.0;
        Console.Write(x.ToString("F" +
          dec).PadLeft(wid));
      }
      if (newLine == true)
        Console.WriteLine("");
    }

  } // Utils

} // ns

Demo data:

# airline_all.txt
# 4 item window
#
1.12, 1.18, 1.32, 1.29, 1.21
1.18, 1.32, 1.29, 1.21, 1.35
1.32, 1.29, 1.21, 1.35, 1.48
1.29, 1.21, 1.35, 1.48, 1.48
1.21, 1.35, 1.48, 1.48, 1.36
1.35, 1.48, 1.48, 1.36, 1.19
1.48, 1.48, 1.36, 1.19, 1.04
1.48, 1.36, 1.19, 1.04, 1.18
1.36, 1.19, 1.04, 1.18, 1.15
1.19, 1.04, 1.18, 1.15, 1.26
1.04, 1.18, 1.15, 1.26, 1.41
1.18, 1.15, 1.26, 1.41, 1.35
1.15, 1.26, 1.41, 1.35, 1.25
1.26, 1.41, 1.35, 1.25, 1.49
1.41, 1.35, 1.25, 1.49, 1.70
1.35, 1.25, 1.49, 1.70, 1.70
1.25, 1.49, 1.70, 1.70, 1.58
1.49, 1.70, 1.70, 1.58, 1.33
1.70, 1.70, 1.58, 1.33, 1.14
1.70, 1.58, 1.33, 1.14, 1.40
1.58, 1.33, 1.14, 1.40, 1.45
1.33, 1.14, 1.40, 1.45, 1.50
1.14, 1.40, 1.45, 1.50, 1.78
1.40, 1.45, 1.50, 1.78, 1.63
1.45, 1.50, 1.78, 1.63, 1.72
1.50, 1.78, 1.63, 1.72, 1.78
1.78, 1.63, 1.72, 1.78, 1.99
1.63, 1.72, 1.78, 1.99, 1.99
1.72, 1.78, 1.99, 1.99, 1.84
1.78, 1.99, 1.99, 1.84, 1.62
1.99, 1.99, 1.84, 1.62, 1.46
1.99, 1.84, 1.62, 1.46, 1.66
1.84, 1.62, 1.46, 1.66, 1.71
1.62, 1.46, 1.66, 1.71, 1.80
1.46, 1.66, 1.71, 1.80, 1.93
1.66, 1.71, 1.80, 1.93, 1.81
1.71, 1.80, 1.93, 1.81, 1.83
1.80, 1.93, 1.81, 1.83, 2.18
1.93, 1.81, 1.83, 2.18, 2.30
1.81, 1.83, 2.18, 2.30, 2.42
1.83, 2.18, 2.30, 2.42, 2.09
2.18, 2.30, 2.42, 2.09, 1.91
2.30, 2.42, 2.09, 1.91, 1.72
2.42, 2.09, 1.91, 1.72, 1.94
2.09, 1.91, 1.72, 1.94, 1.96
1.91, 1.72, 1.94, 1.96, 1.96
1.72, 1.94, 1.96, 1.96, 2.36
1.94, 1.96, 1.96, 2.36, 2.35
1.96, 1.96, 2.36, 2.35, 2.29
1.96, 2.36, 2.35, 2.29, 2.43
2.36, 2.35, 2.29, 2.43, 2.64
2.35, 2.29, 2.43, 2.64, 2.72
2.29, 2.43, 2.64, 2.72, 2.37
2.43, 2.64, 2.72, 2.37, 2.11
2.64, 2.72, 2.37, 2.11, 1.80
2.72, 2.37, 2.11, 1.80, 2.01
2.37, 2.11, 1.80, 2.01, 2.04
2.11, 1.80, 2.01, 2.04, 1.88
1.80, 2.01, 2.04, 1.88, 2.35
2.01, 2.04, 1.88, 2.35, 2.27
2.04, 1.88, 2.35, 2.27, 2.34
1.88, 2.35, 2.27, 2.34, 2.64
2.35, 2.27, 2.34, 2.64, 3.02
2.27, 2.34, 2.64, 3.02, 2.93
2.34, 2.64, 3.02, 2.93, 2.59
2.64, 3.02, 2.93, 2.59, 2.29
3.02, 2.93, 2.59, 2.29, 2.03
2.93, 2.59, 2.29, 2.03, 2.29
2.59, 2.29, 2.03, 2.29, 2.42
2.29, 2.03, 2.29, 2.42, 2.33
2.03, 2.29, 2.42, 2.33, 2.67
2.29, 2.42, 2.33, 2.67, 2.69
2.42, 2.33, 2.67, 2.69, 2.70
2.33, 2.67, 2.69, 2.70, 3.15
2.67, 2.69, 2.70, 3.15, 3.64
2.69, 2.70, 3.15, 3.64, 3.47
2.70, 3.15, 3.64, 3.47, 3.12
3.15, 3.64, 3.47, 3.12, 2.74
3.64, 3.47, 3.12, 2.74, 2.37
3.47, 3.12, 2.74, 2.37, 2.78
3.12, 2.74, 2.37, 2.78, 2.84
2.74, 2.37, 2.78, 2.84, 2.77
2.37, 2.78, 2.84, 2.77, 3.17
2.78, 2.84, 2.77, 3.17, 3.13
2.84, 2.77, 3.17, 3.13, 3.18
2.77, 3.17, 3.13, 3.18, 3.74
3.17, 3.13, 3.18, 3.74, 4.13
3.13, 3.18, 3.74, 4.13, 4.05
3.18, 3.74, 4.13, 4.05, 3.55
3.74, 4.13, 4.05, 3.55, 3.06
4.13, 4.05, 3.55, 3.06, 2.71
4.05, 3.55, 3.06, 2.71, 3.06
3.55, 3.06, 2.71, 3.06, 3.15
3.06, 2.71, 3.06, 3.15, 3.01
2.71, 3.06, 3.15, 3.01, 3.56
3.06, 3.15, 3.01, 3.56, 3.48
3.15, 3.01, 3.56, 3.48, 3.55
3.01, 3.56, 3.48, 3.55, 4.22
3.56, 3.48, 3.55, 4.22, 4.65
3.48, 3.55, 4.22, 4.65, 4.67
3.55, 4.22, 4.65, 4.67, 4.04
4.22, 4.65, 4.67, 4.04, 3.47
4.65, 4.67, 4.04, 3.47, 3.05
4.67, 4.04, 3.47, 3.05, 3.36
4.04, 3.47, 3.05, 3.36, 3.40
3.47, 3.05, 3.36, 3.40, 3.18
3.05, 3.36, 3.40, 3.18, 3.62
3.36, 3.40, 3.18, 3.62, 3.48
3.40, 3.18, 3.62, 3.48, 3.63
3.18, 3.62, 3.48, 3.63, 4.35
3.62, 3.48, 3.63, 4.35, 4.91
3.48, 3.63, 4.35, 4.91, 5.05
3.63, 4.35, 4.91, 5.05, 4.04
4.35, 4.91, 5.05, 4.04, 3.59
4.91, 5.05, 4.04, 3.59, 3.10
5.05, 4.04, 3.59, 3.10, 3.37
4.04, 3.59, 3.10, 3.37, 3.60
3.59, 3.10, 3.37, 3.60, 3.42
3.10, 3.37, 3.60, 3.42, 4.06
3.37, 3.60, 3.42, 4.06, 3.96
3.60, 3.42, 4.06, 3.96, 4.20
3.42, 4.06, 3.96, 4.20, 4.72
4.06, 3.96, 4.20, 4.72, 5.48
3.96, 4.20, 4.72, 5.48, 5.59
4.20, 4.72, 5.48, 5.59, 4.63
4.72, 5.48, 5.59, 4.63, 4.07
5.48, 5.59, 4.63, 4.07, 3.62
5.59, 4.63, 4.07, 3.62, 4.05
4.63, 4.07, 3.62, 4.05, 4.17
4.07, 3.62, 4.05, 4.17, 3.91
3.62, 4.05, 4.17, 3.91, 4.19
4.05, 4.17, 3.91, 4.19, 4.61
4.17, 3.91, 4.19, 4.61, 4.72
3.91, 4.19, 4.61, 4.72, 5.35
4.19, 4.61, 4.72, 5.35, 6.22
4.61, 4.72, 5.35, 6.22, 6.06
4.72, 5.35, 6.22, 6.06, 5.08
5.35, 6.22, 6.06, 5.08, 4.61
6.22, 6.06, 5.08, 4.61, 3.90
6.06, 5.08, 4.61, 3.90, 4.32

Updating My JavaScript Regression Neural Network

Once or twice a year, I revisit my from-scratch JavaScript implementations of a neural network. The system has enough complexity that there are dozens of ideas that can be explored.

My latest regression version makes many small changes from previous versions. The primary change was that I refactored the train() method from a single, very large method, to one that calls three helper functions — zeroOutGrads(), accumGrads(y), updateWeights(lrnRate). This change required me to store the hidden node and output node gradients as class matrices and vectors rather than as objects local to the train() method.

For my demo program, I used one of my standard synthetic datasets. The goal is to predict a person’s income from sex, age, State, and political leaning. The 240-item tab-delimited raw data looks like:

F   24   michigan   29500.00   liberal
M   39   oklahoma   51200.00   moderate
F   63   nebraska   75800.00   conservative
M   36   michigan   44500.00   moderate
F   27   nebraska   28600.00   liberal
. . .

I encoded sex as M = -1, F = 1, and State as Michigan = 100, Nebraska = 010, Oklahoma = 001, and political leaning as conservative = 100, moderate = 010, liberal = 001. I normalized the numeric data. I divided age values by 100, and divided the target income values by 100,000. The resulting encoded and normalized comma-delimited data looks like:

 1, 0.24, 1, 0, 0, 0.2950, 0, 0, 1
-1, 0.39, 0, 0, 1, 0.5120, 0, 1, 0
 1, 0.63, 0, 1, 0, 0.7580, 1, 0, 0
-1, 0.36, 1, 0, 0, 0.4450, 0, 1, 0
 1, 0.27, 0, 1, 0, 0.2860, 0, 0, 1
. . .

I split the data into a 200-item set of training data and a 40-item set of test data.

My neural architecture was 8-25-3 with tanh() hidden node activation and softmax() output node activation. For training I used a batch size of 10, a learning rate of 0.001, and 1,000 epochs.

The resulting model scored 0.9100 accuracy on the training data (182 out of 200 correct) and 0.9500 accuracy on the test data (38 out of 40 correct) where a correct income prediction is one that is within 10% of the true target income. These results are similar to those achieved by a PyTorch neural network and a LightGBM tree-based system.

Good fun!

---

Different versions of a computer program can have a very different look and feel. Different book cover art has a big influence on my perception of a book. Here are covers for three versions of Tolkien’s “The Two Towers”, the second book in the Lord of the Rings trilogy. Left: By artist Jack Gaughan (1930-1985), for the 1965 accidentally unauthorized edition. Center: By artist Barbara Remington (1929-2020), for the first U.S. authorized paperback 1965 edition. Right: Art by Tolkien (1892-1973), adapted for the 1973 edition.

---

Demo code. Very long! Replace “lt” (less than), “gt”, “lte”, “gte”, “and” with Boolean operator symbols.

// people_income.js
// node.js  ES6

// NN regression
// tanh, identity activation, MSE loss

let U = require("..\\Utils\\utilities_lib.js")
let FS = require("fs")

// ----------------------------------------------------------

class NeuralNet
{
  constructor(numInput, numHidden, numOutput, seed)
  {
    this.rnd = new U.Erratic(seed);  // pseudo-random

    this.ni = numInput; 
    this.nh = numHidden;
    this.no = numOutput;

    this.iNodes = U.vecMake(this.ni, 0.0);
    this.hNodes = U.vecMake(this.nh, 0.0);
    this.oNodes = U.vecMake(this.no, 0.0);

    this.ihWeights = U.matMake(this.ni, this.nh, 0.0);
    this.hoWeights = U.matMake(this.nh, this.no, 0.0);

    this.hBiases = U.vecMake(this.nh, 0.0);
    this.oBiases = U.vecMake(this.no, 0.0); // [1] 

    this.ihGrads = U.matMake(this.ni, this.nh, 0.0);
    this.hbGrads = U.vecMake(this.nh, 0.0);
    this.hoGrads = U.matMake(this.nh, this.no, 0.0);
    this.obGrads = U.vecMake(this.no, 0.0);

    this.initWeights();
  }

  initWeights()
  {
    let lo = -0.10;
    let hi = 0.10;
    for (let i = 0; i "lt" this.ni; ++i) {
      for (let j = 0; j "lt" this.nh; ++j) {
        this.ihWeights[i][j] = (hi - lo) * 
          this.rnd.next() + lo;
      }
    }

    for (let j = 0; j "lt" this.nh; ++j) {
      for (let k = 0; k "lt" this.no; ++k) {
        this.hoWeights[j][k] = (hi - lo) * 
          this.rnd.next() + lo;
      }
    }
  } 

  computeOutput(X)
  {
    let hSums = U.vecMake(this.nh, 0.0);
    let oSums = U.vecMake(this.no, 0.0);
    
    this.iNodes = X;

    for (let j = 0; j "lt" this.nh; ++j) {
      for (let i = 0; i "lt" this.ni; ++i) {
        hSums[j] += this.iNodes[i] * this.ihWeights[i][j];
      }
      hSums[j] += this.hBiases[j];
      this.hNodes[j] = U.hyperTan(hSums[j]);
    }

    for (let k = 0; k "lt" this.no; ++k) {
      for (let j = 0; j "lt" this.nh; ++j) {
        oSums[k] += this.hNodes[j] * this.hoWeights[j][k];
      }
      oSums[k] += this.oBiases[k];
    }

    // this.oNodes = U.softmax(oSums);  // multi-class
    // this.oNodes = U.identity(oSums); // regression
    for (let k = 0; k "lt" this.no; ++k) {
      this.oNodes[k] = oSums[k];
    }

    let result = [];  // or use U.vecMake
    for (let k = 0; k "lt" this.no; ++k) {
      result[k] = this.oNodes[k];
    }
    return result[0];  // a single scalar value
  } // computeOutput()

  setWeights(wts)
  {
    // order: ihWts, hBiases, hoWts, oBiases
    let p = 0;

    for (let i = 0; i "lt" this.ni; ++i) {
      for (let j = 0; j "lt" this.nh; ++j) {
        this.ihWeights[i][j] = wts[p++];
      }
    }

    for (let j = 0; j "lt" this.nh; ++j) {
      this.hBiases[j] = wts[p++];
    }

    for (let j = 0; j "lt" this.nh; ++j) {
      for (let k = 0; k "lt" this.no; ++k) {
        this.hoWeights[j][k] = wts[p++];
      }
    }

    for (let k = 0; k "lt" this.no; ++k) {
      this.oBiases[k] = wts[p++];
    }
  } // setWeights()

  getWeights()
  {
    // order: ihWts, hBiases, hoWts, oBiases
    let numWts = (this.ni * this.nh) + this.nh +
      (this.nh * this.no) + this.no;
    let result = U.vecMake(numWts, 0.0);
    let p = 0;
    for (let i = 0; i "lt" this.ni; ++i) {
      for (let j = 0; j "lt" this.nh; ++j) {
        result[p++] = this.ihWeights[i][j];
      }
    }

    for (let j = 0; j "lt" this.nh; ++j) {
      result[p++] = this.hBiases[j];
    }

    for (let j = 0; j "lt" this.nh; ++j) {
      for (let k = 0; k "lt" this.no; ++k) {
        result[p++] = this.hoWeights[j][k];
      }
    }

    for (let k = 0; k "lt" this.no; ++k) {
      result[p++] = this.oBiases[k];
    }
    return result;
  } // getWeights()

  shuffle(v)
  {
    // Fisher-Yates
    let n = v.length;
    for (let i = 0; i "lt" n; ++i) {
      let r = this.rnd.nextInt(i, n);
      let tmp = v[r];
      v[r] = v[i];
      v[i] = tmp;
    }
  }

  // --------------------------------------------------------
  // helpers for train(): zeroOutGrads(), accumGrads(y),
  //   updateWeights(lrnRate)
  // --------------------------------------------------------

  zeroOutGrads()
  {
    for (let i = 0; i "lt" this.ni; ++i)
      for (let j = 0; j "lt" this.nh; ++j)
        this.ihGrads[i][j] = 0.0;

    for (let j = 0; j "lt" this.nh; ++j)
      this.hbGrads[j] = 0.0;

    for (let j = 0; j "lt" this.nh; ++j)
      for (let k = 0; k "lt" this.no; ++k)
        this.hoGrads[j][k] = 0.0;

    for (let k = 0; k "lt" this.no; ++k)
      this.obGrads[k] = 0.0;
  }

  accumGrads(y)
  {
    // y is target scalar
    let oSignals = U.vecMake(this.no, 0.0);
    let hSignals = U.vecMake(this.nh, 0.0);

    // 1. compute output node scratch signals 
    for (let k = 0; k "lt" this.no; ++k) {
      let derivative = 1.0;  // CEE
      // let derivative =
      //  this.oNodes[k] * (1 - this.oNodes[k]); // MSE
      oSignals[k] = derivative *
        (this.oNodes[k] - y);  // CEE
    }

    // 2. accum hidden-to-output gradients 
    for (let j = 0; j "lt" this.nh; ++j)
      for (let k = 0; k "lt" this.no; ++k)
        this.hoGrads[j][k] += oSignals[k] * this.hNodes[j];

    // 3. accum output node bias gradients
    for (let k = 0; k "lt" this.no; ++k)
      this.obGrads[k] += oSignals[k] * 1.0;  // 1.0 dummy 

    // 4. compute hidden node signals
    for (let j = 0; j "lt" this.nh; ++j) {
      let sum = 0.0;
      for (let k = 0; k "lt" this.no; ++k)
        sum += oSignals[k] * this.hoWeights[j][k];

      let derivative =
        (1 - this.hNodes[j]) *
        (1 + this.hNodes[j]);  // assumes tanh
      hSignals[j] = derivative * sum;
    }

    // 5. accum input-to-hidden gradients
    for (let i = 0; i "lt" this.ni; ++i)
      for (let j = 0; j "lt" this.nh; ++j)
        this.ihGrads[i][j] += hSignals[j] * this.iNodes[i];

    // 6. accum hidden node bias gradients
    for (let j = 0; j "lt" this.nh; ++j)
      this.hbGrads[j] += hSignals[j] * 1.0;  // 1.0 dummy
  } // accumGrads
  
  updateWeights(lrnRate)
  {
    // assumes all gradients computed
    // 1. update input-to-hidden weights
    for (let i = 0; i "lt" this.ni; ++i) {
      for (let j = 0; j "lt" this.nh; ++j) {
        let delta = -1.0 * lrnRate * this.ihGrads[i][j];
        this.ihWeights[i][j] += delta;
      }
    }

    // 2. update hidden node biases
    for (let j = 0; j "lt" this.nh; ++j) {
      let delta = -1.0 * lrnRate * this.hbGrads[j];
      this.hBiases[j] += delta;
    }

    // 3. update hidden-to-output weights
    for (let j = 0; j "lt" this.nh; ++j) {
      for (let k = 0; k "lt" this.no; ++k) {
        let delta = -1.0 * lrnRate * this.hoGrads[j][k];
        this.hoWeights[j][k] += delta;
      }
    }

    // 4. update output node biases
    for (let k = 0; k "lt" this.no; ++k) {
      let delta = -1.0 * lrnRate * this.obGrads[k];
      this.oBiases[k] += delta;
    }
  } // updateWeights()

  // --------------------------------------------------------

  train(trainX, trainY, lrnRate, batSize, maxEpochs)
  {
    let n = trainX.length;  // 200
    let batchesPerEpoch = Math.trunc(n / batSize);  // 20
    let freq = Math.trunc(maxEpochs / 10);  // progress
    let indices = U.arange(n);

    // ----------------------------------------------------
    //
    // n = 200; bs = 10
    // batches per epoch = 200 / 10 = 20

    // for epoch = 0; epoch "lt" maxEpochs; ++epoch
    //   for batch = 0; batch "lt" bpe; ++batch
    //     for item = 0; item "lt" bs; ++item
    //       compute output
    //       accum grads
    //     end-item
    //     update weights
    //     zero-out grads
    //   end-batches
    //   shuffle indices
    // end-epochs
    //
    // ----------------------------------------------------

    for (let epoch = 0; epoch "lt" maxEpochs; ++epoch) {
      this.shuffle(indices);
      let ptr = 0;  // points into indices
      for (let batIdx = 0; batIdx "lt" batchesPerEpoch;
        ++batIdx) // 0, 1, . . 19
      {
        for (let i = 0; i "lt" batSize; ++i) { // 0 . . 9
          let ii = indices[ptr++];  // compute output
          let x = trainX[ii];
          let y = trainY[ii];
          this.computeOutput(x);  // into this.oNodes
          this.accumGrads(y);
        }
        this.updateWeights(lrnRate);
        this.zeroOutGrads(); // prep for next batch
      } // batches

      if (epoch % freq == 0) {
        // let mse = 
        // this.meanSqErr(trainX, trainY).toFixed(4);
        let mcee = 
          this.meanSqErr(trainX, trainY).toFixed(4);
        let acc = this.accuracy(trainX, trainY,
          0.10).toFixed(4);

        let s1 = "epoch: " +
          epoch.toString().padStart(6, ' ');
        let s2 = "   MCEE = " + 
          mcee.toString().padStart(8, ' ');
        let s3 = "   acc = " + acc.toString();

        console.log(s1 + s2 + s3);
      }
    } // epoch
  } // train

  // -------------------------------------------------------- 

  meanSqErr(dataX, dataY)
  {
    // for regression
    let sumSE = 0.0;
    for (let i = 0; i "lt" dataX.length; ++i) {
      let X = dataX[i];
      let Y = dataY[i];  // target income
      let oupt = this.computeOutput(X); 
      sumSE += (Y - oupt) * (Y - oupt); 
    }
    return sumSE / dataX.length;  // consider Root MSE
  } 

  accuracy(dataX, dataY, pctClose)
  {
    let nc = 0; let nw = 0;
    for (let i = 0; i "lt" dataX.length; ++i) { 
      let X = dataX[i];
      let y = dataY[i];  // target income
      let pred = this.computeOutput(X);
      if ( Math.abs(y - pred) "lt" Math.abs(y * pctClose) ) {
        ++nc;
      }
      else {
        ++nw;
      }
    }
    return nc / (nc + nw);
  }

  saveWeights(fn)
  {
    let wts = this.getWeights();
    let n = wts.length;
    let s = "";
    for (let i = 0; i "lt" n-1; ++i) {
      s += wts[i].toString() + ",";
    }
    s += wts[n-1];

    FS.writeFileSync(fn, s);
  }

  loadWeights(fn)
  {
    let n = (this.ni * this.nh) + this.nh +
      (this.nh * this.no) + this.no;
    let wts = U.vecMake(n, 0.0);
    let all = FS.readFileSync(fn, "utf8");
    let strVals = all.split(",");
    let nn = strVals.length;
    if (n != nn) {
      throw("Size error in NeuralNet.loadWeights()");
    }
    for (let i = 0; i "lt" n; ++i) {
      wts[i] = parseFloat(strVals[i]);
    }
    this.setWeights(wts);
  }

} // NeuralNet

// ----------------------------------------------------------

function main()
{
  // process.stdout.write("\033[0m");  // reset
  // process.stdout.write("\x1b[1m" + "\x1b[37m"); // white
  console.log("\nBegin JavaScript NN regression demo ");
  console.log("Predict income from sex, age, State, politics ");
  
  // 1. load data
  // -1  0.29  1 0 0  0.65400  0 0 1
  //  1  0.36  0 0 1  0.58300  1 0 0
  console.log("\nLoading data into memory ");
  let trainX = U.loadTxt(".\\Data\\people_train.txt", ",",
    [0,1,2,3,4,6,7,8], "#");
  let trainY = U.loadTxt(".\\Data\\people_train.txt", ",",
    [5], "#");
  trainY = U.matToVec(trainY);
  let testX = U.loadTxt(".\\Data\\people_test.txt", ",",
    [0,1,2,3,4,6,7,8], "#");
  let testY = U.loadTxt(".\\Data\\people_test.txt", ",",
    [5], "#");
  testY = U.matToVec(testY);

  // 2. create network
  console.log("\nCreating 8-25-1 tanh, identity MSE NN ");
  let seed = 0;
  let nn = new NeuralNet(8, 25, 1, seed);

  // 3. train network
  let lrnRate = 0.001;
  let maxEpochs = 1000;
  console.log("\nSetting learn rate = 0.001 ");
  console.log("Setting bat size = 10 ");
  nn.train(trainX, trainY, lrnRate, 10, maxEpochs);
  console.log("Training complete ");

  // 4. evaluate model
  console.log("\nComputing acc within 0.10 of true ");
  let trainAcc = nn.accuracy(trainX, trainY, 0.10);
  let testAcc = nn.accuracy(testX, testY, 0.10);
  console.log("Accuracy on training data = " +
    trainAcc.toFixed(4).toString()); 
  console.log("Accuracy on test data     = " +
    testAcc.toFixed(4).toString());

  // 5. save trained model
  fn = ".\\Models\\people_income_wts.txt";
  console.log("\nSaving model weights and biases to: ");
  console.log(fn);
  nn.saveWeights(fn);

  // 6. use trained model
  console.log("\nPredict for M 34 Oklahoma moderate ");
  let x = [-1, 0.34, 0,0,1, 0,1,0];
  let predicted = nn.computeOutput(x);
  console.log("\nPredicted income: ");
  console.log(predicted.toFixed(5).toString());

  //process.stdout.write("\033[0m");  // reset
  console.log("\nEnd demo");
}

main()

Code for utility functions:

// utilities_lib.js
// ES6

let FS = require('fs');

// ----------------------------------------------------------

function loadTxt(fn, delimit, usecols, comment) {
  // efficient but mildly complicated
  let all = FS.readFileSync(fn, "utf8");  // giant string
  all = all.trim();  // strip final crlf in file
  let lines = all.split("\n");  // array of lines

  // count number non-comment lines
  let nRows = 0;
  for (let i = 0; i "lt" lines.length; ++i) {
    if (!lines[i].startsWith(comment))
      ++nRows;
  }
  let nCols = usecols.length;
  let result = matMake(nRows, nCols, 0.0); 
 
  let r = 0;  // into lines
  let i = 0;  // into result[][]
  while (r "lt" lines.length) {
    if (lines[r].startsWith(comment)) {
      ++r;  // next row
    }
    else {
      let tokens = lines[r].split(delimit);
      for (let j = 0; j "lt" nCols; ++j) {
        result[i][j] = parseFloat(tokens[usecols[j]]);
      }
      ++r;
      ++i;
    }
  }

  return result;
}

// ----------------------------------------------------------

function arange(n)
{
  let result = [];
  for (let i = 0; i "lt" n; ++i) {
    result[i] = Math.trunc(i);
  }
  return result;
}

// ----------------------------------------------------------

class Erratic
{
  constructor(seed)
  {
    this.seed = seed + 0.5;  // avoid 0
  }

  next()
  {
    let x = Math.sin(this.seed) * 1000;
    let result = x - Math.floor(x);  // [0.0,1.0)
    this.seed = result;  // for next call
    return result;
  }

  nextInt(lo, hi)
  {
    let x = this.next();
    return Math.trunc((hi - lo) * x + lo);
  }
}

// ----------------------------------------------------------

function vecMake(n, val)
{
  let result = [];
  for (let i = 0; i "lt" n; ++i) {
    result[i] = val;
  }
  return result;
}

function matMake(rows, cols, val)
{
  let result = [];
  for (let i = 0; i "lt" rows; ++i) {
    result[i] = [];
    for (let j = 0; j "lt" cols; ++j) {
      result[i][j] = val;
    }
  }
  return result;
}

function matToOneHot(m, n)
{
  // convert ordinal (0,1,2 . .) to one-hot
  let rows = m.length;
  let cols = m[0].length;
  let result = matMake(rows, n);
  for (let i = 0; i "lt" rows; ++i) {
    let k = Math.trunc(m[i][0]);  // 0,1,2 . .
    result[i] = vecMake(n, 0.0);  // [0.0  0.0  0.0]
    result[i][k] = 1.0;  // [ 0.0  1.0  0.0]
  }

  return result;
}

function matToVec(m)
{
  let r = m.length;
  let c = m[0].length;
  let result = 	vecMake(r*c, 0.0);
  let k = 0;
  for (let i = 0; i "lt" r; ++i) {
    for (let j = 0; j "lt" c; ++j) {
      result[k++] = m[i][j];
    }
  }
  return result;
}

function vecShow(v, dec, len)
{
  for (let i = 0; i "lt" v.length; ++i) {
    if (i != 0 "and" i % len == 0) {
      process.stdout.write("\n");
    }
    if (v[i] "gte" 0.0) {
      process.stdout.write(" ");  // + or - space
    }
    process.stdout.write(v[i].toFixed(dec));
    process.stdout.write("  ");
  }
  process.stdout.write("\n");
}

function vecShow(vec, dec, wid, nl)
{
  for (let i = 0; i "lt" vec.length; ++i) {
    let x = vec[i];
    if (Math.abs(x) "lt" 0.000001) x = 0.0  // avoid -0.00
    let xx = x.toFixed(dec);
    let s = xx.toString().padStart(wid, ' ');
    process.stdout.write(s);
    process.stdout.write(" ");
  }

  if (nl == true)
    process.stdout.write("\n");
}


function matShow(m, dec, wid)
{
  let rows = m.length;
  let cols = m[0].length;
  for (let i = 0; i "lt" rows; ++i) {
    for (let j = 0; j "lt" cols; ++j) {
      if (m[i][j] "gte" 0.0) {
        process.stdout.write(" ");  // + or - space
      }
      process.stdout.write(m[i][j].toFixed(dec));
      process.stdout.write("  ");
    }
    process.stdout.write("\n");
  }
}

function argmax(v)
{
  let result = 0;
  let m = v[0];
  for (let i = 0; i "lt" v.length; ++i) {
    if (v[i] "gt" m) {
      m = v[i];
      result = i;
    }
  }
  return result;
}

function hyperTan(x)
{
  if (x "lt" -10.0) {
    return -1.0;
  }
  else if (x "gt" 10.0) {
    return 1.0;
  }
  else {
    return Math.tanh(x);
  }
}

function logSig(x)
{
  if (x "lt" -10.0) {
    return 0.0;
  }
  else if (x "gt" 10.0) {
    return 1.0;
  }
  else {
    return 1.0 / (1.0 + Math.exp(-x));
  }
}

function vecMax(vec)
{
  let mx = vec[0];
  for (let i = 0; i "lt" vec.length; ++i) {
    if (vec[i] "gt" mx) {
      mx = vec[i];
    }
  }
  return mx;
}

function softmax(vec)
{
  //let m = Math.max(...vec);  // or 'spread' operator
  let m = vecMax(vec);
  let result = [];
  let sum = 0.0;
  for (let i = 0; i "lt" vec.length; ++i) {
    result[i] = Math.exp(vec[i] - m);
    sum += result[i];
  }
  for (let i = 0; i "lt" result.length; ++i) {
    result[i] = result[i] / sum;
  }
  return result;
}

module.exports = {
  vecMake,
  matMake,
  matToOneHot,
  matToVec,
  vecShow,
  matShow,
  argmax,
  loadTxt,
  arange,
  Erratic,
  hyperTan,
  logSig,
  vecMax,
  softmax
};

Training data:

# people_train.txt
#
# sex (-1 = male, 1 = female), age / 100,
# state (michigan = 100, nebraska = 010, oklahoma = 001),
# income / 100_000,
# politics (conservative = 100, moderate = 010, liberal = 001)
#
1, 0.24, 1, 0, 0, 0.2950, 0, 0, 1
-1, 0.39, 0, 0, 1, 0.5120, 0, 1, 0
1, 0.63, 0, 1, 0, 0.7580, 1, 0, 0
-1, 0.36, 1, 0, 0, 0.4450, 0, 1, 0
1, 0.27, 0, 1, 0, 0.2860, 0, 0, 1
1, 0.50, 0, 1, 0, 0.5650, 0, 1, 0
1, 0.50, 0, 0, 1, 0.5500, 0, 1, 0
-1, 0.19, 0, 0, 1, 0.3270, 1, 0, 0
1, 0.22, 0, 1, 0, 0.2770, 0, 1, 0
-1, 0.39, 0, 0, 1, 0.4710, 0, 0, 1
1, 0.34, 1, 0, 0, 0.3940, 0, 1, 0
-1, 0.22, 1, 0, 0, 0.3350, 1, 0, 0
1, 0.35, 0, 0, 1, 0.3520, 0, 0, 1
-1, 0.33, 0, 1, 0, 0.4640, 0, 1, 0
1, 0.45, 0, 1, 0, 0.5410, 0, 1, 0
1, 0.42, 0, 1, 0, 0.5070, 0, 1, 0
-1, 0.33, 0, 1, 0, 0.4680, 0, 1, 0
1, 0.25, 0, 0, 1, 0.3000, 0, 1, 0
-1, 0.31, 0, 1, 0, 0.4640, 1, 0, 0
1, 0.27, 1, 0, 0, 0.3250, 0, 0, 1
1, 0.48, 1, 0, 0, 0.5400, 0, 1, 0
-1, 0.64, 0, 1, 0, 0.7130, 0, 0, 1
1, 0.61, 0, 1, 0, 0.7240, 1, 0, 0
1, 0.54, 0, 0, 1, 0.6100, 1, 0, 0
1, 0.29, 1, 0, 0, 0.3630, 1, 0, 0
1, 0.50, 0, 0, 1, 0.5500, 0, 1, 0
1, 0.55, 0, 0, 1, 0.6250, 1, 0, 0
1, 0.40, 1, 0, 0, 0.5240, 1, 0, 0
1, 0.22, 1, 0, 0, 0.2360, 0, 0, 1
1, 0.68, 0, 1, 0, 0.7840, 1, 0, 0
-1, 0.60, 1, 0, 0, 0.7170, 0, 0, 1
-1, 0.34, 0, 0, 1, 0.4650, 0, 1, 0
-1, 0.25, 0, 0, 1, 0.3710, 1, 0, 0
-1, 0.31, 0, 1, 0, 0.4890, 0, 1, 0
1, 0.43, 0, 0, 1, 0.4800, 0, 1, 0
1, 0.58, 0, 1, 0, 0.6540, 0, 0, 1
-1, 0.55, 0, 1, 0, 0.6070, 0, 0, 1
-1, 0.43, 0, 1, 0, 0.5110, 0, 1, 0
-1, 0.43, 0, 0, 1, 0.5320, 0, 1, 0
-1, 0.21, 1, 0, 0, 0.3720, 1, 0, 0
1, 0.55, 0, 0, 1, 0.6460, 1, 0, 0
1, 0.64, 0, 1, 0, 0.7480, 1, 0, 0
-1, 0.41, 1, 0, 0, 0.5880, 0, 1, 0
1, 0.64, 0, 0, 1, 0.7270, 1, 0, 0
-1, 0.56, 0, 0, 1, 0.6660, 0, 0, 1
1, 0.31, 0, 0, 1, 0.3600, 0, 1, 0
-1, 0.65, 0, 0, 1, 0.7010, 0, 0, 1
1, 0.55, 0, 0, 1, 0.6430, 1, 0, 0
-1, 0.25, 1, 0, 0, 0.4030, 1, 0, 0
1, 0.46, 0, 0, 1, 0.5100, 0, 1, 0
-1, 0.36, 1, 0, 0, 0.5350, 1, 0, 0
1, 0.52, 0, 1, 0, 0.5810, 0, 1, 0
1, 0.61, 0, 0, 1, 0.6790, 1, 0, 0
1, 0.57, 0, 0, 1, 0.6570, 1, 0, 0
-1, 0.46, 0, 1, 0, 0.5260, 0, 1, 0
-1, 0.62, 1, 0, 0, 0.6680, 0, 0, 1
1, 0.55, 0, 0, 1, 0.6270, 1, 0, 0
-1, 0.22, 0, 0, 1, 0.2770, 0, 1, 0
-1, 0.50, 1, 0, 0, 0.6290, 1, 0, 0
-1, 0.32, 0, 1, 0, 0.4180, 0, 1, 0
-1, 0.21, 0, 0, 1, 0.3560, 1, 0, 0
1, 0.44, 0, 1, 0, 0.5200, 0, 1, 0
1, 0.46, 0, 1, 0, 0.5170, 0, 1, 0
1, 0.62, 0, 1, 0, 0.6970, 1, 0, 0
1, 0.57, 0, 1, 0, 0.6640, 1, 0, 0
-1, 0.67, 0, 0, 1, 0.7580, 0, 0, 1
1, 0.29, 1, 0, 0, 0.3430, 0, 0, 1
1, 0.53, 1, 0, 0, 0.6010, 1, 0, 0
-1, 0.44, 1, 0, 0, 0.5480, 0, 1, 0
1, 0.46, 0, 1, 0, 0.5230, 0, 1, 0
-1, 0.20, 0, 1, 0, 0.3010, 0, 1, 0
-1, 0.38, 1, 0, 0, 0.5350, 0, 1, 0
1, 0.50, 0, 1, 0, 0.5860, 0, 1, 0
1, 0.33, 0, 1, 0, 0.4250, 0, 1, 0
-1, 0.33, 0, 1, 0, 0.3930, 0, 1, 0
1, 0.26, 0, 1, 0, 0.4040, 1, 0, 0
1, 0.58, 1, 0, 0, 0.7070, 1, 0, 0
1, 0.43, 0, 0, 1, 0.4800, 0, 1, 0
-1, 0.46, 1, 0, 0, 0.6440, 1, 0, 0
1, 0.60, 1, 0, 0, 0.7170, 1, 0, 0
-1, 0.42, 1, 0, 0, 0.4890, 0, 1, 0
-1, 0.56, 0, 0, 1, 0.5640, 0, 0, 1
-1, 0.62, 0, 1, 0, 0.6630, 0, 0, 1
-1, 0.50, 1, 0, 0, 0.6480, 0, 1, 0
1, 0.47, 0, 0, 1, 0.5200, 0, 1, 0
-1, 0.67, 0, 1, 0, 0.8040, 0, 0, 1
-1, 0.40, 0, 0, 1, 0.5040, 0, 1, 0
1, 0.42, 0, 1, 0, 0.4840, 0, 1, 0
1, 0.64, 1, 0, 0, 0.7200, 1, 0, 0
-1, 0.47, 1, 0, 0, 0.5870, 0, 0, 1
1, 0.45, 0, 1, 0, 0.5280, 0, 1, 0
-1, 0.25, 0, 0, 1, 0.4090, 1, 0, 0
1, 0.38, 1, 0, 0, 0.4840, 1, 0, 0
1, 0.55, 0, 0, 1, 0.6000, 0, 1, 0
-1, 0.44, 1, 0, 0, 0.6060, 0, 1, 0
1, 0.33, 1, 0, 0, 0.4100, 0, 1, 0
1, 0.34, 0, 0, 1, 0.3900, 0, 1, 0
1, 0.27, 0, 1, 0, 0.3370, 0, 0, 1
1, 0.32, 0, 1, 0, 0.4070, 0, 1, 0
1, 0.42, 0, 0, 1, 0.4700, 0, 1, 0
-1, 0.24, 0, 0, 1, 0.4030, 1, 0, 0
1, 0.42, 0, 1, 0, 0.5030, 0, 1, 0
1, 0.25, 0, 0, 1, 0.2800, 0, 0, 1
1, 0.51, 0, 1, 0, 0.5800, 0, 1, 0
-1, 0.55, 0, 1, 0, 0.6350, 0, 0, 1
1, 0.44, 1, 0, 0, 0.4780, 0, 0, 1
-1, 0.18, 1, 0, 0, 0.3980, 1, 0, 0
-1, 0.67, 0, 1, 0, 0.7160, 0, 0, 1
1, 0.45, 0, 0, 1, 0.5000, 0, 1, 0
1, 0.48, 1, 0, 0, 0.5580, 0, 1, 0
-1, 0.25, 0, 1, 0, 0.3900, 0, 1, 0
-1, 0.67, 1, 0, 0, 0.7830, 0, 1, 0
1, 0.37, 0, 0, 1, 0.4200, 0, 1, 0
-1, 0.32, 1, 0, 0, 0.4270, 0, 1, 0
1, 0.48, 1, 0, 0, 0.5700, 0, 1, 0
-1, 0.66, 0, 0, 1, 0.7500, 0, 0, 1
1, 0.61, 1, 0, 0, 0.7000, 1, 0, 0
-1, 0.58, 0, 0, 1, 0.6890, 0, 1, 0
1, 0.19, 1, 0, 0, 0.2400, 0, 0, 1
1, 0.38, 0, 0, 1, 0.4300, 0, 1, 0
-1, 0.27, 1, 0, 0, 0.3640, 0, 1, 0
1, 0.42, 1, 0, 0, 0.4800, 0, 1, 0
1, 0.60, 1, 0, 0, 0.7130, 1, 0, 0
-1, 0.27, 0, 0, 1, 0.3480, 1, 0, 0
1, 0.29, 0, 1, 0, 0.3710, 1, 0, 0
-1, 0.43, 1, 0, 0, 0.5670, 0, 1, 0
1, 0.48, 1, 0, 0, 0.5670, 0, 1, 0
1, 0.27, 0, 0, 1, 0.2940, 0, 0, 1
-1, 0.44, 1, 0, 0, 0.5520, 1, 0, 0
1, 0.23, 0, 1, 0, 0.2630, 0, 0, 1
-1, 0.36, 0, 1, 0, 0.5300, 0, 0, 1
1, 0.64, 0, 0, 1, 0.7250, 1, 0, 0
1, 0.29, 0, 0, 1, 0.3000, 0, 0, 1
-1, 0.33, 1, 0, 0, 0.4930, 0, 1, 0
-1, 0.66, 0, 1, 0, 0.7500, 0, 0, 1
-1, 0.21, 0, 0, 1, 0.3430, 1, 0, 0
1, 0.27, 1, 0, 0, 0.3270, 0, 0, 1
1, 0.29, 1, 0, 0, 0.3180, 0, 0, 1
-1, 0.31, 1, 0, 0, 0.4860, 0, 1, 0
1, 0.36, 0, 0, 1, 0.4100, 0, 1, 0
1, 0.49, 0, 1, 0, 0.5570, 0, 1, 0
-1, 0.28, 1, 0, 0, 0.3840, 1, 0, 0
-1, 0.43, 0, 0, 1, 0.5660, 0, 1, 0
-1, 0.46, 0, 1, 0, 0.5880, 0, 1, 0
1, 0.57, 1, 0, 0, 0.6980, 1, 0, 0
-1, 0.52, 0, 0, 1, 0.5940, 0, 1, 0
-1, 0.31, 0, 0, 1, 0.4350, 0, 1, 0
-1, 0.55, 1, 0, 0, 0.6200, 0, 0, 1
1, 0.50, 1, 0, 0, 0.5640, 0, 1, 0
1, 0.48, 0, 1, 0, 0.5590, 0, 1, 0
-1, 0.22, 0, 0, 1, 0.3450, 1, 0, 0
1, 0.59, 0, 0, 1, 0.6670, 1, 0, 0
1, 0.34, 1, 0, 0, 0.4280, 0, 0, 1
-1, 0.64, 1, 0, 0, 0.7720, 0, 0, 1
1, 0.29, 0, 0, 1, 0.3350, 0, 0, 1
-1, 0.34, 0, 1, 0, 0.4320, 0, 1, 0
-1, 0.61, 1, 0, 0, 0.7500, 0, 0, 1
1, 0.64, 0, 0, 1, 0.7110, 1, 0, 0
-1, 0.29, 1, 0, 0, 0.4130, 1, 0, 0
1, 0.63, 0, 1, 0, 0.7060, 1, 0, 0
-1, 0.29, 0, 1, 0, 0.4000, 1, 0, 0
-1, 0.51, 1, 0, 0, 0.6270, 0, 1, 0
-1, 0.24, 0, 0, 1, 0.3770, 1, 0, 0
1, 0.48, 0, 1, 0, 0.5750, 0, 1, 0
1, 0.18, 1, 0, 0, 0.2740, 1, 0, 0
1, 0.18, 1, 0, 0, 0.2030, 0, 0, 1
1, 0.33, 0, 1, 0, 0.3820, 0, 0, 1
-1, 0.20, 0, 0, 1, 0.3480, 1, 0, 0
1, 0.29, 0, 0, 1, 0.3300, 0, 0, 1
-1, 0.44, 0, 0, 1, 0.6300, 1, 0, 0
-1, 0.65, 0, 0, 1, 0.8180, 1, 0, 0
-1, 0.56, 1, 0, 0, 0.6370, 0, 0, 1
-1, 0.52, 0, 0, 1, 0.5840, 0, 1, 0
-1, 0.29, 0, 1, 0, 0.4860, 1, 0, 0
-1, 0.47, 0, 1, 0, 0.5890, 0, 1, 0
1, 0.68, 1, 0, 0, 0.7260, 0, 0, 1
1, 0.31, 0, 0, 1, 0.3600, 0, 1, 0
1, 0.61, 0, 1, 0, 0.6250, 0, 0, 1
1, 0.19, 0, 1, 0, 0.2150, 0, 0, 1
1, 0.38, 0, 0, 1, 0.4300, 0, 1, 0
-1, 0.26, 1, 0, 0, 0.4230, 1, 0, 0
1, 0.61, 0, 1, 0, 0.6740, 1, 0, 0
1, 0.40, 1, 0, 0, 0.4650, 0, 1, 0
-1, 0.49, 1, 0, 0, 0.6520, 0, 1, 0
1, 0.56, 1, 0, 0, 0.6750, 1, 0, 0
-1, 0.48, 0, 1, 0, 0.6600, 0, 1, 0
1, 0.52, 1, 0, 0, 0.5630, 0, 0, 1
-1, 0.18, 1, 0, 0, 0.2980, 1, 0, 0
-1, 0.56, 0, 0, 1, 0.5930, 0, 0, 1
-1, 0.52, 0, 1, 0, 0.6440, 0, 1, 0
-1, 0.18, 0, 1, 0, 0.2860, 0, 1, 0
-1, 0.58, 1, 0, 0, 0.6620, 0, 0, 1
-1, 0.39, 0, 1, 0, 0.5510, 0, 1, 0
-1, 0.46, 1, 0, 0, 0.6290, 0, 1, 0
-1, 0.40, 0, 1, 0, 0.4620, 0, 1, 0
-1, 0.60, 1, 0, 0, 0.7270, 0, 0, 1
1, 0.36, 0, 1, 0, 0.4070, 0, 0, 1
1, 0.44, 1, 0, 0, 0.5230, 0, 1, 0
1, 0.28, 1, 0, 0, 0.3130, 0, 0, 1
1, 0.54, 0, 0, 1, 0.6260, 1, 0, 0

Test data:

# people_test.txt
#
-1, 0.51, 1, 0, 0, 0.6120, 0, 1, 0
-1, 0.32, 0, 1, 0, 0.4610, 0, 1, 0
1, 0.55, 1, 0, 0, 0.6270, 1, 0, 0
1, 0.25, 0, 0, 1, 0.2620, 0, 0, 1
1, 0.33, 0, 0, 1, 0.3730, 0, 0, 1
-1, 0.29, 0, 1, 0, 0.4620, 1, 0, 0
1, 0.65, 1, 0, 0, 0.7270, 1, 0, 0
-1, 0.43, 0, 1, 0, 0.5140, 0, 1, 0
-1, 0.54, 0, 1, 0, 0.6480, 0, 0, 1
1, 0.61, 0, 1, 0, 0.7270, 1, 0, 0
1, 0.52, 0, 1, 0, 0.6360, 1, 0, 0
1, 0.3, 0, 1, 0, 0.3350, 0, 0, 1
1, 0.29, 1, 0, 0, 0.3140, 0, 0, 1
-1, 0.47, 0, 0, 1, 0.5940, 0, 1, 0
1, 0.39, 0, 1, 0, 0.4780, 0, 1, 0
1, 0.47, 0, 0, 1, 0.5200, 0, 1, 0
-1, 0.49, 1, 0, 0, 0.5860, 0, 1, 0
-1, 0.63, 0, 0, 1, 0.6740, 0, 0, 1
-1, 0.3, 1, 0, 0, 0.3920, 1, 0, 0
-1, 0.61, 0, 0, 1, 0.6960, 0, 0, 1
-1, 0.47, 0, 0, 1, 0.5870, 0, 1, 0
1, 0.3, 0, 0, 1, 0.3450, 0, 0, 1
-1, 0.51, 0, 0, 1, 0.5800, 0, 1, 0
-1, 0.24, 1, 0, 0, 0.3880, 0, 1, 0
-1, 0.49, 1, 0, 0, 0.6450, 0, 1, 0
1, 0.66, 0, 0, 1, 0.7450, 1, 0, 0
-1, 0.65, 1, 0, 0, 0.7690, 1, 0, 0
-1, 0.46, 0, 1, 0, 0.5800, 1, 0, 0
-1, 0.45, 0, 0, 1, 0.5180, 0, 1, 0
-1, 0.47, 1, 0, 0, 0.6360, 1, 0, 0
-1, 0.29, 1, 0, 0, 0.4480, 1, 0, 0
-1, 0.57, 0, 0, 1, 0.6930, 0, 0, 1
-1, 0.2, 1, 0, 0, 0.2870, 0, 0, 1
-1, 0.35, 1, 0, 0, 0.4340, 0, 1, 0
-1, 0.61, 0, 0, 1, 0.6700, 0, 0, 1
-1, 0.31, 0, 0, 1, 0.3730, 0, 1, 0
1, 0.18, 1, 0, 0, 0.2080, 0, 0, 1
1, 0.26, 0, 0, 1, 0.2920, 0, 0, 1
-1, 0.28, 1, 0, 0, 0.3640, 0, 0, 1
-1, 0.59, 0, 0, 1, 0.6940, 0, 0, 1