Multi-Class Classification Using PyTorch 1.12.1 on Windows 10/11

One of the challenges of working with the PyTorch neural network library is that there are constant updates. Over the past two years there have been approximately 16 new releases — roughly a new release every six or seven weeks or so.

I noticed that PyTorch 1.12.1 was released a few weeks ago so I figured I’d do a quick multi-class classification demo to make sure there were no breaking changes.

There’s a strong coupling between PyTorch version and Python version. I currently use Python 3.7.6 from the Anaconda 2020.02 distribution. I located the appropriate .whl file at https://download.pytorch.org/whl/torch_stable.html — torch-1.12.1+cpu-cp37-cp37m-win_amd64.whl. Even though I have installed PyTorch hundreds of times, I have grabbed the wrong .whl file more than once.

I opened a Windows command shell with admin privileges. I uninstalled my existing PyTorch 1.10.0 using the command “pip uninstall torch”. Then I navigated to the directory holding the new .whl file and installed it with the command “pip install torch-1.12-etc-.whl”. There were no problems.

I used one of my standard datasets for the multi-class example. The data looks like:

 1   0.24   1   0   0   0.2950   2
-1   0.39   0   0   1   0.5120   1
 1   0.63   0   1   0   0.7580   0
-1   0.36   1   0   0   0.4450   1
. . . 

Each line of data represents a person. The fields are sex (male = -1, female = 1), age (normalized by dividing by 100), state (michigan = 100, nebraska = 010, oklahoma = 001), annual income (divided by 100,000), and politics type (0 = conservative, 1 = moderate, 2 = liberal). The goal is to predict politics type from sex, age, state, income.

My demo network used a 6-(10-10)-3 architecture with tanh() hidden activation and log_softmax() activation on the output nodes. I used explicit weight and bias initialization:

class Net(T.nn.Module):
  def __init__(self):
    super(Net, self).__init__()
    self.hid1 = T.nn.Linear(6, 10)  # 6-(10-10)-3
    self.hid2 = T.nn.Linear(10, 10)
    self.oupt = T.nn.Linear(10, 3)

    T.nn.init.xavier_uniform_(self.hid1.weight)
    T.nn.init.zeros_(self.hid1.bias)
    T.nn.init.xavier_uniform_(self.hid2.weight)
    T.nn.init.zeros_(self.hid2.bias)
    T.nn.init.xavier_uniform_(self.oupt.weight)
    T.nn.init.zeros_(self.oupt.bias)

  def forward(self, x):
    z = T.tanh(self.hid1(x))
    z = T.tanh(self.hid2(z))
    z = T.log_softmax(self.oupt(z), dim=1)  # NLLLoss() 
    return z

For training, I used a batch size of 10, SGD optimization with a fixed learning rate of 0.01, and NLLLoss().

I didn’t run into any problems. PyTorch is slowly but surely stabilizing. Most of the version changes are related to advanced architectures such as Transformers rather than standard architectures.

Good exercise.



Airline first class vs. economy class. Left: By Gahan Wilson. Right: By Gary Larson.


Demo code and data. Replace “lt”, “gt”, “lte”, “gte” with Boolean operator symbols.

# people_politics.py
# predict politics type from sex, age, state, income
# PyTorch 1.12.1-CPU Anaconda3-2020.02  Python 3.7.6
# Windows 10/11 

import numpy as np
import torch as T
device = T.device('cpu')  # apply to Tensor or Module

# -----------------------------------------------------------

class PeopleDataset(T.utils.data.Dataset):
  # sex  age    state    income   politics
  # -1   0.27   0  1  0   0.7610   2
  # +1   0.19   0  0  1   0.6550   0
  # sex: -1 = male, +1 = female
  # state: michigan, nebraska, oklahoma
  # politics: conservative, moderate, liberal

  def __init__(self, src_file):
    all_xy = np.loadtxt(src_file, usecols=range(0,7),
      delimiter="\t", comments="#", dtype=np.float32)
    tmp_x = all_xy[:,0:6]   # cols [0,6) = [0,5]
    tmp_y = all_xy[:,6]     # 1-D

    self.x_data = T.tensor(tmp_x, 
      dtype=T.float32).to(device)
    self.y_data = T.tensor(tmp_y,
      dtype=T.int64).to(device)  # 1-D

  def __len__(self):
    return len(self.x_data)

  def __getitem__(self, idx):
    preds = self.x_data[idx]
    trgts = self.y_data[idx] 
    return preds, trgts  # as a Tuple

# -----------------------------------------------------------

class Net(T.nn.Module):
  def __init__(self):
    super(Net, self).__init__()
    self.hid1 = T.nn.Linear(6, 10)  # 6-(10-10)-3
    self.hid2 = T.nn.Linear(10, 10)
    self.oupt = T.nn.Linear(10, 3)

    T.nn.init.xavier_uniform_(self.hid1.weight)
    T.nn.init.zeros_(self.hid1.bias)
    T.nn.init.xavier_uniform_(self.hid2.weight)
    T.nn.init.zeros_(self.hid2.bias)
    T.nn.init.xavier_uniform_(self.oupt.weight)
    T.nn.init.zeros_(self.oupt.bias)

    # self.my_tanh = T.nn.Tanh()

  def forward(self, x):
    z = T.tanh(self.hid1(x))
    z = T.tanh(self.hid2(z))
    z = T.log_softmax(self.oupt(z), dim=1)  # NLLLoss() 
    return z

# -----------------------------------------------------------

def accuracy(model, ds):
  # assumes model.eval()
  # item-by-item version
  n_correct = 0; n_wrong = 0
  for i in range(len(ds)):
    X = ds[i][0].reshape(1,-1)  # make it a batch
    Y = ds[i][1].reshape(1)  # 0 1 or 2, 1D
    with T.no_grad():
      oupt = model(X)  # logits form

    big_idx = T.argmax(oupt)  # 0 or 1 or 2
    if big_idx == Y:
      n_correct += 1
    else:
      n_wrong += 1

  acc = (n_correct * 1.0) / (n_correct + n_wrong)
  return acc

# -----------------------------------------------------------

def accuracy_quick(model, dataset):
  # assumes model.eval()
  X = dataset[0:len(dataset)][0]
  # Y = T.flatten(dataset[0:len(dataset)][1])
  Y = dataset[0:len(dataset)][1]
  with T.no_grad():
    oupt = model(X)
  # (_, arg_maxs) = T.max(oupt, dim=1)
  arg_maxs = T.argmax(oupt, dim=1)  # argmax() is new
  num_correct = T.sum(Y==arg_maxs)
  acc = (num_correct * 1.0 / len(dataset))
  return acc.item()

# -----------------------------------------------------------

def main():
  # 0. get started
  print("\nBegin People predict politics type ")
  T.manual_seed(1)
  np.random.seed(1)
  
  # 1. create DataLoader objects
  print("\nCreating People Datasets ")

  train_file = ".\\Data\\people_train.txt"
  train_ds = PeopleDataset(train_file)  # 200 rows

  test_file = ".\\Data\\people_test.txt"
  test_ds = PeopleDataset(test_file)    # 40 rows

  bat_size = 10
  train_ldr = T.utils.data.DataLoader(train_ds,
    batch_size=bat_size, shuffle=True)

# -----------------------------------------------------------

  # 2. create network
  print("\nCreating 6-(10-10)-3 neural network ")
  net = Net().to(device)
  net.train()

# -----------------------------------------------------------

  # 3. train model
  max_epochs = 1000
  ep_log_interval = 100
  lrn_rate = 0.01

  loss_func = T.nn.NLLLoss()  # assumes log_softmax()
  optimizer = T.optim.SGD(net.parameters(), lr=lrn_rate)

  print("\nbat_size = %3d " % bat_size)
  print("loss = " + str(loss_func))
  print("optimizer = SGD")
  print("max_epochs = %3d " % max_epochs)
  print("lrn_rate = %0.3f " % lrn_rate)

  print("\nStarting training ")
  for epoch in range(0, max_epochs):
    # T.manual_seed(epoch+1)  # checkpoint reproducibility
    epoch_loss = 0  # for one full epoch

    for (batch_idx, batch) in enumerate(train_ldr):
      X = batch[0]  # inputs
      Y = batch[1]  # correct class/label/politics

      optimizer.zero_grad()
      oupt = net(X)
      loss_val = loss_func(oupt, Y)  # a tensor
      epoch_loss += loss_val.item()  # accumulate
      loss_val.backward()
      optimizer.step()

    if epoch % ep_log_interval == 0:
      print("epoch = %5d  |  loss = %10.4f" % \
        (epoch, epoch_loss))

  print("Training done ")

# -----------------------------------------------------------

  # 4. evaluate model accuracy
  print("\nComputing model accuracy")
  net.eval()
  acc_train = accuracy(net, train_ds)  # item-by-item
  print("Accuracy on training data = %0.4f" % acc_train)
  acc_test = accuracy(net, test_ds) 
  print("Accuracy on test data = %0.4f" % acc_test)

  # 5. make a prediction
  print("\nPredicting politics for M  30  oklahoma  $50,000: ")
  X = np.array([[-1, 0.30,  0,0,1,  0.5000]], dtype=np.float32)
  X = T.tensor(X, dtype=T.float32).to(device) 

  with T.no_grad():
    logits = net(X)  # do not sum to 1.0
  probs = T.exp(logits)  # sum to 1.0
  probs = probs.numpy()  # numpy vector prints better
  np.set_printoptions(precision=4, suppress=True)
  print(probs)

  # 6. save model (state_dict approach)
  print("\nSaving trained model state ")
  fn = ".\\Models\\people_model.pt"
  T.save(net.state_dict(), fn)

  # model = Net()  # requires class definition
  # model.load_state_dict(T.load(fn))
  # use model to make prediction(s)

  print("\nEnd People predict politics demo ")

if __name__ == "__main__":
  main()

Training data. Replace commas with tab characters and save as people_train.txt.

# people_train.txt
# sex (M=-1 F=1)  age  state (michigan 
# nebraska oklahoma) income
# politics (con mod lib)
#
1,0.24,1,0,0,0.2950,2
-1,0.39,0,0,1,0.5120,1
1,0.63,0,1,0,0.7580,0
-1,0.36,1,0,0,0.4450,1
1,0.27,0,1,0,0.2860,2
1,0.50,0,1,0,0.5650,1
1,0.50,0,0,1,0.5500,1
-1,0.19,0,0,1,0.3270,0
1,0.22,0,1,0,0.2770,1
-1,0.39,0,0,1,0.4710,2
1,0.34,1,0,0,0.3940,1
-1,0.22,1,0,0,0.3350,0
1,0.35,0,0,1,0.3520,2
-1,0.33,0,1,0,0.4640,1
1,0.45,0,1,0,0.5410,1
1,0.42,0,1,0,0.5070,1
-1,0.33,0,1,0,0.4680,1
1,0.25,0,0,1,0.3000,1
-1,0.31,0,1,0,0.4640,0
1,0.27,1,0,0,0.3250,2
1,0.48,1,0,0,0.5400,1
-1,0.64,0,1,0,0.7130,2
1,0.61,0,1,0,0.7240,0
1,0.54,0,0,1,0.6100,0
1,0.29,1,0,0,0.3630,0
1,0.50,0,0,1,0.5500,1
1,0.55,0,0,1,0.6250,0
1,0.40,1,0,0,0.5240,0
1,0.22,1,0,0,0.2360,2
1,0.68,0,1,0,0.7840,0
-1,0.60,1,0,0,0.7170,2
-1,0.34,0,0,1,0.4650,1
-1,0.25,0,0,1,0.3710,0
-1,0.31,0,1,0,0.4890,1
1,0.43,0,0,1,0.4800,1
1,0.58,0,1,0,0.6540,2
-1,0.55,0,1,0,0.6070,2
-1,0.43,0,1,0,0.5110,1
-1,0.43,0,0,1,0.5320,1
-1,0.21,1,0,0,0.3720,0
1,0.55,0,0,1,0.6460,0
1,0.64,0,1,0,0.7480,0
-1,0.41,1,0,0,0.5880,1
1,0.64,0,0,1,0.7270,0
-1,0.56,0,0,1,0.6660,2
1,0.31,0,0,1,0.3600,1
-1,0.65,0,0,1,0.7010,2
1,0.55,0,0,1,0.6430,0
-1,0.25,1,0,0,0.4030,0
1,0.46,0,0,1,0.5100,1
-1,0.36,1,0,0,0.5350,0
1,0.52,0,1,0,0.5810,1
1,0.61,0,0,1,0.6790,0
1,0.57,0,0,1,0.6570,0
-1,0.46,0,1,0,0.5260,1
-1,0.62,1,0,0,0.6680,2
1,0.55,0,0,1,0.6270,0
-1,0.22,0,0,1,0.2770,1
-1,0.50,1,0,0,0.6290,0
-1,0.32,0,1,0,0.4180,1
-1,0.21,0,0,1,0.3560,0
1,0.44,0,1,0,0.5200,1
1,0.46,0,1,0,0.5170,1
1,0.62,0,1,0,0.6970,0
1,0.57,0,1,0,0.6640,0
-1,0.67,0,0,1,0.7580,2
1,0.29,1,0,0,0.3430,2
1,0.53,1,0,0,0.6010,0
-1,0.44,1,0,0,0.5480,1
1,0.46,0,1,0,0.5230,1
-1,0.20,0,1,0,0.3010,1
-1,0.38,1,0,0,0.5350,1
1,0.50,0,1,0,0.5860,1
1,0.33,0,1,0,0.4250,1
-1,0.33,0,1,0,0.3930,1
1,0.26,0,1,0,0.4040,0
1,0.58,1,0,0,0.7070,0
1,0.43,0,0,1,0.4800,1
-1,0.46,1,0,0,0.6440,0
1,0.60,1,0,0,0.7170,0
-1,0.42,1,0,0,0.4890,1
-1,0.56,0,0,1,0.5640,2
-1,0.62,0,1,0,0.6630,2
-1,0.50,1,0,0,0.6480,1
1,0.47,0,0,1,0.5200,1
-1,0.67,0,1,0,0.8040,2
-1,0.40,0,0,1,0.5040,1
1,0.42,0,1,0,0.4840,1
1,0.64,1,0,0,0.7200,0
-1,0.47,1,0,0,0.5870,2
1,0.45,0,1,0,0.5280,1
-1,0.25,0,0,1,0.4090,0
1,0.38,1,0,0,0.4840,0
1,0.55,0,0,1,0.6000,1
-1,0.44,1,0,0,0.6060,1
1,0.33,1,0,0,0.4100,1
1,0.34,0,0,1,0.3900,1
1,0.27,0,1,0,0.3370,2
1,0.32,0,1,0,0.4070,1
1,0.42,0,0,1,0.4700,1
-1,0.24,0,0,1,0.4030,0
1,0.42,0,1,0,0.5030,1
1,0.25,0,0,1,0.2800,2
1,0.51,0,1,0,0.5800,1
-1,0.55,0,1,0,0.6350,2
1,0.44,1,0,0,0.4780,2
-1,0.18,1,0,0,0.3980,0
-1,0.67,0,1,0,0.7160,2
1,0.45,0,0,1,0.5000,1
1,0.48,1,0,0,0.5580,1
-1,0.25,0,1,0,0.3900,1
-1,0.67,1,0,0,0.7830,1
1,0.37,0,0,1,0.4200,1
-1,0.32,1,0,0,0.4270,1
1,0.48,1,0,0,0.5700,1
-1,0.66,0,0,1,0.7500,2
1,0.61,1,0,0,0.7000,0
-1,0.58,0,0,1,0.6890,1
1,0.19,1,0,0,0.2400,2
1,0.38,0,0,1,0.4300,1
-1,0.27,1,0,0,0.3640,1
1,0.42,1,0,0,0.4800,1
1,0.60,1,0,0,0.7130,0
-1,0.27,0,0,1,0.3480,0
1,0.29,0,1,0,0.3710,0
-1,0.43,1,0,0,0.5670,1
1,0.48,1,0,0,0.5670,1
1,0.27,0,0,1,0.2940,2
-1,0.44,1,0,0,0.5520,0
1,0.23,0,1,0,0.2630,2
-1,0.36,0,1,0,0.5300,2
1,0.64,0,0,1,0.7250,0
1,0.29,0,0,1,0.3000,2
-1,0.33,1,0,0,0.4930,1
-1,0.66,0,1,0,0.7500,2
-1,0.21,0,0,1,0.3430,0
1,0.27,1,0,0,0.3270,2
1,0.29,1,0,0,0.3180,2
-1,0.31,1,0,0,0.4860,1
1,0.36,0,0,1,0.4100,1
1,0.49,0,1,0,0.5570,1
-1,0.28,1,0,0,0.3840,0
-1,0.43,0,0,1,0.5660,1
-1,0.46,0,1,0,0.5880,1
1,0.57,1,0,0,0.6980,0
-1,0.52,0,0,1,0.5940,1
-1,0.31,0,0,1,0.4350,1
-1,0.55,1,0,0,0.6200,2
1,0.50,1,0,0,0.5640,1
1,0.48,0,1,0,0.5590,1
-1,0.22,0,0,1,0.3450,0
1,0.59,0,0,1,0.6670,0
1,0.34,1,0,0,0.4280,2
-1,0.64,1,0,0,0.7720,2
1,0.29,0,0,1,0.3350,2
-1,0.34,0,1,0,0.4320,1
-1,0.61,1,0,0,0.7500,2
1,0.64,0,0,1,0.7110,0
-1,0.29,1,0,0,0.4130,0
1,0.63,0,1,0,0.7060,0
-1,0.29,0,1,0,0.4000,0
-1,0.51,1,0,0,0.6270,1
-1,0.24,0,0,1,0.3770,0
1,0.48,0,1,0,0.5750,1
1,0.18,1,0,0,0.2740,0
1,0.18,1,0,0,0.2030,2
1,0.33,0,1,0,0.3820,2
-1,0.20,0,0,1,0.3480,0
1,0.29,0,0,1,0.3300,2
-1,0.44,0,0,1,0.6300,0
-1,0.65,0,0,1,0.8180,0
-1,0.56,1,0,0,0.6370,2
-1,0.52,0,0,1,0.5840,1
-1,0.29,0,1,0,0.4860,0
-1,0.47,0,1,0,0.5890,1
1,0.68,1,0,0,0.7260,2
1,0.31,0,0,1,0.3600,1
1,0.61,0,1,0,0.6250,2
1,0.19,0,1,0,0.2150,2
1,0.38,0,0,1,0.4300,1
-1,0.26,1,0,0,0.4230,0
1,0.61,0,1,0,0.6740,0
1,0.40,1,0,0,0.4650,1
-1,0.49,1,0,0,0.6520,1
1,0.56,1,0,0,0.6750,0
-1,0.48,0,1,0,0.6600,1
1,0.52,1,0,0,0.5630,2
-1,0.18,1,0,0,0.2980,0
-1,0.56,0,0,1,0.5930,2
-1,0.52,0,1,0,0.6440,1
-1,0.18,0,1,0,0.2860,1
-1,0.58,1,0,0,0.6620,2
-1,0.39,0,1,0,0.5510,1
-1,0.46,1,0,0,0.6290,1
-1,0.40,0,1,0,0.4620,1
-1,0.60,1,0,0,0.7270,2
1,0.36,0,1,0,0.4070,2
1,0.44,1,0,0,0.5230,1
1,0.28,1,0,0,0.3130,2
1,0.54,0,0,1,0.6260,0

Test data. Replace commas with tab characters and save as people_test.txt.

-1,0.51,1,0,0,0.6120,1
-1,0.32,0,1,0,0.4610,1
1,0.55,1,0,0,0.6270,0
1,0.25,0,0,1,0.2620,2
1,0.33,0,0,1,0.3730,2
-1,0.29,0,1,0,0.4620,0
1,0.65,1,0,0,0.7270,0
-1,0.43,0,1,0,0.5140,1
-1,0.54,0,1,0,0.6480,2
1,0.61,0,1,0,0.7270,0
1,0.52,0,1,0,0.6360,0
1,0.30,0,1,0,0.3350,2
1,0.29,1,0,0,0.3140,2
-1,0.47,0,0,1,0.5940,1
1,0.39,0,1,0,0.4780,1
1,0.47,0,0,1,0.5200,1
-1,0.49,1,0,0,0.5860,1
-1,0.63,0,0,1,0.6740,2
-1,0.30,1,0,0,0.3920,0
-1,0.61,0,0,1,0.6960,2
-1,0.47,0,0,1,0.5870,1
1,0.30,0,0,1,0.3450,2
-1,0.51,0,0,1,0.5800,1
-1,0.24,1,0,0,0.3880,1
-1,0.49,1,0,0,0.6450,1
1,0.66,0,0,1,0.7450,0
-1,0.65,1,0,0,0.7690,0
-1,0.46,0,1,0,0.5800,0
-1,0.45,0,0,1,0.5180,1
-1,0.47,1,0,0,0.6360,0
-1,0.29,1,0,0,0.4480,0
-1,0.57,0,0,1,0.6930,2
-1,0.20,1,0,0,0.2870,2
-1,0.35,1,0,0,0.4340,1
-1,0.61,0,0,1,0.6700,2
-1,0.31,0,0,1,0.3730,1
1,0.18,1,0,0,0.2080,2
1,0.26,0,0,1,0.2920,2
-1,0.28,1,0,0,0.3640,2
-1,0.59,0,0,1,0.6940,2
This entry was posted in PyTorch. Bookmark the permalink.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s