Dealing With PyTorch Library Versioning

It’s a major challenge to deal with the compatibility of the dozens of Python based libraries needed to work with the PyTorch neural network library.

I was currently using PyTorch version 1.8 (for CPU) along with the Anaconda3-2020.02 distribution which contains Python 3.7.6 and roughly 500 mostly-compatible Python libraries. For natural language processing I was using TorchText 0.9 which is significantly different from its 0.8 predecessor. For image processing I was using TorchVision 0.9 (the fact that both TorchText and TorchVision are version 0.9 is coincidence).

One morning, I noticed that PyTorch version 1.9 had been released a few days earlier. So I decided to install it and run my basic Iris Dataset demo program as a sanity check. I knew before I started that there would be problems.



Briefly, I uninstalled PyTorch 1.8 but during installation of the 1.9 version, I got a warning during installation that my TorchVision 0.9 would not work with my upgraded PyTorch 1.9 version. This was somewhat expected — TorchText and TorchVision typically lag behind new versions of PyTorch by two to three months.

Furthermore, even though I didn’t get a warning, my existing TochVision 0.9 version failed to load with the latest PyTorch 1.9 version. But my basic multi-class classification demo program on the Iris Dataset worked fine.

The moral to all of this is that even though Python / PyTorch versioning compatibility is much better than it used to be, versioning compatibility is still a major headache.

In a weird way, dealing with PyTorch versioning is a hidden barrier to entry for data scientists who are learning deep neural techniques with PyTorch. I often teach employees at my tech company, and I always tell beginners that they should not underestimate how difficult it is to deal with PyTorch versioning.



Versions of automobiles. A blue 1973 Porsche 911 had a big influence on my life. In 1973, one of my college roommates was Scott Putney. Scott’s father lived in Sherman Oaks and had his brand new blue Porsche Targa stolen. The car was miraculously recovered in Florida. Scott and I flew to Florida (actually Alabama), picked up the car and drove it to California. We had many adventures, including getting thrown in jail in Junction, Texas. I had great admiration for Scott’s father and always aspired to be successful enough to own my own Porsche someday. This seemed out of reach because I was poor and studying cognitive psychology in school, which wasn’t exactly a top money-making major.

Left: An early Porsche model 901 (from 1964). Center: A 1973 Targa model, just like the one Scott and I drove across the country. Right: I eventually fulfilled my goal when I bought a model 996 (1999). I still own the car, mostly as a reminder of the value of persistence and striving for a long-term goal. And I still stay in touch with Scott and our other roommate, Ed Koolish.


Code for my sanity check Iris Dataset demo program below. Long.

# iris_nn.py
# iris example
# PyTorch 1.9.0-CPU Anaconda3-2020.02  Python 3.7.6
# Windows 10 

import numpy as np
import torch as T
device = T.device("cpu")  # apply to Tensor or Module

# -----------------------------------------------------------

class IrisDataset(T.utils.data.Dataset):
  def __init__(self, src_file, num_rows=None):
    # 5.0, 3.5, 1.3, 0.3, 0
    tmp_x = np.loadtxt(src_file, max_rows=num_rows,
      usecols=range(0,4), delimiter=",", skiprows=0,
      dtype=np.float32)
    tmp_y = np.loadtxt(src_file, max_rows=num_rows,
      usecols=4, delimiter=",", skiprows=0,
      dtype=np.int64)

    self.x_data = T.tensor(tmp_x, dtype=T.float32)
    self.y_data = T.tensor(tmp_y, dtype=T.int64)

  def __len__(self):
    return len(self.x_data)

  def __getitem__(self, idx):
    if T.is_tensor(idx):
      idx = idx.tolist()
    preds = self.x_data[idx]
    spcs = self.y_data[idx] 
    sample = { 'predictors' : preds, 'species' : spcs }
    return sample

# -----------------------------------------------------------

class Net(T.nn.Module):
  def __init__(self):
    super(Net, self).__init__()
    self.hid1 = T.nn.Linear(4, 7)  # 4-7-3
    self.oupt = T.nn.Linear(7, 3)

    T.nn.init.xavier_uniform_(self.hid1.weight)
    T.nn.init.zeros_(self.hid1.bias)
    T.nn.init.xavier_uniform_(self.oupt.weight)
    T.nn.init.zeros_(self.oupt.bias)

  def forward(self, x):
    z = T.tanh(self.hid1(x))
    z = self.oupt(z)  # no softmax: CrossEntropyLoss() 
    return z

# -----------------------------------------------------------

def accuracy(model, dataset):
  # assumes model.eval()
  dataldr = T.utils.data.DataLoader(dataset, batch_size=1,
    shuffle=False)
  n_correct = 0; n_wrong = 0
  for (_, batch) in enumerate(dataldr):
    X = batch['predictors'] 
    # Y = T.flatten(batch['species'])
    Y = batch['species']  # already flattened by Dataset
    with T.no_grad():
      oupt = model(X)  # logits form

    big_idx = T.argmax(oupt)
    # if big_idx.item() == Y.item():
    if big_idx == Y:
      n_correct += 1
    else:
      n_wrong += 1

  acc = (n_correct * 1.0) / (n_correct + n_wrong)
  return acc

# -----------------------------------------------------------

def main():
  # 0. get started
  print("\nBegin Iris dataset using PyTorch 1.9 demo \n")
  T.manual_seed(1)
  np.random.seed(1)
  
  # 1. create DataLoader objects
  print("Creating Iris train and test DataLoader ")

  train_file = ".\\Data\\iris_train.txt"
  test_file = ".\\Data\\iris_test.txt"

  train_ds = IrisDataset(train_file, num_rows=120)
  test_ds = IrisDataset(test_file)  # 120 

  bat_size = 4
  train_ldr = T.utils.data.DataLoader(train_ds,
    batch_size=bat_size, shuffle=True)
  test_ldr = T.utils.data.DataLoader(test_ds,
    batch_size=1, shuffle=False)

  # 2. create network
  net = Net().to(device)

  # 3. train model
  max_epochs = 12
  ep_log_interval = 2
  # ep_log_ct = 10
  # ep_log_interval = max_epochs // ep_log_count
  lrn_rate = 0.05

  loss_func = T.nn.CrossEntropyLoss()  # applies softmax()
  optimizer = T.optim.SGD(net.parameters(), lr=lrn_rate)

  print("\nbat_size = %3d " % bat_size)
  print("loss = " + str(loss_func))
  print("optimizer = SGD")
  print("max_epochs = %3d " % max_epochs)
  print("lrn_rate = %0.3f " % lrn_rate)

  print("\nStarting training")
  net.train()
  for epoch in range(0, max_epochs):
    epoch_loss = 0  # for one full epoch
    num_lines_read = 0

    for (batch_idx, batch) in enumerate(train_ldr):
      X = batch['predictors']  # [10,4]
      # Y = T.flatten(batch['species'])  # [10,1] to [10]
      Y = batch['species']  # OK; alreay flattened
      # num_lines_read += bat_size  # early exit
      optimizer.zero_grad()
      oupt = net(X)
      loss_obj = loss_func(oupt, Y)  # a tensor
      epoch_loss += loss_obj.item()  # accumulate
      loss_obj.backward()
      optimizer.step()

    if epoch % ep_log_interval == 0:
      print("epoch = %4d   loss = %0.4f" % (epoch, epoch_loss))
  print("Done ")

  # 4. evaluate model accuracy
  print("\nComputing model accuracy")
  net.eval()
  acc = accuracy(net, test_ds)  # item-by-item
  print("Accuracy on test data = %0.4f" % acc)

  # 5. make a prediction
  print("\nPredicting species for [6.1, 3.1, 5.1, 1.1]: ")
  unk = np.array([[6.1, 3.1, 5.1, 1.1]], dtype=np.float32)
  unk = T.tensor(unk, dtype=T.float32).to(device) 

  with T.no_grad():
    logits = net(unk).to(device)  # values do not sum to 1.0
  probs = T.softmax(logits, dim=1).to(device)
  T.set_printoptions(precision=4)
  print(probs)

  # 6. save model (state_dict approach)
  print("\nSaving trained model state")
  fn = ".\\Models\\iris_model.pth"
  T.save(net.state_dict(), fn)

  # saved_model = Net()
  # saved_model.load_state_dict(T.load(fn))
  # use saved_model to make prediction(s)

  print("\nEnd Iris demo")

if __name__ == "__main__":
  main()
This entry was posted in PyTorch. Bookmark the permalink.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s