Assigning Fixed Weight and Bias Values to a PyTorch Neural Network

Sometimes it’s useful to assign fixed weight and bias values to a neural network. To do so requires a knowledge of how those values are stored.

I wrote a short demo program to illustrate the technique. The demo creates a 3-4-2 neural network. The single hidden layer is named hid1 and has a total of 3 x 4 = 12 weights and 4 biases. PyTorch sores the weight values in a 4×3 shaped matrix named self.hid1.weight.data. The biases values are stored in self.hid1.bias.data.

Similarly, the output layer is named oupt and has a total of 4 x 2 = 8 weights and 2 biases. They’re stored in a 2×4 shaped matrix named self.oupt.weight.data and self.oupt.bias.data.

The demo code iterates through the weights and biases and stores 0.01, 0.02, . . 0.26 into the network.

The diagram above shows the conceptual view of the neural network, and a representation of the weight and bias data structures.

Important note: My demo sets the values of the weights and biases in the Net class __init__() method. If you want to modify the weights and biases of a neural network after the network has been instantiated, you need to do so inside a torch.no_grad() block so that the gradients don’t get messed up.

---

Software system conceptual diagrams are a facade over the reality and complexity of the underlying code. Here are three remarkable building facades that give a flat wall the appearance of 3D complexity.

---

Demo code.

# layer_assign_wts.py

# PyTorch 1.10.0-CPU Anaconda3-2020.02  Python 3.7.6
# Windows 10 

import torch as T
device = T.device("cpu")  # apply to Tensor or Module

class Net(T.nn.Module):
  def __init__(self):
    super(Net, self).__init__()
    self.hid1 = T.nn.Linear(3, 4)  # 3-4-2
    self.oupt = T.nn.Linear(4, 2)

    v = 0.01

    for i in range(4):      # hid1 4x3
      for j in range(3):
        self.hid1.weight.data[i][j] = v
        v += 0.01
    for i in range(4):
      self.hid1.bias.data[i] = v
      v += 0.01

    for i in range(2):      # oupt 2x4
      for j in range(4):
        self.oupt.weight.data[i][j] = v
        v += 0.01
    for i in range(2):
      self.oupt.bias.data[i] = v
      v += 0.01

  def forward(self, x):
    z = T.tanh(self.hid1(x))
    z = self.oupt(z)  # no softmax for CrossEntropyLoss() 
    return z

def main():
  print("\nBegin ")
  T.manual_seed(1)

  print("\nCreating a 3-4-2 network with fixed wts and biases ")
  net = Net().to(device)

  print("\nhid1 wts and biases: ")
  print(net.hid1.weight.data)
  print(net.hid1.bias.data)

  print("\noupt wts and biases: ")
  print(net.oupt.weight.data)
  print(net.oupt.bias.data)

  print("\nEnd ")

if __name__ == "__main__":
  main()