Category Archives: PyTorch

When you train a neural network from scratch, the weights and biases of the network are initialized with random values. In warm-start training, instead of using random values for initialization, you use the weights and biases of a different trained … Continue reading →

A few days ago, a work colleague asked me for some advice on machine learning with audio. My quick reply was that I know very little about working with audio data. So, I pointed him to some other ML experts. … Continue reading →

The MNIST (Modified National Institute of Standards and Technology) dataset has 60,000 training and 10,000 test data items. Each item is a crude image of a handwritten digit from ‘0’ to ‘9’. Each image is 28 by 28 pixels, and … Continue reading →

Neural network weight and bias initialization is a surprisingly tricky topic. In most cases, the default initialization code works fine, even though it’s poorly documented and the source code is extremely complex. Just to test my understanding of weight and … Continue reading →

Transformer architecture (TA) neural networks were designed for natural language processing (NLP). I’ve been exploring the idea of applying TA to tabular data. The problem is that in NLP all inputs are integers that represent words/tokens. For example, an input … Continue reading →

The MNIST (Modified National institute of Standards and Technology) dataset contains images of handwritten digits from ‘0’ to ‘9’. Each image is 28 by 28 pixels and each pixel is a grayscale value between 0 and 255. The MNIST data … Continue reading →

The PyTorch neural library has a torch.nn.Embedding() layer that converts a word integer token to a vector. For example, “the” = 5 might be converted to a vector like [0.1234, -1.1044, 0.9876, 1.0234], assuming the embed_dim = 4. The values … Continue reading →

Bottom line: Using the torch.set_num_threads() in a PyTorch program that has a Transformer module can significantly change the behavior of the program (in my case, for the better). I was experimenting with a PyTorch program that uses a TransformerEncoder to … Continue reading →