Category Archives: PyTorch

An Example of Warm-Start Training for PyTorch

When you train a neural network from scratch, the weights and biases of the network are initialized with random values. In warm-start training, instead of using random values for initialization, you use the weights and biases of a different trained … Continue reading

Posted in PyTorch | Leave a comment

A Quick Look at PyTorch Audio

A few days ago, a work colleague asked me for some advice on machine learning with audio. My quick reply was that I know very little about working with audio data. So, I pointed him to some other ML experts. … Continue reading

Posted in PyTorch | Leave a comment

Fashion-MNIST Classification Example Using PyTorch

The MNIST (Modified National Institute of Standards and Technology) dataset has 60,000 training and 10,000 test data items. Each item is a crude image of a handwritten digit from ‘0’ to ‘9’. Each image is 28 by 28 pixels, and … Continue reading

Posted in PyTorch | Leave a comment

PyTorch Custom Weight Initialization Example

Neural network weight and bias initialization is a surprisingly tricky topic. In most cases, the default initialization code works fine, even though it’s poorly documented and the source code is extremely complex. Just to test my understanding of weight and … Continue reading

Posted in PyTorch | Leave a comment

PyTorch Transformer Sequence-to-Sequence: Good Examples are Hard to Find

I’ve been looking at deep neural Transformer Architecture (TA) systems for several months. In terms of conceptual ideas and engineering details, they are probably the most complex software systems I’ve ever worked with. Update: A few weeks after I wrote … Continue reading

Posted in PyTorch, Transformers | Leave a comment

A Custom Embedding Layer for Numeric Input for PyTorch

Transformer architecture (TA) neural networks were designed for natural language processing (NLP). I’ve been exploring the idea of applying TA to tabular data. The problem is that in NLP all inputs are integers that represent words/tokens. For example, an input … Continue reading

Posted in PyTorch | Leave a comment

Converting Fashion-MNIST Binary Files to Text Files

The MNIST (Modified National institute of Standards and Technology) dataset contains images of handwritten digits from ‘0’ to ‘9’. Each image is 28 by 28 pixels and each pixel is a grayscale value between 0 and 255. The MNIST data … Continue reading

Posted in PyTorch | Leave a comment

Transformer Based Anomaly Detection for Tabular Data

I recently explored an anomaly detection system based on a Transformer encoder and reconstruction error. The inputs for my example were the UCI Digits dataset items. Each data item is 64 pixel values between 0 and 16. I used UCI … Continue reading

Posted in Transformers | Leave a comment

PyTorch Word Embedding Layer from Scratch

The PyTorch neural library has a torch.nn.Embedding() layer that converts a word integer token to a vector. For example, “the” = 5 might be converted to a vector like [0.1234, -1.1044, 0.9876, 1.0234], assuming the embed_dim = 4. The values … Continue reading

Posted in PyTorch | Leave a comment

PyTorch Transformers and the torch.set_num_threads() Function

Bottom line: Using the torch.set_num_threads() in a PyTorch program that has a Transformer module can significantly change the behavior of the program (in my case, for the better). I was experimenting with a PyTorch program that uses a TransformerEncoder to … Continue reading

Posted in PyTorch | Leave a comment