# Category Archives: Transformers

## Regression (People Income) Using a PyTorch Neural Network with a Transformer Component

A regression problem is one where the goal is to predict a single numeric value. I’ve been looking at incorporating a Transformer component into a PyTorch neural network to see if the technique works or not. Bottom line: I got … Continue reading

Posted in PyTorch, Transformers | Leave a comment

## Binary Classification (People Gender) Using a PyTorch Neural Network with a Transformer Component

A binary classification problem is one where the goal is to predict a discrete value where there are only two possibilities. For example, you might want to predict the sex of a person based on their age, income, and so … Continue reading

Posted in PyTorch, Transformers | Leave a comment

## Anomaly Detection for Tabular Data Using a PyTorch Transformer with Numeric Embedding

I’ve been looking at unsupervised anomaly detection using a PyTorch Transformer module. My first set of experiments used the UCI Digits dataset because the inputs (64 pixels with values between 0 and 16 — a scaled down MNIST) are all … Continue reading

Posted in PyTorch, Transformers | Leave a comment

## Example of a PyTorch Multi-Class Classifier Using a Transformer

I’d been experimenting with the idea of using a Transformer module as the core of a multi-class classifier. The idea is rather weird because Transformer systems were designed to accept sequential information, such as a sequence of words, where order … Continue reading

Posted in PyTorch, Transformers | Leave a comment

## A Parameterized Not-Fully-Connected PyTorch Layer Example

I had recently been exploring the idea of creating a PyTorch neural network layer that’s not fully connected. After many hours of experiments, I got a demo working for the Iris dataset where the neural architecture was (4–8)-7-3 meaning there … Continue reading

Posted in PyTorch, Transformers | Leave a comment

## PyTorch Transformer System Naive Approach Classifier – Fail

The vast majority of my blog posts show a successful program of some sort. But behind the scenes, every successful program is preceded by many, many failures. For example, I’ve been looking at PyTorch Transformer modules. Transformer architecture is the … Continue reading

Posted in PyTorch, Transformers | Leave a comment

## A Naive Transformer Architecture for MNIST Classification Using PyTorch

Transformer architecture neural networks are very complex. Unlike some of my colleagues, I’m not a naturally brilliant guy. But my primary strength is persistence. I continue to probe the complexities of transformer systems, one example at a time, day after … Continue reading

Posted in PyTorch, Transformers | Leave a comment

## Transformer Based Dataset Similarity for MNIST

Computing a measure of the similarity of two datasets is a very difficult problem. Several months ago, I worked on a project and devised an algorithm based on a neural autoencoder. See jamesmccaffrey.wordpress.com/2021/04/02/dataset-similarity-sometimes-ideas-work-better-than-expected/. I wondered if I could adapt my … Continue reading

Posted in PyTorch, Transformers | Leave a comment

## The Distance Between Two Datasets Using Transformer Encoding

Several months ago I devised an algorithm that computes a value that represents the distance (difference) between two datasets. Computing the distance between two datasets is a remarkably difficult task — I consider it one of the unsolved fundamental problems … Continue reading

Posted in PyTorch, Transformers | Leave a comment

## Using the Simplest Possible Transformer Sequence-to-Sequence Example

I’ve been exploring PyTorch Transformer Architecture models sequence-to-sequence problems for several months. TA architecture systems are among the most complicated software things I’ve ever worked with. I recently completed a demo implementation of my idea of the simplest possible sequence-to-sequence. … Continue reading

Posted in PyTorch, Transformers | Leave a comment