Category Archives: Transformers

Using the Simplest Possible Transformer Sequence-to-Sequence Example

I’ve been exploring PyTorch Transformer Architecture models sequence-to-sequence problems for several months. TA architecture systems are among the most complicated software things I’ve ever worked with. I recently completed a demo implementation of my idea of the simplest possible sequence-to-sequence. … Continue reading

Posted in PyTorch, Transformers | Leave a comment

The Simplest Possible PyTorch Transformer Sequence-to-Sequence Example

I’ve been looking at PyTorch transformer architecture (TA) networks. TA networks are among the most complex software components I’ve ever worked with, in terms of both conceptual complexity and engineering difficulty. I set out to implement the simplest possible transformer … Continue reading

Posted in PyTorch, Transformers | Leave a comment

PyTorch Transformer Sequence-to-Sequence: Good Examples are Hard to Find

I’ve been looking at deep neural Transformer Architecture (TA) systems for several months. In terms of conceptual ideas and engineering details, they are probably the most complex software systems I’ve ever worked with. Update: A few weeks after I wrote … Continue reading

Posted in PyTorch, Transformers | Leave a comment

Transformer Based Anomaly Detection for Tabular Data

I recently explored an anomaly detection system based on a Transformer encoder and reconstruction error. The inputs for my example were the UCI Digits dataset items. Each data item is 64 pixel values between 0 and 16. I used UCI … Continue reading

Posted in Transformers | Leave a comment

Transformer Based Reconstruction Error Anomaly Detection

I’ve been experimenting with Transformer Architecture (TA) neural networks for several months. I reached a milestone recently when I created an end-to-end demo of using PyTorch TA for unsupervised anomaly detection. Briefly, source data is fed to a TA network … Continue reading

Posted in PyTorch, Transformers | Leave a comment

IMDB Classification using PyTorch Transformer Architecture

I have been exploring Transformer Architecture for natural language processing. I reached a big milestone when I put together a successful demo of the IMDB dataset problem using a PyTorch TransformerEncoder network. As is often the case, once I had … Continue reading

Posted in PyTorch, Transformers | Leave a comment

Positional Encoding for PyTorch Transformer Architecture Models

A Transformer Architecture (TA) model is most often used for natural language sequence-to-sequence problems. One example is language translation, such as translating English to Latin. A TA network is usually constructed from a built-in library Embedding layer, a program-defined Positional … Continue reading

Posted in Transformers | Leave a comment

PyTorch Transformer Layer Input-Output

The PyTorch neural library has a Transformer layer that can be used to construct a Transformer Architecture (TA) model. Typically, a library-defined Embedding layer, and a program-defined Positional layer, and a library-defined Linear layer are combined with a library-defined Transformer … Continue reading

Posted in Transformers | Leave a comment