Extended Transformers

Extended Transformers

The main type of Transformer that gets all the hype is the Generative Pre-Trained Transformer (GPT). This is the basic text processing Transformer that can process words and generate output with surprisingly human-like elegance.

Modern research has been applying Transformers to other types of input and uses cases. The result has been various new extensions of Transformer architectures.

• Multi-modal Transformer. This refers to Transformers that can accept inputs in images (or video) rather than simple text prompts.
• Vision Transformer (ViT). These are the use of Transformer technologies for computer vision applications, such as self-driving cars.
• Bidirectional Transformer. This is a research type used in the past, that hasn't received as much attention lately. The idea is that it can examine its input data from both directions at the same time. The main example is “Bidirectional Encoder Representations from Transformers” (BERT) and its many variants.
• Retrieval Augmentation Generation (RAG). This is an architecture where a Transformer is combined with a separate component that “retrieves” extra data (e.g. a document search mechanism). The idea is to extend the AI engine to new data without extra training.
• Ensemble inference. An “ensemble cast” is a Hollywood term that means a film with a group of famous actors all starring together in the same story. Someone with a sense of humor (or very large ambitions) decided to use the same term for a group of AI models all working together to create the same masterpiece.

Some of the major areas of Transformer research involve addressing the resource-hungry nature of their execution. For example, a basic Transformer has quadratic cost complexity in terms of the input length. Hence, there are numerous modifications in Transformer architectures being created, both in industry and research labs. See Part VII of this book for a full literature review of the extensive body of research related to Transformers.

• Next: • Up: Table of Contents • Buy: Generative AI in C++: Coding Transformers and LLMs

The new AI programming book by Aussie AI co-founders: AI coding in C++ Transformer engine speedups LLM models Phone and desktop AI Code examples Research citations Get your copy from Amazon: Generative AI in C++