AI glossary
What is a transformer in AI?
The transformer is the neural network architecture behind almost every current LLM. Its key piece, attention, relates each token to all the others.
It was introduced in 2017, in a paper by Google researchers titled "Attention Is All You Need". Until then, language models read text one word at a time and struggled to remember what came far back.
The attention mechanism means that, to process each token, the model looks at all the others and decides how much each one matters. That is how it works out that in "the dog did not cross the street because it was tired", the one that was tired is the dog. And because it processes tokens in parallel, it can be trained on GPUs at a scale that was not possible before.
That is where the T in GPT comes from: Generative Pre-trained Transformer.
An example
Claude, GPT, Gemini and Llama are all transformers inside, with different sizes, data and tuning.