Education

Attention in transformers, step-by-step | Deep Learning Chapter 6

📚 Main topics

From embeddings to contextA transformer starts with a vector for each token, but attention lets those vectors absorb information from surrounding words and represent richer, context-dependent meanings. 0:30
Queries, keys, and attention patternsQuery and key matrices create vectors whose dot products measure how relevant each token is to another. A column-wise softmax turns those scores into normalized weights called an attention pattern. 5:52
Masking and context sizeGPT-style causal masking prevents later tokens from influencing earlier ones by setting their scores to negative infinity before softmax. Since an attention pattern grows with the square of the context length, longer contexts are costly. 11:12
Values and embedding updatesA value map produces information that tokens can pass to other tokens. Attention weights determine how much of each value contributes to an update, which is added to the original embedding. 13:22
Multi-head attention and cross-attentionMultiple heads run distinct attention operations in parallel, allowing different kinds of contextual relationships to be learned. Cross-attention is a related setup where queries and keys come from different data sources. 18:03

✨ Key takeaways

Attention enables contextual meaningThe same token can take on different meanings depending on its context, and attention helps adjust its embedding accordingly. 2:04
Each head learns a different patternA head’s query, key, and value maps are learned from data; the adjective-to-noun example is only an illustration of behavior a head might learn. 5:19
Value maps are factored for efficiencyRather than using a large square matrix, the value map can be split into down and up projections, forming a low-rank transformation. 16:30
Attention is only part of a transformerTransformer layers also include other operations, such as multi-layer perceptrons, and repeated layers can build increasingly abstract representations. 23:23
Parallelism supports scaleAttention’s computations are highly parallelizable, making it well suited to GPUs and large-scale models. 24:27

🧠 Lessons learned

Weights indicate relevance, not fixed rulesAttention patterns show which tokens influence one another, but the model learns the specific patterns and updates that help predict the next token. 9:06
More heads mean more kinds of contextRunning many attention heads in parallel gives the model capacity to learn multiple distinct ways that context can change meaning. 20:11
Parameter counts add up quicklyIn the GPT-3 example, attention accounts for just under 58 billion parameters across its layers, about a third of the model’s total. 23:54

🏁 Conclusion/next steps

What comes nextThe next chapter will cover multi-layer perceptrons and more of the training process. The video also points viewers toward further resources on transformers, language models, and their history. 24:27

Transcript excerpt

0:00 In the last chapter, you and I started to step through the internal workings of a transformer. This is one of the key pieces of technology inside large language models, and a lot of other tools in the modern wave of AI. It first hit the scene in a now-famous 2017 paper called Attention is All You Need, and in this chapter you and I will dig into what this attention mechanism is, visualizing how it processes data. As a quick recap, here's the important context I want you to have in mind.

0:30 The goal of the model that you and I are studying is to take in a piece of text and predict what word comes next. The input text is broken up into little pieces that we call tokens, and these are very often words or pieces of words, but just to make the examples in this video easier for you and me to think about, let's simplify by pretending that tokens are always just words. The first step in a transformer is to associate each token with a high-dimensional vector, what we call its embedding. The most important idea I want you to have in mind is how directions in this

🔒 The full, searchable transcript is available with Pro.

Suggestions