A working knowledge of Pytorch<\/a> is required to understand the programming examples, but these can also be safely skipped.<\/p>\n\n\n\n The fundamental operation of any transformer architecture is the self-attention operation<\/em>.We’ll explain where the name “self-attention” comes from later. For now, don’t read too much in to it.<\/p>\n\n\n\n Self-attention is a sequence-to-sequence operation: a sequence of vectors goes in, and a sequence of vectors comes out. Let\u2019s call the input vectors \ud835\udc311,\ud835\udc312,\u2026,\ud835\udc31t and the corresponding output vectors \ud835\udc321,\ud835\udc322,\u2026,\ud835\udc32t. The vectors all have dimension k.<\/p>\n\n\n\n To produce output vector \ud835\udc32i, the self attention operation simply takes a weighted average over all the input vectors<\/em><\/p>\n\n\n\n \ud835\udc32i=\u2211jwij\ud835\udc31j.<\/p>\n\n\n\n Where j indexes over the whole sequence and the weights sum to one over all j. The weight wij is not a parameter, as in a normal neural net, but it is derived<\/em> from a function over \ud835\udc31i and \ud835\udc31j. The simplest option for this function is the dot product:<\/p>\n\n\n\n w\u2032ij=\ud835\udc31iT\ud835\udc31j.Note that \ud835\udc31i is the input vector at the same position as the current output vector \ud835\udc32i. For the next output vector, we get an entirely new series of dot products, and a different weighted sum.<\/p>\n\n\n\n The dot product gives us a value anywhere between negative and positive infinity, so we apply a softmax to map the values to [0,1] and to ensure that they sum to 1 over the whole sequence:<\/p>\n\n\n\n wij=exp w\u2032ij\u2211jexp w\u2032ij.<\/p>\n\n\n\n And that\u2019s the basic operation of self attention.<\/p>\n\n\n\n A few other ingredients are needed for a complete transformer, which we\u2019ll discuss later, but this is the fundamental operation. More importantly, this is the only operation in the whole architecture that propagates information between<\/em> vectors. Every other operation in the transformer is applied to each vector in the input sequence without interactions between vectors.<\/p>\n\n\n\n Despite its simplicity, it\u2019s not immediately obvious why self-attention should work so well. To build up some intuition, let\u2019s look first at the standard approach to movie recommendation<\/em>.<\/p>\n\n\n\n Let\u2019s say you run a movie rental business and you have some movies, and some users, and you would like to recommend movies to your users that they are likely to enjoy.<\/p>\n\n\n\n One way to go about this, is to create manual features for your movies, such as how much romance there is in the movie, and how much action, and then to design corresponding features for your users: how much they enjoy romantic movies and how much they enjoy action-based movies. If you did this, the dot product between the two feature vectors would give you a score for how well the attributes of the movie match what the user enjoys.<\/p>\n\n\n\n If the signs of a feature match for the user and the movie\u2014the movie is romantic and the user loves romance or the movie is unromantic<\/em> and the user hates romance\u2014then the resulting dot product gets a positive term for that feature. If the signs don\u2019t match\u2014the movie is romantic and the user hates romance or vice versa\u2014the corresponding term is negative.<\/p>\n\n\n\n Furthermore, the magnitudes<\/em> of the features indicate how much the feature should contribute to the total score: a movie may be a little romantic, but not in a noticeable way, or a user may simply prefer no romance, but be largely ambivalent.<\/p>\n\n\n\n Of course, gathering such features is not practical. Annotating a database of millions of movies is very costly, and annotating users with their likes and dislikes is pretty much impossible.<\/p>\n\n\n\n What happens instead is that we make the movie features and user features parameters<\/em> of the model. We then ask users for a small number of movies that they like and we optimize the user features and movie features so that their dot product matches the known likes.<\/p>\n\n\n\n Even though we don\u2019t tell the model what any of the features should mean, in practice, it turns out that after training the features do actually reflect meaningful semantics about the movie content.<\/p>\n\n\n\n See this lecture<\/a> for more details on recommender systems. For now, this suffices as an explanation of how the dot product helps us to represent objects and their relations.<\/p>\n\n\n\n This is the basic principle at work in the self-attention. Let\u2019s say we are faced with a sequence of words. To apply self-attention, we simply assign each word t in our vocabulary an embedding vector<\/em> \ud835\udc2ft (the values of which we\u2019ll learn). This is what\u2019s known as an embedding layer<\/em> in sequence modeling. It turns the word sequence the,cat,walks,on,the,street into the vector sequence<\/p>\n\n\n\n \ud835\udc2fthe,\ud835\udc2fcat,\ud835\udc2fwalks,\ud835\udc2fon,\ud835\udc2fthe,\ud835\udc2fstreet.<\/p>\n\n\n\n If we feed this sequence into a self-attention layer, the output is another sequence of vectors<\/p>\n\n\n\n \ud835\udc32the,\ud835\udc32cat,\ud835\udc32walks,\ud835\udc32on,\ud835\udc32the,\ud835\udc32streetwhere \ud835\udc32cat is a weighted sum over all the embedding vectors in the first sequence, weighted by their (normalized) dot-product with \ud835\udc2fcat.<\/p>\n\n\n\n Since we are learning<\/em> what the values in \ud835\udc2ft should be, how “related” two words are is entirely determined by the task. In most cases, the definite article the is not very relevant to the interpretation of the other words in the sentence; therefore, we will likely end up with an embedding \ud835\udc2fthe that has a low or negative dot product with all other words. On the other hand, to interpret what walks means in this sentence, it’s very helpful to work out who<\/em> is doing the walking. This is likely expressed by a noun, so for nouns like cat and verbs like walks, we will likely learn embeddings \ud835\udc2fcat and \ud835\udc2fwalks that have a high, positive dot product together.<\/p>\n\n\n\n This is the basic intuition behind self-attention. The dot product expresses how related two vectors in the input sequence are, with \u201crelated\u201d defined by the learning task, and the output vectors are weighted sums over the whole input sequence, with the weights determined by these dot products.<\/p>\n\n\n\n Before we move on, it\u2019s worthwhile to note the following properties, which are unusual for a sequence-to-sequence operation:<\/p>\n\n\n\n What I cannot create, I do not understand, as Feynman said. So we\u2019ll build a simple transformer as we go along. We\u2019ll start by implementing this basic self-attention operation in Pytorch.<\/p>\n\n\n\n The first thing we should do is work out how to express the self attention in matrix multiplications. A naive implementation that loops over all vectors to compute the weights and outputs would be much too slow.<\/p>\n\n\n\n We\u2019ll represent the input, a sequence of t vectors of dimension k as a t by k matrix \ud835\udc17. Including a minibatch dimension b, gives us an input tensor of size (b,t,k).<\/p>\n\n\n\n The set of all raw dot products w\u2032ij forms a matrix, which we can compute simply by multiplying \ud835\udc17 by its transpose:<\/p>\n\n\n\n Then, to turn the raw weights w\u2032ij into positive values that sum to one, we apply a row-wise<\/em> softmax:<\/p>\n\n\n\n Finally, to compute the output sequence, we just multiply the weight matrix by \ud835\udc17. This results in a batch of output matrices \ud835\udc18 of size That\u2019s all. Two matrix multiplications and one softmax gives us a basic self-attention.<\/p>\n\n\n\n The actual self-attention used in modern transformers relies on three additional tricks.<\/p>\n\n\n\n Every input vector \ud835\udc31i is used in three different ways in the self attention operation:<\/p>\n\n\n\n These roles are often called the query<\/strong>, the key<\/strong> and the value<\/strong> (we’ll explain where these names come from later). In the basic self-attention we’ve seen so far, each input vector must play all three roles. We make its life a little easier by deriving new vectors for each role, by applying a linear transformation to the original input vector. In other words, we add three k\u00d7k weight matrices \ud835\udc16q, \ud835\udc16k,\ud835\udc16v and compute three linear transformations of each xi, for the three different parts of the self attention:<\/p>\n\n\n\n \ud835\udc2ai=\ud835\udc16q\ud835\udc31i\ud835\udc24i=\ud835\udc16k\ud835\udc31i\ud835\udc2fi=\ud835\udc16v\ud835\udc31i<\/p>\n\n\n\n w\u2032ijwij\ud835\udc32i=\ud835\udc2aiT\ud835\udc24j=softmax(w\u2032ij)=\u2211jwij\ud835\udc2fj.<\/p>\n\n\n\n This gives the self-attention layer some controllable parameters, and allows it to modify the incoming vectors to suit the three roles they must play.<\/p>\n\n\n\n The softmax function can be sensitive to very large input values. These kill the gradient, and slow down learning, or cause it to stop altogether. Since the average value of the dot product grows with the embedding dimension k, it helps to scale the dot product back a little to stop the inputs to the softmax function from growing too large:<\/p>\n\n\n\n w\u2032ij=\ud835\udc2aiT\ud835\udc24jk\u2212\u2212\u221aWhy k\u2212\u2212\u221a? Imagine a vector in \u211dk with values all c. Its Euclidean length is k\u2212\u2212\u221ac. Therefore, we are dividing out the amount by which the increase in dimension increases the length of the average vectors.<\/p>\n\n\n\n Finally, we must account for the fact that a word can mean different things to different neighbours. Consider the following example.<\/p>\n\n\n\n mary,gave,roses,to,susan<\/p>\n\n\n\n We see that the word gave has different relations to different parts of the sentence. mary expresses who\u2019s doing the giving, roses expresses what\u2019s being given, and susan expresses who the recipient is.<\/p>\n\n\n\n In a single self-attention operation, all this information just gets summed together. The inputs \ud835\udc31mary and \ud835\udc31susan can influence the output \ud835\udc32gave by different amounts, depending on their dot-product with \ud835\udc31gave, but they can\u2019t influence it in different ways<\/em>. If, for instance, we want the information about who gave the roses and who received them to end up in different parts of \ud835\udc32gave, we need a little more flexibility.This leaves aside how we figure out who gave the roses. We can do that based on prior knowledge about Mary and Susan, encoded in the embeddings. We can also look at the order of the words, but we’ll look at how to achieve that later.<\/p>\n\n\n\n We can give the self attention greater power of discrimination, by combining several self-attention mechanisms (which we’ll index with r), each with different matrices \ud835\udc16rq, \ud835\udc16rk,\ud835\udc16rv. These are called attention heads<\/em>.<\/p>\n\n\n\n For input \ud835\udc31i each attention head produces a different output vector \ud835\udc32ri. We concatenate these, and pass them through a linear transformation to reduce the dimension back to k.<\/p>\n\n\n\n Efficient multi-head self-attention.<\/strong> The simplest way to understand multi-head self-attention is to see it as a small number of copies of the self-attention mechanism applied in parallel, each with their own key, value and query transformation. This works well, but for R heads, the self-attention operation is R times as slow.<\/p>\n\n\n\n It turns out we can have our cake and eat it too: there is a way to implement multi-head self-attention so that it is roughly as fast as the single-head version, but we still get the benefit of having different self-attention operations in parallel. To accomplish this, each head receives low-dimensional keys queries and values. If the input vector has k=256 dimensions, and we have h=4 attention heads, we multiply the input vectors by a 256\u00d764 matrix to project them down to a sequence of 64 dimansional vectors. For every head, we do this 3 times: for the keys, the queries and the values.<\/p>\n\n\n\n Here is the whole process illustrated in one image.<\/p>\n\n\n\n This requires 3h matrices of size k by k\/h. In total, this gives us 3hkkh=3k2 parameters to compute the inputs to the multi-head self-attention: the same as we had for the single-head self-attention.The only difference is the matrix Wo, used at the end of the multi-head self attention. This adds k2 parameters compared to the single-head version. In most transformers, the first thing that happens after each self attention is a feed-forward layer, so this may not be strictly necessary. I’ve never seen a proper ablation to test whether Wo can be removed.<\/p>\n\n\n\n We can even implement this with just three k\u00d7k matrix multiplications as in the single-head self-attention. The only extra operation we need is to slice the resulting sequence of vectors into chunks.<\/p>\n\n\n\n Let\u2019s now implement a self-attention module with all the bells and whistles. We\u2019ll package it into a Pytorch module, so we can reuse it later.<\/p>\n\n\n\n Note the assert: the embedding dimension needs to be divisible by the number of heads.<\/p>\n\n\n\n Next, we set up some linear transformations with We can now implement the computation of the self-attention (the module\u2019s This gives us three vector sequences of the full embedding dimension This simply reshapes the tensors to add a dimension that iterations over the heads. For a single vector in our sequence you can think of it as reshaping a vector of dimension Next, we need to compute the dot products. This is the same operation for every head, so we fold the heads into the batch dimension. This ensures that we can use Since the head and batch dimension are not next to each other, we need to transpose before we reshape. (This is costly, but it seems to be unavoidable.)<\/p>\n\n\n\n You can avoid these calls to As before, the dot products can be computed in a single matrix multiplication, but now between the queries and the keys.<\/p>\n\n\n\n We apply the self attention weights To unify the attention heads, we transpose again, so that the head dimension and the embedding dimension are next to each other, and reshape to get concatenated vectors of dimension And there you have it: multi-head, scaled dot-product self attention. You can see the complete implementation here<\/a>.The implementation can be made more concise using einsum notation<\/a> (see an example here<\/a>).<\/p>\n\n\n\n A transformer is not just a self-attention layer, it is an architecture<\/em>. It\u2019s not quite clear what does and doesn\u2019t qualify as a transformer, but here we\u2019ll use the following definition:<\/p>\n\n\n\n Any architecture designed to process a connected set of units\u2014such as the tokens in a sequence or the pixels in an image\u2014where the only interaction between units is through self-attention.<\/p>\n<\/blockquote>\n\n\n\n As with other mechanisms, like convolutions, a more or less standard approach has emerged for how to build self-attention layers up into a larger network. The first step is to wrap the self-attention into a block<\/em> that we can repeat.<\/p>\n\n\n\n There are some variations on how to build a basic transformer block, but most of them are structured roughly like this:<\/p>\n\n\n\n That is, the block applies, in sequence: a self attention layer, layer normalization, a feed forward layer (a single MLP applied independently to each vector), and another layer normalization. Residual connections are added around both, before the normalization. The order of the various components is not set in stone; the important thing is to combine self-attention with a local feedforward, and to add normalization and residual connections.Normalization and residual connections are standard tricks used to help deep neural networks train faster and more accurately. The layer normalization is applied over the embedding dimension only.<\/p>\n\n\n\n Here\u2019s what the transformer block looks like in pytorch.<\/p>\n\n\n\n We\u2019ve made the relatively arbitrary choice of making the hidden layer of the feedforward 4 times as big as the input and output. Smaller values may work as well, and save memory, but it should be bigger than the input\/output layers.<\/p>\n\n\n\n The simplest transformer we can build is a sequence classifier<\/em>. We\u2019ll use the IMDb sentiment classification dataset: the instances are movie reviews, tokenized into sequences of words, and the classification labels are The heart of the architecture will simply be a large chain of transformer blocks. All we need to do is work out how to feed it the input sequences, and how to transform the final output sequence into a a single classification.<\/p>\n\n\n\n The whole experiment can be found here<\/a>. We won\u2019t deal with the data wrangling in this blog post. Follow the links in the code to see how the data is loaded and prepared.<\/p>\n\n\n\n The most common way to build a sequence classifier out of sequence-to-sequence layers, is to apply global average pooling to the final output sequence, and to map the result to a softmaxed class vector.<\/p>\n\n\n\n We\u2019ve already discussed the principle of an embedding layer. This is what we\u2019ll use to represent the words.<\/p>\n\n\n\n However, as we\u2019ve also mentioned already, we\u2019re stacking permutation equivariant layers, and the final global average pooling is permutation in<\/em>variant, so the network as a whole is also permutation invariant. Put more simply: if we shuffle up the words in the sentence, we get the exact same classification, whatever weights we learn. Clearly, we want our state-of-the-art language model to have at least some sensitivity to word order, so this needs to be fixed.<\/p>\n\n\n\n The solution is simple: we create a second vector of equal length, that represents the position of the word in the current sentence, and add this to the word embedding. There are two options.position embeddings. We simply\u00a0embed<\/em>\u00a0the positions like we did the words. Just like we created embedding vectors\u00a0\ud835\udc2fcat\u00a0and\u00a0\ud835\udc2fsusan, we create embedding vectors\u00a0\ud835\udc2f12\u00a0and\u00a0\ud835\udc2f25. Up to however long we expect sequences to get. The drawback is that we have to see sequences of every length during training, otherwise the relevant position embeddings don’t get trained. The benefit is that it works pretty well, and it’s easy to implement.position encodings. Position encodings work in the same way as embeddings, except that we don’t\u00a0learn<\/em>\u00a0the position vectors, we just choose some function\u00a0f:\u2115\u2192\u211dk\u00a0to map the positions to real valued vectors, and let the network figure out how to interpret these encodings. The benefit is that for a well chosen function, the network should be able to deal with sequences that are longer than those it’s seen during training (it’s unlikely to perform well on them, but at least we can check). The drawbacks are that the choice of encoding function is a complicated hyperparameter, and it complicates the implementation a little.<\/p>\n\n\n\n For the sake of simplicity, we\u2019ll use position embeddings in our implementation.<\/p>\n\n\n\n Here is the complete text classification transformer in pytorch.<\/p>\n\n\n\n At depth 6, with a maximum sequence length of 512, this transformer achieves an accuracy of about 85%, competitive with results from RNN models, and much faster to train. To see the real near-human performance of transformers, we\u2019d need to train a much deeper model on much more data. More about how to do that later.<\/p>\n\n\n\n The next trick we\u2019ll try is an autoregressive<\/em> model. We\u2019ll train a character<\/em> level transformer to predict the next character in a sequence. The training regime is simple (and has been around for far longer than transformers have<\/a>). We give the sequence-to-sequence model a sequence, and we ask it to predict the next character at each point in the sequence. In other words, the target output is the same sequence shifted one character to the left:<\/p>\n\n\n\n With RNNs this is all we need to do, since they cannot look forward into the input sequence: output i depends only on inputs 0 to i. With a transformer, the output depends on the entire input sequence, so prediction of the next character becomes vacuously easy, just retrieve it from the input.<\/p>\n\n\n\n To use self-attention as an autoregressive model, we\u2019ll need to ensure that it cannot look forward into the sequence. We do this by applying a mask to the matrix of dot products, before the softmax is applied. This mask disables all elements above the diagonal of the matrix.<\/p>\n\n\n\n Since we want these elements to be zero after the softmax, we set them to \u2212\u221e. Here\u2019s how that looks in pytorch:<\/p>\n\n\n\n After we\u2019ve handicapped the self-attention module like this, the model can no longer look forward in the sequence.<\/p>\n\n\n\n We train on the standard We train on sequences of length 256, using a model of 12 transformer blocks and 256 embedding dimension. After about 24 hours training on an RTX 2080Ti (some 170K batches of size 32), we let the model generate from a 256-character seed: for each character, we feed it the preceding 256 characters, and look what it predicts for the next character (the last output vector). We sample from that with a temperature<\/a> of 0.5, and move to the next character.<\/p>\n\n\n\n The output looks like this:<\/p>\n\n\n\n 1228X Human & Rousseau. Because many of his stories were originally published in long-forgotten magazines and journals, there are a number of [[anthology|anthologies]] by different collators each containing a different selection. His original books have been considered an anthologie in the [[Middle Ages]], and were likely to be one of the most common in the [[Indian Ocean]] in the [[1st century]]. As a result of his death, the Bible was recognised as a counter-attack by the [[Gospel of Matthew]] (1177-1133), and the [[Saxony|Saxons]] of the [[Isle of Matthew]] (1100-1138), the third was a topic of the [[Saxony|Saxon]] throne, and the [[Roman Empire|Roman]] troops of [[Antiochia]] (1145-1148). The [[Roman Empire|Romans]] resigned in [[1148]] and [[1148]] began to collapse. The [[Saxony|Saxons]] of the [[Battle of Valasander]] reported the y<\/p>\n\n\n\n Note that the Wikipedia link tag syntax is correctly used, that the text inside the links represents reasonable subjects for links. Most importantly, note that there is a rough thematic consistency; the generated text keeps on the subject of the bible, and the Roman empire, using different related terms at different points. While this is far from the performance of a model like GPT-2<\/a>, the benefits over a similar RNN model are clear already: faster training (a similar RNN model would take many days to train) and better long-term coherence.In case you’re curious, the Battle of Valasander seems to be an invention of the network.<\/p>\n\n\n\n At this point, the model achieves a compression of 1.343 bits per byte on the validation set, which is not too far off the state of the art of 0.93 bits per byte, achieved by the GPT-2 model (described below).<\/p>\n\n\n\n To understand why transformers are set up this way, it helps to understand the basic design considerations that went into them. The main point of the transformer was to overcome the problems of the previous state-of-the-art architecture, the RNN (usually an LSTM or a GRU). Unrolled<\/a>, an RNN looks like this:<\/p>\n\n\n\n The big weakness here is the recurrent connection. while this allows information to propagate along the sequence, it also means that we cannot compute the cell at time step i until we\u2019ve computed the cell at timestep i\u22121. Contrast this with a 1D convolution:<\/p>\n\n\n\n In this model, every output vector can be computed in parallel with every other output vector. This makes convolutions much faster. The drawback with convolutions, however, is that they\u2019re severely limited in modeling long range dependencies<\/em>. In one convolution layer, only words that are closer together than the kernel size can interact with each other. For longer dependence we need to stack many convolutions.<\/p>\n\n\n\n The transformer is an attempt to capture the best of both worlds. They can model dependencies over the whole range of the input sequence just as easily as they can for words that are next to each other (in fact, without the position vectors, they can\u2019t even tell the difference). And yet, there are no recurrent connections, so the whole model can be computed in a very efficient feedforward fashion.<\/p>\n\n\n\n The rest of the design of the transformer is based primarily on one consideration: depth. Most choices follow from the desire to train big stacks of transformer blocks. Note for instance that there are only two places in the transformer where non-linearities occur: the softmax in the self-attention and the ReLU in the feedforward layer. The rest of the model is entirely composed of linear transformations, which perfectly preserve the gradient.I suppose the layer normalization is also nonlinear, but that is one nonlinearity that actually helps to keep the gradient stable as it propagates back down the network.<\/p>\n\n\n\n If you\u2019ve read other introductions to transformers, you may have noticed that they contain some bits I\u2019ve skipped. I think these are not necessary to understand modern transformers. They are, however, helpful to understand some of the terminology and some of the writing about<\/em> modern transformers. Here are the most important ones.<\/p>\n\n\n\n Before self-attention was first presented, sequence models consisted mostly of recurrent networks or convolutions stacked together. At some point, it was discovered that these models could be helped by adding attention mechanisms<\/em>: instead of feeding the output sequence of the previous layer directly to the input of the next, an intermediate mechanism was introduced, that decided which elements of the input were relevant for a particular word of the output.<\/p>\n\n\n\n The general mechanism was as follows. We call the input the values<\/strong>. Some (trainable) mechanism assigns a key<\/strong> to each value. Then to each output, some other mechanism assigns a query<\/strong>.<\/p>\n\n\n\n These names derive from the datastructure of a key-value store. In that case we expect only one item in our store to have a key that matches the query, which is returned when the query is executed. Attention is a softened version of this: every<\/em> key in the store matches the query to some extent. All are returned, and we take a sum, weighted by the extent to which each key matches the query.<\/p>\n\n\n\n The great breakthrough of self-attention was that attention by itself is a strong enough mechanism to do all the learning. Attention is all you need<\/a>, as the authors put it. The key, query and value are all the same vectors (with minor linear transformations). They attend to themselves<\/em> and stacking such self-attention provides sufficient nonlinearity and representational power to learn very complicated functions.<\/p>\n\n\n\n But the authors did not dispense with all the complexity of contemporary sequence modeling. The standard structure of sequence-to-sequence models in those days was an encoder-decoder architecture, with teacher forcing<\/a>.<\/p>\n\n\n\n The encoder<\/em> takes the input sequence and maps it to a latent<\/em> representation of the whole sequence. This can be either a sequence of latent vectors, or a single one as in the image above. This vector is then passed to a decoder<\/em> which unpacks it to the desired target sequence (for instance, the same sentence in another language).<\/p>\n\n\n\n Teacher forcing<\/em> refers to the technique of also allowing the decoder access to the input sentence, but in an autoregressive fashion. That is, the decoder generates the output sentence word for word based both on the latent vector and the words it has already generated. This takes some of the pressure off the latent representation: the decoder can use word-for-word sampling to take care of the low-level structure like syntax and grammar and use the latent vector to capture more high-level semantic structure. Decoding twice with the same latent representation would, ideally, give you two different sentences with the same meaning.<\/p>\n\n\n\n In later transformers, like BERT and GPT-2, the encoder\/decoder configuration was entirely dispensed with. A simple stack of transformer blocks was found to be sufficient to achieve state of the art in many sequence based tasks.This approach is sometimes called a decoder-only transformer (for an autoregressive model) or an encoder-only transformer (for a model without masking).<\/p>\n\n\n\n Here\u2019s a small selection of some modern transformers and their most characteristic details.<\/p>\n\n\n\n BERT was one of the first models to show that transformers could reach human-level performance on a variety of language based tasks: question answering, sentiment classification or classifying whether two sentences naturally follow one another.<\/p>\n\n\n\n BERT consists of a simple stack of transformer blocks, of the type we\u2019ve described above. This stack is pre-trained<\/em> on a large general-domain corpus consisting of 800M words from English books (modern work, from unpublished authors), and 2.5B words of text from English Wikipedia articles (without markup).<\/p>\n\n\n\n Pretraining is done through two tasks:MaskingA certain number of words in the input sequence are: masked out, replaced with a random word or kept as is. The model is then asked to predict, for these words, what the original words were. Note that the model doesn’t need to predict the entire denoised sentence, just the modified words. Since the model doesn’t know which words it will be asked about, it learns a representation for every word in the sequence.Next sequence classificationTwo sequences of about 256 words are sampled that either (a) follow each other directly in the corpus, or (b) are both taken from random places. The model must then predict whether a or b is the case.<\/p>\n\n\n\n BERT uses WordPiece tokenization, which is somewhere in between word-level and character level sequences. It breaks words like walking up into the tokens walk and ##ing. This allows the model to make some inferences based on word structure: two verbs ending in -ing have similar grammatical functions, and two verbs starting with walk- have similar semantic function.<\/p>\n\n\n\n The input is prepended with a special <cls> token. The output vector corresponding to this token is used as a sentence representation in sequence classification tasks like the next sentence classification (as opposed to the global average pooling over all vectors that we used in our classification model above).<\/p>\n\n\n\n After pretraining, a single task-specific layer is placed after the body of transformer blocks, which maps the general purpose representation to a task specific output. For classification tasks, this simply maps the first output token to softmax probabilities over the classes. For more complex tasks, a final sequence-to-sequence layer is designed specifically for the task.<\/p>\n\n\n\n The whole model is then re-trained to finetune the model for the specific task at hand.<\/p>\n\n\n\n In an ablation experiment, the authors show that the largest improvement as compared to previous models comes from the bidirectional nature of BERT. That is, previous models like GPT used an autoregressive mask, which allowed attention only over previous tokens. The fact that in BERT all attention is over the whole sequence is the main cause of the improved performance.This is why the B in BERT stands for “bidirectional”.<\/p>\n\n\n\n The largest BERT model uses 24 transformer blocks, an embedding dimension of 1024 and 16 attention heads, resulting in 340M parameters.<\/p>\n\n\n\n GPT-2 is the first transformer model that actually made it into the mainstream news<\/a>, after the controversial decision by OpenAI not to release the full model.The reason was that GPT-2 could generate sufficiently believable text that large-scale fake news campaigns of the kind seen in the 2016 US presidential election would become effectively a one-person job.<\/p>\n\n\n\n The first trick that the authors of GPT-2 employed was to create a new high-quality dataset. While BERT used high-quality data, their sources (lovingly crafted books and well-edited wikipedia articles) had a certain lack of diversity in the writing style. To collect more diverse data without sacrificing quality the authors used links posted on the social media site Reddit<\/em> to gather a large collection of writing with a certain minimum level of social support (expressed on Reddit as karma<\/em>).<\/p>\n\n\n\n GPT2 is fundamentally a language generation<\/em> model, so it uses masked self-attention like we did in our model above. It uses byte-pair encoding to tokenize the language, which , like the WordPiece encoding breaks words up into tokens that are slightly larger than single characters but less than entire words.<\/p>\n\n\n\n GPT2 is built very much like our text generation model above, with only small differences in layer order and added tricks to train at greater depths. The largest model uses 48 transformer blocks, a sequence length of 1024 and an embedding dimension of 1600, resulting in 1.5B parameters.<\/p>\n\n\n\n They show state-of-the art performance on many tasks. On the wikipedia compression task that we tried above, they achieve 0.93 bits per byte.<\/p>\n\n\n\n While the transformer represents a massive leap forward in modeling long-range dependency, the models we have seen so far are still fundamentally limited by the size of the input. Since the size of the dot-product matrix grows quadratically in the sequence length, this quickly becomes the bottleneck as we try to extend the length of the input sequence. Transformer-XL is one of the first succesful transformer models to tackle this problem.<\/p>\n\n\n\n During training, a long sequence of text (longer than the model could deal with) is broken up into shorter segments. Each segment is processed in sequence, with self-attention computed over the tokens in the curent segment and the previous segment<\/em>. Gradients are only computed over the current segment, but information still propagates as the segment window moves through the text. In theory at layer n, information may be used from n segments ago.A similar trick in RNN training is called truncated backpropagation through time. We feed the model a very long sequence, but backpropagate only over part of it. The first part of the sequence, for which no gradients are computed, still influences the values of the hidden states in the part for which they are.<\/p>\n\n\n\n To make this work, the authors had to let go of the standard position encoding\/embedding scheme. Since the position encoding is absolute<\/em>, it would change for each segment and not lead to a consistent embedding over the whole sequence. Instead they use a relative<\/em> encoding. For each output vector, a different sequence of position vectors is used that denotes not the absolute position, but the distance to the current output.<\/p>\n\n\n\n This requires moving the position encoding into the attention mechanism (which is detailed in the paper). One benefit is that the resulting transformer will likely generalize much better to sequences of unseen length.<\/p>\n\n\n\nSelf-attention<\/h2>\n\n\n\n
![\"\"\/](\"https:\/\/peterbloem.nl\/files\/transformers\/self-attention.svg\")
Understanding why self-attention works<\/h3>\n\n\n\n
![\"\"\/](\"https:\/\/peterbloem.nl\/files\/transformers\/dot-product.svg\")
<\/figure>\n\n\n\n![\"\"\/](\"https:\/\/peterbloem.nl\/files\/transformers\/movie-features.svg\")
\n
In Pytorch: basic self-attention<\/h3>\n\n\n\n
import torch\nimport torch.nn.functional as F\n\n# assume we have some tensor x with size (b, t, k)\nx = ...\n\nraw_weights = torch.bmm(x, x.transpose(1, 2))\n# - torch.bmm is a batched matrix multiplication. It\n# applies matrix multiplication over batches of\n# matrices.<\/code><\/pre>\n\n\n\nweights = F.softmax(raw_weights, dim=2)<\/code><\/pre>\n\n\n\n(b, t, k)<\/code> whose rows are weighted sums over the rows of \ud835\udc17.<\/p>\n\n\n\ny = torch.bmm(weights, x)<\/code><\/pre>\n\n\n\nAdditional tricks<\/h3>\n\n\n\n
1) Queries, keys and values<\/h4>\n\n\n\n
\n
![\"\"\/](\"https:\/\/peterbloem.nl\/files\/transformers\/key-query-value.svg\")
2) Scaling the dot product<\/h4>\n\n\n\n
3) Multi-head attention<\/h4>\n\n\n\n
![\"A](\"https:\/\/peterbloem.nl\/files\/transformers\/multi-head.svg\")
![\"A](\"https:\/\/peterbloem.nl\/files\/transformers\/kqv-computation.svg\")
In Pytorch: complete self-attention<\/h3>\n\n\n\n
import<\/strong> torch\nfrom<\/strong> torch import<\/strong> nn\nimport<\/strong> torch.nn.functional as<\/strong> F\n\nclass<\/strong> SelfAttention<\/strong>(nn.Module):\n def<\/strong> __init__<\/strong>(self, k, heads=4, mask=False):\n \n super().__init__()\n \n assert<\/strong> k % heads == 0\n \n self.k, self.heads = k, heads\n<\/code><\/pre>\n\n\n\nemb<\/code> by emb<\/code> matrices. The nn.Linear<\/code> module with the bias disabled gives us such a projection, and provides a reasonable initialization for us.<\/p>\n\n\n\n \n # These compute the queries, keys and values for all\n # heads\n self.tokeys = nn.Linear(k, k, bias=False)\n self.toqueries = nn.Linear(k, k, bias=False)\n self.tovalues = nn.Linear(k, k, bias=False)\n\n\t# This will be applied after the multi-head self-attention operation.\n self.unifyheads = nn.Linear(k, k)\n<\/code><\/pre>\n\n\n\nforward<\/code> function). First, we compute the queries, keys and values for all heads:<\/p>\n\n\n\n def<\/strong> forward<\/strong>(self, x):\n\n b, t, k = x.size()\n h = self.heads\n\n queries = self.toqueries(x)\n keys = self.tokeys(x) \n values = self.tovalues(x)\n<\/code><\/pre>\n\n\n\nk<\/code>. As we saw above we can now cut these into h<\/code> chunks. we can do this with a simple view operation:<\/p>\n\n\n\n\n\ts = k \/\/ h\n\n\tkeys = keys.view(b, t, h, s)\n\tqueries = queries.view(b, t, h, s)\n\tvalues = values.view(b, t, h, s)\n<\/code><\/pre>\n\n\n\nk<\/code> into a matrix of h<\/code> by k\/\/h<\/code>:<\/p>\n\n\n\n![\"\"\/](\"https:\/\/peterbloem.nl\/files\/transformers\/reshape.svg\")
<\/figure>\n\n\n\ntorch.bmm()<\/code> as before, and the whole collection of keys, queries and values will just be seen as a slightly larger batch.<\/p>\n\n\n\n # - fold heads into the batch dimension\n keys = keys.transpose(1, 2).contiguous().view(b * h, t, s)\n queries = queries.transpose(1, 2).contiguous().view(b * h, t, s)\n values = values.transpose(1, 2).contiguous().view(b * h, t, s)\n<\/code><\/pre>\n\n\n\ncontiguous()<\/code> by using reshape()<\/code> instead of view()<\/code> but I prefer to make it explicit when we are copying a tensor, and when we are just viewing it. See this notebook<\/a> for an explanation of the difference.<\/p>\n\n\n\n \n # Get dot product of queries and keys, and scale\n dot = torch.bmm(queries, keys.transpose(1, 2))\n # -- dot has size (b*h, t, t) containing raw weights\n\n # scale the dot product\n dot = dot \/ (s ** (1\/2))\n\t\n # normalize \n dot = F.softmax(dot, dim=2)\n # - dot now contains row-wise normalized weights\n<\/code><\/pre>\n\n\n\ndot<\/code> to the values, giving us the output for each attention head<\/p>\n\n\n\n \n # apply the self attention to the values\n out = torch.bmm(dot, values).view(b, h, t, s)\n<\/code><\/pre>\n\n\n\ne<\/code>. We then pass these through the unifyheads<\/code> layer for a final projection.<\/p>\n\n\n\n # swap h, t back, unify heads\n out = out.transpose(1, 2).contiguous().view(b, t, s * h)\n \n return<\/strong> self.unifyheads(out)\n<\/code><\/pre>\n\n\n\nBuilding transformers<\/em><\/h2>\n\n\n\n
\n
The transformer block<\/h3>\n\n\n\n
![\"\"\/](\"https:\/\/peterbloem.nl\/files\/transformers\/transformer-block.svg\")
<\/figure>\n\n\n\nclass<\/strong> TransformerBlock<\/strong>(nn<\/strong>.Module<\/strong>):\n def<\/strong> __init__<\/strong>(self<\/strong>, k, heads):\n super<\/strong>().__init__()\n\n self<\/strong>.attention = SelfAttention(k, heads=heads)\n\n self<\/strong>.norm1 = nn.LayerNorm(k)\n self<\/strong>.norm2 = nn.LayerNorm(k)\n\n self<\/strong>.ff = nn.Sequential(\n nn.Linear(k, 4 * k),\n nn.ReLU(),\n nn.Linear(4 * k, k))\n\n def<\/strong> forward<\/strong>(self<\/strong>, x):\n attended = self<\/strong>.attention(x)\n x = self<\/strong>.norm1(attended + x)\n\n fedforward = self<\/strong>.ff(x)\n return<\/strong> self<\/strong>.norm2(fedforward + x)<\/code><\/pre>\n\n\n\nClassification transformer<\/h3>\n\n\n\n
positive<\/code> and negative<\/code> (indicating whether the review was positive or negative about the movie).<\/p>\n\n\n\nOutput: producing a classification<\/h4>\n\n\n\n
![\"\"\/](\"https:\/\/peterbloem.nl\/files\/transformers\/classifier.svg\")
Input: using the positions<\/h4>\n\n\n\n
Pytorch<\/h4>\n\n\n\n
class<\/strong> Transformer<\/strong>(nn.Module):\n def<\/strong> __init__<\/strong>(self, k, heads, depth, seq_length, num_tokens, num_classes):\n super().__init__()\n\n self.num_tokens = num_tokens\n self.token_emb = nn.Embedding(num_tokens, k)\n self.pos_emb = nn.Embedding(seq_length, k)\n\n\t\t# The sequence of transformer blocks that does all the\n\t\t# heavy lifting\n tblocks = []\n for<\/strong> i in<\/strong> range(depth):\n tblocks.append(TransformerBlock(k=k, heads=heads))\n self.tblocks = nn.Sequential(*tblocks)\n\n\t\t# Maps the final output sequence to class logits\n self.toprobs = nn.Linear(k, num_classes)\n\n def<\/strong> forward<\/strong>(self, x):\n \"\"\"\n :param x: A (b, t) tensor of integer values representing\n words (in some predetermined vocabulary).\n :return: A (b, c) tensor of log-probabilities over the\n classes (where c is the nr. of classes).\n \"\"\"\n\t\t# generate token embeddings\n tokens = self.token_emb(x)\n b, t, k = tokens.size()\n\n\t\t# generate position embeddings\n\t\tpositions = torch.arange(t)\n positions = self.pos_emb(positions)[None, :, :].expand(b, t, k)\n\n x = tokens + positions\n x = self.tblocks(x)\n\n # Average-pool over the t dimension and project to class\n # probabilities\n x = self.toprobs(x.mean(dim=1))\n return<\/strong> F.log_softmax(x, dim=1)\n<\/code><\/pre>\n\n\n\nText generation transformer<\/h3>\n\n\n\n
![\"\"\/](\"https:\/\/peterbloem.nl\/files\/transformers\/generator.svg\")
<\/figure>\n\n\n\n![\"\"\/](\"https:\/\/peterbloem.nl\/files\/transformers\/masked-attention.svg\")
dot = torch.bmm(queries, keys.transpose(1, 2))\n\nindices = torch.triu_indices(t, t, offset=1)\ndot[:, indices[0], indices[1]] = float('-inf')\n\ndot = F.softmax(dot, dim=2)\n<\/code><\/pre>\n\n\n\nenwik8<\/code> dataset (taken from the Hutter prize<\/a>), which contains 108 characters of Wikipedia text (including markup). During training, we generate batches by randomly sampling subsequences from the data.<\/p>\n\n\n\nDesign considerations<\/h2>\n\n\n\n
![\"\"\/](\"https:\/\/peterbloem.nl\/files\/transformers\/recurrent-connection.svg\")
<\/figure>\n\n\n\n![\"\"\/](\"https:\/\/peterbloem.nl\/files\/transformers\/convolutional-connection.svg\")
<\/figure>\n\n\n\nHistorical baggage<\/h2>\n\n\n\n
Why is it called self-attention<\/em>?<\/h3>\n\n\n\n
The original transformer: encoders and decoders<\/h3>\n\n\n\n
![\"\"\/](\"https:\/\/peterbloem.nl\/files\/transformers\/encoder-decoder.svg\")
<\/figure>\n\n\n\nModern transformers<\/h2>\n\n\n\n
BERT<\/a><\/h3>\n\n\n\n
GPT-2<\/a><\/h3>\n\n\n\n
Transformer-XL<\/a><\/h3>\n\n\n\n