fast.ai: Deep Learning Part 1 (v2) (2018)
Today is a very busy lesson! We first learn how to interpret the collaborative filtering embeddings we created last week, and use that knowledge to answer the question: “what is the worst movie of all time?”
Then we cover what is perhaps the most practically important topic in the whole course: how to use deep learning to model “structured data” such as database tables and spreadsheets, as well as time series. It turns out that not only is deep learning often the most accurate modeling approach for this tasks, it can be the easiest approach to develop too.
We close out the lesson with an introduction to recurrent neural networks (RNNs), and use an RNN to write a new philosophical treatise…
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Welcome back less than six, so this is our penultimate lesson and believe it or not a couple of weeks ago in Lesson four I mentioned, I was going to share that lesson with this terrific. You know P researcher Sebastian Reuter, which I did and he he said he loved it and he's gone on to yesterday released this new post. He called optimization for deep learning highlights in 2017, in which he covered, basically everything that we talked about in that lesson and with some very nice shout outs to some of the work that some of the students here have done, including when he talked about this separation Of the separation of weight decay from the momentum term, and so he actually mentions here the opportunities in terms of improved kind of software decoupling. This allows and actually links to the commits from an answer hah actually showing how to implement this in fastai. So, first a eyes code is actually being used as a bit of a role model. Now he then covers some of these learning rate tuning techniques that we've talked about, and this is the SGD our schedule. It looks a bit different to what you're used to seeing this is on a log curve. This is the way that they show it on the paper and for more information again links to two blog posts, one from vitaly about this topic and and again, ananza ha is blog post on this topic.
So it's great to see that some of the work from faster, our students is already getting noticed and picked up and shared, and this blog post went on to get on the front page of hacker news. So that's pretty cool and hopefully more and more of this work or be picked up on sisters.
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Released publicly so last week we were kind of doing a deep dive into collaborative filtering and, let's remind ourselves of kind of what our final model looked like. So, in the end, we kind of ended up rebuilding the model, that's actually in the first, a a library where we had an embedding. So we had this little get embedding function that grabbed an embedding and randomly initialize the weights for the users and for the items. That's the kind of generic term. In our case the items are movies and the bias for the users, the bias for the items, and we had n factors embedding size for each for each one. Of course, the biases just had a single one, and then we grabbed the users and item in weddings. Multiply them together. Summed it up each row and add it on the bias terms pop that through a sigmoid, to put it into the range that we wanted. So that was our model and one of you asked if we can kind of interpret this information in some way, and I promised this week we would see how to do that. So, let's take a look, so we're going to start with the model we built here, where we just used that fastai library, collaborative data set from CSP and then that get learner, and then we fitted it in three epochs 19 seconds. We've got a pretty good result, so what we can now do is to analyze that model.
So you may remember right back when we started, we read in the movies CSV file, but that's just a mapping from the ID of the movie to the name of the movie and so we're just going to use that for display purposes. So we can see what we're doing, because not all of us have watched every movie, I'm just going to limit this to the top 500, most populous or 3,000 most popular movies. So we might have more chance of recognizing the movies we're looking at and then I'll go ahead and change it from the movie IDs from movie lens to those unique IDs that we're using the contiguous IDs, because that's what a model has alright so inside the learn. Object that we create inside alona, we can always grab the pytorch model itself, just by saying, learn: model okay and like I'm, going to kind of show you more and more of the code at the moment. So, let's take a look at the definition of model, and so a model is a property. So if you haven't seen a property before a property is just something in Python which looks like a method when you define it that you can call it without parentheses, as we do here, alright, and so it kind of looks when you call it like it's a Regular attribute, but it looks like when you define it like it's a method, so every time you call it, it actually runs this code, okay, and so in this case it's just a shortcut to grab something called dot models model.
So you may be interested to know what that looks like learn about models, and so this is there's a fastai model type is a very thin wrapper for pite watch models. So we could take a look at this code filter model and see what that is. It's only one line of code: okay and yeah - we'll talk more about these in part two right, but basically that there's this very thin wrapper and the main thing one of the main things that fast i out does is we have this concept of layer groups where Basically, when you say here, though, different learning rates and they're going to apply two different sets of layers and that's something that's not in paid watch. So when you say I want to use this pytorch model, all this with one thing we have to do, which is to say like okay, one hour later, groups yeah, so the details aren't terribly important, but in general, if you want to create a little wrapper For some other pipe watch model, you could just write something like this, so to get to get inside that to grab the actual pytorch model itself, its models, dot model, that's the pytorch model and then the learn object has a shortcut to that. Okay. So we're going to set m to be the pytorch model, and so, when you print out a pipe watch model, it prints it out. Basically, by listing out all of the layers that you created in the constructor, it's quite it's quite nifty. Actually, when you kind of think about the way, this works thanks to kind of some very handy stuff in Python, we're actually able to use standard, oh wow, to kind of define these modules in these layers and they basically automatically kind of register themselves with pipe which So back in our embedding bias, we just had a bunch of things where we said: okay, each of these things are equal to these things, and then it automatically knows how to represent that.
So you can see. There's the name. Is you, and so the name is just literally whatever we called it, yeah you and then the definition is it's this kind of layer? Okay, so that's our height watch model, so we can look inside that basically use that. So if we say m dot I be then that's referring to the embedding layer for an item which is the bias layer. So an item bias in this case is the movie bias, so each move either a 9000 of them has a single bias element. Okay, now the really nice thing about high torch layers and models is that they all look the same. They basically got to use them, you call them as if they were action, so we can go m.i.b, parenthesis right and that basically says I want you to return. The value of that layer and that layer could be a full-on model right so to actually get a prediction from a play: torch model you just, I would go m and pass in my variable, okay, and so in this case my B and pass in my top Movie indexes now models remember layers. They require variables, not tensors, because it needs to keep track of the derivatives. Okay, and so we use this capital V to turn the tensor into a variable and was just announced this week that pytorch 0.4, which is the version after the one. That's just about to be released is going to get rid of variables and will actually be able to use tensors directly to keep track of derivatives.
So, if you're watching this on the MOOC and you're looking at point four, then you'll probably notice that the code doesn't have this V unit anymore, and so that would be pretty exciting when that happens. But for now we have to remember if we're going to pass something into a model to turn it into a variable. First and remember: a variable has a strict superset of the API of a tensor, so anything you can do to a tensor. You can do to a variable and it up will take its log or whatever okay, so that's going to return a variable which consists of going through each of these movie IDs, putting it through this embedding layer to get its bias. Okay and that's going to return a variable. Let's take a look so before I press shift down to here. You can have a think about what I'm going to have. I've got a list of 3,000 movies going in turning into variable, putting it through this embedding layer. So just have a think about what we expect to come out: okay and we have a variable of size 3,000 by one. Hopefully that doesn't surprise you. We had 3000 movies that we are looking up each one hadn't had a one, long, embedding, okay, so there's our three thousand one you'll notice, it's a variable, just not surprising, because we fed it a variable, so we've got a variable back and it's a variable.
That's on the GPU right doc, CUDA, okay, so we have a little shortcut in fastai, because we we very often when I take variables, turn them into tensors and move them back to the CPU. So we can play with them more easily. So two NP is is two numpy okay, and that does all of those things and it works regardless of whether it's a tensor or a variable. It works, regardless of whether it's on the CPU or GPU it'll end up giving you a a numpy array from that. Okay, so if we do that, that gives us exactly the same thing as we just looked at, but now in numpy form. Okay, so that's a super handy thing to use when you're playing around with pytorch. My approach to things is: I try to use numpy for everything, except when I explicit and
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
You need something to run on the GPU or I need its derivatives right, in which case I use pytorch because, like none part like I kind of find none PI's, often easier to work with it's been around many years longer than pytorch. So you know and lots of things like the Python, imaging library, OpenCV and lots and lots of stuff like pandas. It works with numpy. So my approach is kind of like do as much as I can in num pile and finally, when I'm ready to do something on the GPU or take its derivative to pytorch and then as soon as I can. I put it back in vampire and you'll see that the fastai library really works. This way, like all the transformations and stuff happen in lamb pie, which is different to most high torch computer vision, libraries, which tend to do it all as much as possible in pytorch. I try to do as much as possible in non pipe. So let's say we wanted to transfer build a model in the GPU with the GPU and train it. Then we want to bring this to production. So would we call to numpy on the model itself or would we have to iterate through all the different layers and then call to NP yeah good question, so it's very likely that you want to do inference on a cpu rather than a GPU? It's it's more scalable, you don't have to worry about putting things in batches, you know and so forth. So you can move a model onto the cpu just by typing m dot, CPU, and that model is now on the cpu.
And so, therefore, you can also then put your variable on the CPU by doing exactly the same thing, so you can say like so now. Having said that, if you're, if you'll serve, it doesn't have a GPU or CUDA GPU, you don't have to do this because it won't put it on the GPU at all. So if for inferencing on the server, if you're running it on, you know some t2 instance or something it'll, work fine and will run on the on the cpu, automatically quick follow-up. And if we train the model on the GPU, and then we save those embeddings and the weights, would we have to do anything special to load? You know you won't. We have something. Well, it kind of depends how much of faster I you're using so I'll. Show you how you can do that in case. You have to do it manually, one of the students figure this out, which is really handy when we there's a load model function and you'll see what it does, but it does torch dot load. Is it basically? This is like some magic incantation that, like normally, it has to load it onto the same GPU or saved on, but this will like load it into what it was, what it is available. So there's a Andy discovery thanks for the great questions and to put that back on the GPU I'll need to say doc, CUDA and now there we go. I can run it again, okay, so it's really important to know about the zip function in Python, which iterates through a number of lists at the same time. So in this case, I want to grab each movie along with its bias term, so that I can just pop it into our list of tuples.
So if I just go zip like that, that's going to iterate through each movie ID and each bias term, and so then I can use that in a list comprehension to grab the name of each movie along with its place.
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Okay, so having done that, I can then sort - and so here are - I told you that John John Travolta Scientology movie at the most negative of the quiet by a lot. If this was a cable competition, Battlefield Earth would have like won by miles or this seven seven, seven, ninety six, so here's the worst movie of all time according to IMDB and like it's interesting when you think about what this means right, because this is like a Much more authentic way to find out how bad this movie is because, like some people are just more negative about movies right and like it more of them watch your movie, like you, know, highly critical audience. They're gon na read it badly. So if you take an average, it's not quite fair right, and so what this is? You know what this is doing is saying once we, you know, remove the fact that different people have different overall, positive or negative experiences and different people watch different kinds of movies, and we correct for all that this is the worst movie of all time. So that's a good thing to know, so this is how we can yeah look inside our our model and and interpret the bias vectors you'll see here, I've sorted by the zeroth element of each tuple by using a lambda. Originally, I used this special item ghetto. This is part of pythons operator library, and this creates a function that returns the zeroth element of something in order to save time, and then I actually realize that the lambda is only one more character to write.
Then the item get us, so maybe we don't need to know this after all, so yeah really useful to make sure you know how to write lambdas in Python. So this is this is a function, okay and so sort. The sort is going to call this function. Every time it decides like is this thing higher or lower than that other thing, and this fact this is going to return the zeroth element. Okay, so here's the same thing and item get a format, and here is the reverse and Shawshank Redemption right at the top. I'll definitely agree with that: Godfather usual suspects yeah. These are
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
All pretty great movies, twelve Angry Men. Absolutely so there you go, there's how we can look at the base. So then the second piece to look at would be the the embeddings. How can we look at the embeddings, so we can do the same thing so remember. I was the item embeddings rather than IV. With the item bias, we can pass in our list of movies as a variable turn it into numpy and here's our movie embedding so for each of the 3,000. Most popular movies here are its 50 embeddings. So it's very hard unless you're Geoffrey Hinton, to visualize a 50 dimensional space. So what we'll do is we'll turn it into a three dimensional space. So we can compress high dimensional spaces down into lower dimensional spaces, using lots of different techniques. Perhaps one of the most common and popular is called PCA. Pca stands for principle, components, analysis, it's a linear technique, but when your techniques generally work fine for this kind of embedding I'm not going to teach you about PCA now, but I will say in Rachel's, computation or linear algebra class, which you can get to you from. First, at AI, we cover PCA in detail and it's a really important technique. It. Actually, it turns out to be almost identical to something called singular value, decomposition which is a type of matrix decomposition which actually does turn up in deep learning a little bit from time to time, it's kind of somewhat worth knowing if you were going to dig more Into linear algebra, you know SPD and PCA, along with eigenvalues and eigenvectors, which are all slightly different versions.
Is this kind of the same thing or all worth knowing, but for now just know that you can grab PCA from SK, learn to calm position, say how much you want to reduce the dimensionality too. So I want to find three components, and what this is going to do is it's going to find three linear combinations of the 50 dimensions which capture as much as the variation as possible? Badar is different to each other as possible. Okay, so we would call this a lower rank approximation of our matrix all right. So then we can grab the components, so that's going to be their three dimensions, and so once we've done that, we've now got three by three thousand, and so we can now take a look at the first of them and we'll do the same thing of using Zip to look at each one along with its movie, and so here's the thing right. We we don't know ahead of time. What this PCA thing is.  You know it's. It's kind of the the main axis in this space of latent factors, and so what we can do is we can look at it and see if we can figure out what it's about right. So, given that police academy for is high up here, along with water world, where else Fargo Pulp Fiction and God further a high up here - I'm gon na guess that a high value is not going to represent like critically acclaimed movies or serious watching. So I kind of like all this yeah okay. I call this easy.
What she is serious all right, but like this is kind of how you have to interpret your embeddings is like take a look at what they seem to be showing and decide what you think it means. So this is the kind of the the principal axis in this set of embedding, so we can look at the next one. So do the same thing and look at the the first index one embedding this one's a little bit harder to kind of figure out. What's going on, but with things like Mulholland Drive and Purple Rose of Cairo, these look more kind of dialog II, kind of ones, or else things like Lord of the Rings in the Latin and Star Wars. These book more like kind of modern, CGI II kind of ones, so you could kind of imagine that on that pair of dimensions, it probably represents a lot of you know differences between how people read movies. You know some people, like you, know: purple rise of Cairo type movies. You know Woody Allen, kind of classic and some people like these. You know big Hollywood spectacles, some people, presumably like police academy, for more than they like Fargo, so yeah. So I'm like you, can kind of get the idea of what's happened. It's it's done a you know through a model which was you know, for a model which was literally multiply, two things together and Adam hop. It's learnt quite a lot. You know which is kind of cool, so that's what we can do with with that and then we could. We could plot them if we wanted to.
I just grabbed a small subset to plot on those first, two asses all right. So that's that so I wanted to next kind of dig in a layer deeper into what actually happens when we say fit alright. So when we said
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Learn fit: what's it doing, for something like the store model? Is it a way to interpret the embeddings for something like this? The rustman one? Yes, yeah, we'll see that in a moment. Well, let's jump straight there, what the hell, okay, so so for the rustman. How much are we going to sell at each store on each date model we? This is from the paper gore and burke on it. So it's a great paper by the way well worth you know like pretty accessible. I think any of you would, at this point, be able to at least get the gist of it. If you know, and much of the detail as well, particularly as you've also done the machine learning course, and they actually make this point in the paper. This is in the paper that the equivalent of what they call entity embedding layers, so an embedding of a categorical variable is identical to a one hot encoding, followed by a matrix multiply. That's why they're basically saying if you've got three embeddings, that's the same as doing three one hot encodings putting each through one through a matrix, multiply and then put that through a a dense layer. Well, what pytorch would call a linear, oh yeah, right one of the nice things here is because this is kind of like well. They thought it was. The first paper is actually the second, I think paper to show the idea of using categorical embeddings for this kind of data set. They really go to clean too. Quite a lot of detail to you know right back to the the detailed stuff that we learnt about.
So it's kind of a second, you know a second cat of thinking about what embeddings are doing so one of the interesting things that they did was they said: okay, after we've trained a neural net with these embeddings. What else could we do with it? So they got a winning result with a neural network where the entity meetings, but then they said hey, you know what we could take those empty embeddings and replace each categorical variable with the learnt entity embeddings and then feed that into a GBM right. So, in other words like, rather than passing into the GBM, a one modern coded version or an ordinal version, let's actually replace the categorical variable with its embedding for the appropriate level for that row right. So it's actually a way of create. You know feature engineering and so the main average percent error, without that for gbms, I'm using just 100 codings, was 0.15, but with that it was 0.11 that random forests, without that was point one six with that 0.108, nearly as good as the neural net right. So this is kind of an interesting technique, because what it means is in your organization, you can train a neural net that has an embedding of stores and an embedding of product types and an embedding of I don't know whatever kind of high cardinality or even medium Cardinality categorical variables you have and then everybody else in the organization can now like chuck those into their.
You know JVM or random forest or whatever, and I'm use them, and what this is saying is they won't get. In fact, you can even use K nearest neighbors with this technique and get nearly as good a result right. So this is a good way of kind of giving the power of neural nets to everybody in your organization without having them do the faster idea of learning course. First, you know they can just use whatever SK, learn or R or whatever that they're used to and like those those embeddings could literally be in a database table. Because if you think about an embedding is just an index lookup right, which is the same as an inner join in SQL right. So if you've got a table on each product along with its embedding vector, then you can literally do in a joint. And now you have every row in your table along with its product, embedding vector. So that's a really. This is. This is a really useful idea and gbm's and random forests learn a lot quicker than neural nets. Do all right! So that's like, even if you do know how to train your on its. This is still potentially quite handy. So here's what happened when they took the various different states of Germany and plotted the first two principal components of their embedding vectors, and they basically here is where they were in that 2d space and wacken lee enough.
I've circled in red three cities and i've circled here: the three cities in Germany and here I've circled in purple, so blue here at the blue: here's the green here's, the green. So it's actually drawn a map of Germany, even though it never was told anything about how far these states are away from each other or the very concept of geography didn't exist, so that's pretty crazy, so that was from there paper. So I went ahead and looked well. Here's another thing. I think this is also from their paper. They took every pair of places and they looked at how far away they are on a map versus how far away are they in embedding space and they've got this beautiful correlation. Alright, so again it kind of. Apparently, you know it's doors that are near by each other, physically, have similar characteristics in terms of when people buy more or less stuff from them. So I looked at the same thing four days of the week right. So here's an embedding of the days of the week from our model and I just kind of joined up Monday, Tuesday, Wednesday, Tuesday, Thursday, Friday Saturday Sunday. I did the same thing for the months of the year. All right again, you can see you know: here's! Here's winter here's summer so yeah, I think, like visualize embeddings, can be interesting like it's good to like. First of all check you can see things you would expect to see you know and then you could like try and see.
Like maybe things you didn't expect to see, so you could try all kinds of clusterings or or whatever - and this is not something which has been widely studied at all right. So I'm not going to tell you what the limitations are of this technique or whatever. Oh yes, so I've heard of other ways to generate embeddings like skip grams uh-huh wondering if you could say is there one better than the other using your own Network, sir skip grams, so screwed grams is quite specific to NLP right so, like I'm, not sure. If we'll cover it in this course, but basically the the approach to original kind of word to vac approach to generating embeddings was to say you know what we actually don't have. We don't actually have our labelled data set. You know they said all we have is like google books, and so they have an unsupervised, learning problem, unlabeled problem, and so the best way, in my opinion, to turn an unlabeled problem into a labelled problem is to kind of invent some labels and so what they Did in the word to vet case was they said: okay, here's a sentence with 11 words in it right and then they said: okay, let's delete the middle word and replace it for the random word, and so you know originally it said cat and they say no. Let's replace that with justice all right so before it said the cute little cat sat on the fuzzy mat and now it says the cute little justice sat on the fuzzy man right and what they do.
Is they do that, so they have one sentence where they keep exactly as is, and then they make a copy of it and they do the replacement. And so then they have a label where they say it's a one. If it was unchanged, it was the original and zero, otherwise, okay, and so basically, then you now have something you can build a machine learning model on, and so they went and build a machine learning model on this. So the model was like try and find the effect sentences, not because they were interested in a fake sentence binder, but because, as a result, they now have embeddings that, just like we discussed you can now use for other purposes, and that became word to vet. Now it turns out that if you do this as just a kind of a effectively like a single matrix multiply, rather than make it a deep neural net, you can train this super quickly, and so that's basically what they did with they'd met there, though they kind Of decided we're going to make a pretty crappy model like a shallow learning model rather than a deep model. You know with the downside, it's a less powerful model, but a number of upsides. The first thing we can train it on a really large data set and then also really importantly, we're going to end up with embeddings, which have really very linear characteristics. So we can like add them together and subtract them and stuff like that. Okay, so that so there's a lot of stuff, we can learn about there from like for other types of embedding like categorical embeddings and specifically, if we want categorical embeddings, which we can kind of draw nicely and expect them to us to be able to add and Subtract them and behave linearly. You know, probably if we want to use them in k-nearest, neighbors and stuff, we should probably use shallow learning.
If we want something, that's going to be more predictive, we probably want to use a neural net and so actually an NLP. I'm really pushing the idea that we need to move past word to backhand glove, these linear based methods, because it turns out that those embeddings are way less predictive than embeddings learnt from models, and so the language model that we learned about, which ended up getting a State-Of-The-Art on sentiment, analysis didn't used a lot more work to vet that instead we pre trained a deep, recurrent neural network and we ended up with not just a pre trained word vectors, but a for pre-trained model. So it looks like Duke, creates embeddings for entities. We need like a dummy task, not necessarily a dummy task like in this case. We had a real task right, so we created the embeddings for Rossmann by trying to predict store sales. You only need this isn't just in this, isn't just for learning embeddings for learning. Any kind of feature space you either need label data or you need to invent some kind of fake task. So does that task matter like if I choose a task and train and lettings? If I choose another task and train and lettings like which one is it's a great question, and it's not something - that's been studied nearly enough right, I'm not sure that many people even quite understand that when they say unsupervised learning now about nowadays, they almost nearly always Mean fake tasks, labeled learning and so the idea of like what makes a good fake task. I don't know that I've seen a paper on that right that intuitively you know, we need something where the kinds of relationships it's going to learn likely to be the kinds of relationships that you probably care about right. So, for example, in in computer vision, one kind of fake task people use is to say, like let's take some images and use some kind of like unreal and unreasonable data.
Augmentation like like recolor them too much or whatever, and then we'll ask the neural net to like predict, which one was the Augmented, which one was not. You admitted yeah. So it's I think, it's a fascinating area, one which you know would be really interesting for people to you know maybe some of the students here they're looking to further. It's like take some interesting semi-supervised tour, unsupervised, datasets and try and come up with some like more clever fake tasks and see like does it matter. You know how much does it matter in general like if you can't come up with a fake task that you think seems great, I would say, use it use the best. You can it's an often surprising how how little you need like the ultimately crappy fake task. Is called the auto encoder and the auto encoder is the thing which which one the claims prediction - competition that just finished on cattle. They had lots of examples of insurance policies where we knew this was how much was claimed and then lots of examples of insurance policies where I guess there must have been still still open. We didn't yet know how much they claimed right and so what they did was they said, okay, so for all of the ones. So, let's basically start off by grabbing every policy right and we'll take a single policy and we'll put it through a neural net right and we'll try and have it reconstruct itself.
But in these intermediate layers, and at least one of those intermediate layers will make sure there's less activations and there were inputs. So let's say if there was a hundred variables on the insurance policy. You know we'll have something in the middle that only has like twenty activations all right, and so, when you basically are saying hey reconstruct your own input like it's, not a different kind of model doesn't require any special code. It's literally just passing. You can use any standard, pipe torch or fastai learner. You just say my output equals my input right and that's that's like the the most uncreated. You know invented task you can create and that's called an autoencoder and it works surprisingly. Well, in fact, to the point that it literally just won a cackle competition, they took the features that it learnt and chucked it into another neural net and yeah and one you know. Maybe if we have enough students taking an interest in this, then you know we'll be able to cover covered unsupervised learning in more detail in in part two specially, given this cattle have a win. I think this may be related to the previous question when training language models is the language model example trained on the archive data. Is that useful at all in the movie great question you know I was just talking to Sebastian about this question read about this this week and we thought would try and do some research on this in January. It's it's again. It's not well done.
We know that in computer vision, it's shockingly effective to train on cats and dogs and use that fruit train network to do lung cancer diagnosis and CT scans in the NLP world. Nobody much seems to have tried this, the NLP research as I've spoken to other than Sebastian about this assume that it wouldn't work and they generally haven't bother trying. I think it would work great so so, since we're talking about
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Ruspin I just mentioned during the week. I was interested to see like how good this solution actually actually was, because I noticed that on the public leader board, it didn't look like it was going to be that great, and I also thought it'd be good to see. Like what does it actually take to use a test set properly with this kind of structured data? So if you have a look at ruspin now, I've pushed some changes that actually run the test set through as well, and so you can get a sense of how to do this so you'll see. Basically, every line appears twice one for tests and one-foot one for train when we get there yeah test train test trains, history. Obviously, you could do this on a lot fewer lines of code by putting all of the steps into a method and then pass either the train data set. Well, the test data set up dataframe to it. In this case, i wanted to kind of put for teaching purposes, you'd, be able to see step and to experiment to see what each step looks like, but you could certainly simplify this code so yeah. So we do this for every data frame and then some of these you can see I kind of lived through the data frame in joined and the joint test right training. Just this whole thing about the durations. I basically put two lines here: one that said data frame equals train columns, one that says data frame equals test columns, and so my you know basically ideas you'd run this line first and then you would skip the next one and you'd run everything beneath it and Then you'd go back and run this line and then run everything believe it. So some people on the forum were asking how come this code wasn't working this week, which is a good reminder that the code is not designed to be code that you always run top to bottom without thinking right, you're meant to like think like what is this Code here should I be running it right now: okay, and so like the early lessons, I tried to make it so you can run it top to bottom, but increasingly as we go along, I kind of make it more and more that, like you, actually have to Think about what's going on so Jimmy you're, talking about shadow learning and deep learning, could you define that a bit better by sure? I'm learning, I think I just mean anything that doesn't have a hidden layer, so something that's like a dot product matrix multiplier.
Basically, okay, so so we end up with a training and a test version, and then everything else is basically the same. One thing to note on a lot of these details of this: we cover in the machine learning course by the way, because it's not really deep, learning specific so check that out. If you're, just in the details, I should mention you know, we use apply cats rather than train cats to make sure the test set and the training set have the same categorical codes and that they join too. We also need to make sure that we keep track of the mapper. This is the thing which basically says: what's the mean and standard deviation of each continuous column and then apply that same method test set, and so when we do all that, that's basically it then the rest is easy. We just have to pass you in the test. Data frame in the usual way when we create our model data object and there's no changes through all here. We trained it in the same way and then once we finish training it, we can then call predict, as per usual, passing in true to say this is the test set rather than the validation set and pass that off to cattle, and so it was really interesting Because this was my submission, it got a public score of 103, which would put us in about 300, and some things place which looks awful right and our private score of 107 need a board private.
Here's about fifth right so like if you're competing in a cable competition - and you don't haven't thoughtfully, created a validation set of your own and you're, relying on publicly the board feedback. This could totally happen to you, but the other way around you'll be like. Oh I'm. In the top ten I'm doing great and then oh, for example, at the moment, the ice Berg's competition recognizing icebergs, a very large percentage of the public leaderboard set, is synthetically generated data augmentation data like totally meaningless, and so your validation set is going to be much More helpful and the public leaderboard feedback right so yeah be very careful. So our final score here is kind of within statistical noise of the actual third-place getters. So I'm pretty confident that we've we've captured their approach, and so that's that's pretty interesting. Something to mention. There's a nice kernel about the rustman, I quite a few nice kernels actually, but you can go back and see like, particularly if you're doing the groceries competition go and have a look at the Rossmann kernels, because actually quite a few of them, a higher quality than The ones for the Ecuadorian groceries competition, one of them, for example, showed how, on four particular stores like straw, eighty five, the sales for non Sundays and the sale for Sunday's looked very different.
Where else there are some other stores where the sales on Sunday don't look any
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Different and it can kind of like get a sense of why you need these kind of interactions. The one I particularly wanted to point out is the one. I think I briefly mentioned that the third-place winners, whose approach we used, they didn't notice - is this one?  Alright and here again closed bang right, so this third-place winner actually deleted all of the closed store rows before they started doing any analysis. Right so remember how we talked about like don't touch your data unless you, first of all analyze, to see whether that thing you're doing is actually okay, no assumptions right. So, in this case I am sure, like I haven't, tried it, but I'm sure they would have one otherwise right because like well, though there weren't actually any store closures to my knowledge in the test set period. The problem is that their model was trying to fit to these like really extreme things and so and because it wasn't able to do it very well, it was gon na end up getting a little bit confused. It's not gon na break the model, but it's definitely gon na harm it, because it's kind of trying to do computations to fit something which it literally doesn't have the data for your neck.
Can you pass that back there all right, so that Russman model again like it's nice to kind of look inside to see what's actually going on right and so that Russman model? I want to make sure you kind of know how to find your way around the code, so you can answer these questions for yourself, so it's inside columnar model data. Now um. We started out by kind of saying: hey if you want to look at the code for something you couldn't like a question mark question mark like this and oh okay, I need to I haven't, got this reading, but you can use question mark question mark to get The source code for something right, but obviously like that's, not really a great way, because often you look at that source code and it turns out. You need to look at something else right and so for those of you that haven't done much coding, you might not be aware that almost certainly the editor you're using probably has the ability to both open up stuff directly off SSH and to navigate through it. So you can jump straight from place to place right so want to show you what I mean. So if I were to find columnar model data - and I have to be using vim here - I can basically say tag columnar model data and it will jump straight to the definition of that plus right. And so then, I notice here that, like oh, it's actually building up a data loader, that's interesting.
If I get control right square bracket, it'll jump to the definition of the thing that was under my cursor and after I finished reading it for a while. I can hit ctrl T to jump back up to where I came from right and you kind of get the idea right or if I want to find it for usage of this in this file of columnar model data, I can hit star to jump to the Next place, it's new used, you know and so forth. Alright, so in this case, get learner was the thing which actually got the model and we want to find out what kind of model it is, and apparently it uses a I'm not using collaborative filtering are. We were using columnar model data, sorry columnar model data, okay learner, which users - and so here you can see mixed input model is the pytorch model, and then it wraps it in the structured learner, which is the the first day. I learn a type which wraps the data and the model together. So if we want to see the definition of this actual pytorch model, I can go to control right square bracket to see it right, and so here is the model right and nearly all of this we can now understand right. So we got past, we got past a list of embedding sizes in the mixed model that we saw. Does it always expect categorical and continuous together? Yes, it does and the the model data behind the scenes if there are no none of the other type, it creates a column of ones or zeros or something okay. So if it is null, it can still work.
Yeah yeah yeah it's kind of ugly and hacky and will you know hopefully improve it, but yeah? You can pass in an empty list of categorical or continuous variables to the model data and it will basically yeah it'll, basically pass an unused column of zeros. To avoid things breaking and I'm I'm leaving fixing some of these slightly hacky edge cases, because height or 0.4, as well as you're, getting rid of variables. They're going to also add rank 0 tensors, which is to say, if you grab a single thing out of. Like a rent, 110, sir, rather than getting back at a number which is like qualitatively different you're, actually going to get back like a tensor that just happens to have no rank now it
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Turns out that a lot of this kind of codes gon na be actually easier to write, then so, and for now it's it's a little bit more happier than it needs to be Jeremy. You talk about this a little bit before, where maybe it's a good time at some points, talk about how can we write something that is slightly different for worries in the library yeah? I think we'll cover that a little bit next week that I'm mainly going to do that in part to like Pat who's going to cover quite a lot of stuff. One of the main things were cover in part. Two is what it called generative models, so things where the output is a whole sentence or a whole image. But you know I also dig into like powder really either customize the first day I library or use it on more custom models. But if we have time we'll touch on it a little bit next week, okay, so the the learner we were passing in a list of embedding sizes and, as you can see, that embedding sizes list was literally just the number of rows and the number of columns In each embedding right and the number of code rose was just coming from literally how many stores are there in the store category, for example, and the number of columns was just a quarter that divided by two and a maximum of 50. So that thing that list of tuples was coming in, and so you can see here how we use it right, we go through each of those tuples grab the number of categories and the size of the embedding and construct an embedding all right, and so that's a That's a list right, one minor thing, height or specific thing we haven't talked about before is for it to be able to like register.
Remember how we kind of said, like it registers your parameters. It registers your your layers like someone we like listed the model. It actually printed out the Novation varying an age bias. It can't do that if they're hidden inside a list right, they have to be like a there, have to be a an actual n n dot module subclass. So there's a special thing called an N n dot module list which takes a list and it basically says I want you to register. Everything in here has been part of this model. Okay, so it's just a minor tweak, so yeah. So our mixed input model has a list of embeddings and then I do the same thing for a list of linear layers right. So when I said here, 1000 comma 500, this was saying how many activations I wanted featured. My lineal is okay, and so here I just go through that list and create a linear layer that goes from this size to the next size. Okay, so you can see like how easy it is to kind of construct your own, not just your own model, but a kind of a model which you can pass parameters to have a constructed on the fly dynamically and that's normal talk about next week. This is initialization, we've mentioned climbing her initialization before, and we mentioned it last week and then drop out same thing right. We have here a list of how much drop out to apply to each layer right so again. Here it's just like go through each thing.
In that list and create a drop out layer for it, okay, so this constructor, we understand everything in it except for batch norm, which we don't have to worry about for now, so that's the constructor, and so then the forward also, you know all stuff we're aware Of go through each of those embedding layers that we just saw and remember, we've just treated like as a function, so call it with the ithe categorical variable and then concatenate them all together, put that through drop out and then go through each one of our linear Layers and call it apply relia to it, apply, dropout and then finally apply the final linear layer and the final linear layer has this as its size, which is here right size, one there's a single unit, sales, okay. So we're kind of getting to the point where oh and then, of course, at the end, if this I mentioned would come back to this, if you passed in a Y underscore range parameter, then we're going to do the thing we just learned about last week, which Is to use a sigmoid right, and this is a cool little trick to make you're not just to make your collaborative filtering better. But in this case my basic idea was you know: sales are going to be greater than zero and probably less than the largest sale. They've ever had so I just pass in that as Y range, and so we do a sigmoid and multiply with the sigmoid by the range that I passed it all right. And so hopefully we can find that here yeah here it is right.
So I actually said: hey, maybe the range is between zero and you know the highest x. One point two: you know cuz, maybe maybe the next two weeks we have one bigger, but this is kind of like again try to make it a little bit easier for it to give us the kind of results that it thinks is right. So, like increasingly, you know, I'd love your wall to kind of try to not treat these learners and models as black boxes, but to feel like you now have the information you need to look inside them and remember. You could then copy and paste this plus paste it into a cell in duple, notebook and start fiddling with it to create your own versions. Okay, I think what I might do is we might take a bit of a early break because we've got a lot to cover and I want to do it all in one big go. So, let's take a let's take a break until 7:45 and then we're going to come back and talk about recurrent neural networks, all right,
PAUSE
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
So we're going to talk about Aaron ends before we do. We've got to kind of dig a little bit deeper into SGD because I just want to make sure everybody's totally comfortable with with SGD, and so what we're going to look at is we're going to look at lesson: six SGD notebook and we're going to look at A really simple example of using SGD to learn y equals ax, plus B, and so what we're going to do here is we're going to create, like the simplest possible model y equals ax, plus, B, okay, and then we're going to generate some random data. That looks like so so: here's our X and here's our Y. We want to predict Y from X and we passed in 3 & amp 8 as our a and B, so we're going to kind of try and recover that right, and so the idea is that if we can solve something like this, which has two parameters, we Can use the same technique to solve? We can use the same technique to solve something with a hundred million parameters right without any changes at all. So, in order to find a and a B that fits this, we need a loss function and this is a regression problem because we have a continuous output so for continuous output regression, we tend to use, mean squared error, all right and obviously all of this stuff. There's there's implementations in non pious implementations in flight or we're just doing stuff by hand. So you can see all the steps right.
So there's MSE, okay y hat is we often call our predictions Y hat mitis y squared mean there's, I meant whatever okay. So, for example, if we had ten and five where a and B then there's our mean square R, squared error three point: two: five: okay, so if we've got an A and a B and we've got an x and a y, then our mean square error. Loss is just the mean squared error of our linear. That's our predictions and our way. Okay!  So that's a loss function right, and so when we talk about combining linear layers and loss functions and optionally nonlinear layers, this is all we're doing right is we're putting a function inside a function. Yeah, that's that's all like. I know people draw these clever, looking dots and lines all over the screen when they're saying this is what a neural network is, but it's just it's just a function of a function of a function. Okay, so here we've got a prediction: function: being a linear layer, followed by a loss, function being MSE, and now we can say like oh well, let's just define this as MSA Lost's and we'll use that in the future. Okay, so there's our loss function, which incorporates our prediction function: okay, so let's
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Generate 10,000 items or thick data and let's show them in two variables, so we can use them with pytorch, because Jeremy doesn't like taking derivatives so we're going to use pytorch for that and let's create random, wait for a and B. So a single random number and we want the gradients of these to be calculated as we start computing with them, because these are the actual things we need to update in our SGD okay, so here's our a and B 0.029 0.111 all right. So let's pick a learning rate, okay and let let's do 10,000 epochs of SGD. In fact, this isn't really SGD. It's not stochastic gradient. It said this is actually full gradient descent we're going to each each loop is going to look at all of the data. Okay, stochastic gradient descent would be looking at a subset each time so to do gradient descent. We basically calculate loss right so remember: we've started out with a random a and B okay, and so this is going to compute some amount of loss and then it's nice from time to time. So one way of saying from time to time is: if the epoch number mod, a thousand is zero right, so every thousand epochs just print out the loss. So you have it. Do it? Okay! So now that we've computed the loss, we can compute our gradients right, and so you just remember this thing here is both a number, a single number, that is our lost, something we can print, but it's also a variable because we passed variables into it and therefore It also has a method type backward, which means calculate the gradients of everything that we asked it to.
Everything where we said requires radical is true. Okay, so at this point we now have a dot grad property inside a and inside P, and here they are here - is that grant grad property? Okay. So now that we've calculated the gradients for a and B, we can update them by saying a is equal to whatever it used to be the learning rate times. The gradient, okay, dot data, because a is a variable and a variable contains a tensor and it's dot. Data property - and we again this is going to disappear in height, which point four, but for now it's actually the ten so that we need to update okay, so update the tensor inside here, with whatever it used to be the learning rate times. The gradient.  Descent is okay, so it's it's as simple, as we claimed there's one extra step in pytorch, which is that you might have like multiple different loss functions or like lots of lots of output layers, all contributing to the gradient, and you like to have to add Them all together, and so, if you've got multiple loss functions, you could be calling loss, stop backward on each of them and what it does is an ad sit to the gradients right, and so you have to tell it when to set the gradients back to zero. Okay, so that's where you just go: okay, set a to zero and gradients in set B gradients to zero, okay, and so this is wrapped up inside the you know: op TMS, JD class right so when we say up Tim dot, SGD - and we just say you Know dot step, it's just doing these for us.
So when we say dot, zero gradients is just doing this force and this underscore here every pretty much every function. That applies to a tensor in pytorch. If you stick an underscore on the end, it means do it in place.  So that's basically it we can look at the same thing without pytorch, which
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Means we actually do have to do some calculus. So if we generate some fake data again, we're just going to create 50 data points this time just to make this fast and easy to look at, and so let's create a function called update right, we're just going to use numpy no pytorch, okay. So our predictions is equal to again linear and in this case we actually gon na calculate the derivatives. So the derivative of the square of the loss is just two times and then the derivative is the vector. A is just that you can confirm that yourself. If you want to - and so here our we're going to update a minus equals learning rate times the derivative of loss with respect to a and for B, it's learning rate times derivative with respect to B, okay, and so what we can do, let's just run all This so just for fun, rather than looping through manually, we can use the map flop, matplotlib, func, animation, command, to run the animate function, a bunch of times, and the animate function is going to run 30 epochs and at the end of each epoch, it's going to Print out on the plot, where the line currently is - and that creates this at all movie okay, so you can actually see that the line moving at a place right. So if you want to play around with like understanding how high torque gradients actually work step-by-step, here's like the world's simplest at all example.
Okay - and you know it's kind of like it's kind of weird to say like that's: that's it like when you're optimizing a hundred million parameters in a neural net, it's doing the same thing, but it it actually is alright. You can actually look at the pytorch code and see it's this. Is it right, there's no trick? Well, we load a couple of minor tricks last time, which was like momentum and atom right that, if you could do it in Excel, you can do it invite them so. Okay,
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
So let's do talk about our lens, so we're now in less than six hour and in notebook and we're going to study Nietzsche, as you should so Nietzsche says. Supposing that truth is a woman? What, then I love this? Apparently all philosophers have failed to understand women. So apparently, at the point that Nietzsche was alive, there was no female philosophers, or at least those that were around didn't understand women either so anyway. So this is the philosopher. Apparently, we've chosen to study it. Leech is actually much less worse than people think he is, but it's a different era. I guess alright, so we're going to learn to write philosophy like Nietzsche and so we're going to do it one character at a time. So this is like the language model that we did in Lesson, four, where we did it a word at the time, but this time we're going to do a character at a time, and so the main thing I'm going to try and convince you is an RNN. Is no different to anything you've already learned, okay and so to show you that going to build it from plain pytorch layers, all of which are extremely familiar already: okay, and eventually we're going to use something really complex, which is a for loop. Okay. So that's when we're going to make a really sophisticated, so the basic idea of our n ends is that you want to keep track of the main thing.
Is you want to keep track of kind of state over long term dependencies? So, for example, if you're trying to model something like this kind of template language right then at the end of your percent comment: blue percent, you need a percent common end percent right, and so somehow your model needs to keep track of the fact that it's like Inside a comment over all of these different characters: right and so this is this idea of state. It's kind of memory right - and this is quite a difficult thing to do with, like just a calm, confident it turns out actually to be possible, but it's it's. You know a little bit tricky where elsewhere, as an iron in it turns out to be pretty straightforward all right. So these are the basic ideas. If you want the stateful representation, where you kind of keeping track of like where are we now have memory, have long term dependencies and potentially even have variable length sequences? These are all difficult things to do with confidence, they're very straightforward, with arid ends. So, for example, SwiftKey a year or so ago, did a blog post about how they had a new language model where they
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Basically, this is from their blog post. We basically said like, of course, this is what their neural net looks like. Somehow they always look like this on the internet. You know, you've got a bunch of words and it's basically going to take your particular words in their particular orders and try and figure out what the next words going to be, which is to say they built a language model. They actually have a pretty good language model if you've used SwiftKey, they seem to do better predictions than anybody else still. Another cool example was andre. Capaci a couple of years ago showed that he could use character. Level are a 10 to actually create an entire latex document, so he didn't actually tell it in any way what life looks like he just passed. The some may tech text, like this and said, generate more low text text and it literally started writing something which means about as much to me as most math papers. Do this, okay, so we're gon na start with something: that's not an RN and I'm going to introduce Jeremy's patented neural network, notation involving boxes circles and triangles. So let me explain: what's going on as a rectangle is an input, an arrow is a layer as a circle.  The rectangle is the input activations. The circle is a hidden activations and a triangle is an output. Activations and arrow is a layer, operation right or possibly more than one all right.
So here my rectangle is an input of number of rows. Equal a batch size and number of columns equal to the number of number of inputs number of variables, all right, and so my first arrow. My first operation is going to represent a matrix product, followed by our Lu and that's going to generate a set of activation. Remember activations, like an activation, is a number that an activation is a number, a number that's being calculated by a value or a matrix product or whatever it's a number right. So this circle here represents a matrix of activations. All of the numbers that come out when we take the inputs, we do a matrix product, followed by a value, so we started with batch size, byte number of inputs, and so after we do this matrix operation, we now have batch size by you know. Whatever the number of columns in our matrix product was by number of hidden units, okay, and so if we now take these activations, but it's the matrix and we put it through another operation in this case another matrix product and the softmax, we get a triangle. That's our output, activations, another matrix of activations and again number of roses. Batch size number of columns number is equal to the number of classes. Again, however, many columns our matrix in this matrix product head. So that's a that's a neuro net right.
That's our basic kind of one hidden layer, neural net and if you haven't written one of these from scratch, try it you know and in fact, in lessons nine ten and eleven of the machine learning course we do this right. We create one of these from scratch. So if you're not quite sure how to do it, you can check out the machine learning costs yeah in general. The machine learning cost is much more like building stuff up from the foundations. Where else this course is much more like best practices, kind of top-down all right, so if we were doing like a cognate with a single, dense, hidden layer, our input would be equal to actually number yeah. That's very implied, watch number of channels by height by width, right and notice that here batch size appeared every time. So I'm not gon na I'm not gon na write it anymore. Okay, so I've removed the batch size. Also the activation function. It's always basically value or something similar for all the hidden layers and softmax at the end for classification. So I'm not going to write that either okay, so I'm kind of edge picture, I'm going to simplify it a little bit alright. So I'm not gon na mention batch size. It's still there we're not going to mention real you or softmax, but it's still there. So here's our input - and so in this case rather than a matrix product, will do a convolution, let's drive to convolution, so we'll skip over every second one or could be a convolution, followed by a mac spool in either case.
We end up with something which is replaced number of channels with number of filters right, and we have now height, divided by two and width divided by 2. Okay, and then we can flatten that out. Somehow we'll talk next week about the main way. We do that nowadays, which is basically to do something called an adaptive max pooling where we basically get an average across the height and the width and turn that into a vector anyway. Somehow we flatten it out into a vector we can do a matrix product or a couple of matrix products. We actually tend to do in fastai, so that'll be our fully connected layer with some number of activations final matrix product give us some number of classes. Okay, so this is our basic component, remembering rectangles input circle is hidden. Triangle is output, all other shapes represent a tensor of activations. All of the arrows represent a operation or lay operation all right. So now that's going to jump to the one, the first one that we're going to actually try to try to create for NLP and we're going to basically do exactly the same thing as here right and we're going to try and predict the third character. In a three character: sequence, based on the previous two characters, so our input and again remember: we've removed the batch size dimension. We're not saying that we're still here, okay, and also here, I've removed the names of the layer operations entirely. Okay, just keeping simplifying things.
So, for example, our first import would be the first character of each string in our mini batch. Okay and assuming this is one hot encoded, then the width is just. However many items there are in the vocabulary: how many unique characters could we have? Okay, we probably won't really one hot encoder will feed it in as an integer and pretend it's one hot encoded by using an embedding layer which is mathematically, identical, okay, and then we that's going to give us some activations, which we can stick through a fully connected Layer - okay, so we we put that through. If we click through a fully connected layer to get some activations, we can then put that another fully connected layer and now we're going to bring in the input of character to alright. So the character to input will be exactly the same dimensionality as the character one input, and we now need to somehow combine these two arrows together. So we could just add them up, for instance, right because remember this arrow here represents a matrix product, so this matrix product is going to spit out the same dimensionality as this matrix product. So we could just add them up to create these activations, and so now we can put that through another matrix product and of course, remember all these metrics products have a RAL you as well, and this final one will have a softmax instead to create our predicted Set of characters right, so it's a standard, you know two hidden layer.
I guess it's actually three matrix products neural net. This first one is coming through an embedding layer. The only difference is that we're also got a second input coming in here that we're just adding in right, but it's kind of conceptually identical. So let's let's implement that for Nietzsche all right, so I'm not going to use torch text. I'm gon na try not to use almost any fastai, so we can see it all kind of again from raw right. So here's the first 400 characters of the collected works. Let's grab a set of all of the letters that we see there and sort them. Okay and so a set creates all the unique letters, so we've got 85 unique letters in our vocab. Let's pop up, it's nice to put an empty kind of a null or some some kind of padding character in there for padding. So we're gon na put a parenting character at the start. Right, and so here is what our vocab looks like. Okay, so so Kars is our bouquet. So, as per usual, we want some way to map every character, to a unique ID and every unique ID to a character. And so now we can just go through our collected works of niche and grab the index of each one of those characters. So now we've just turned it into this right. So rather than quote PR e, we now have 40 42, 29, okay. So so that's basically the first step and just to confirm. We can now take each of those indexes and turn them back into characters and join them together and yeah there.
It is okay, so from now on we're just going to work with this IDX list, the list of character members in the connected works of Nietzsche. Yes, so, Jeremy, why are we doing like a model of characters and not a model of words? I just thought it seemed simpler. You know with a vocab of 80-ish items, we can kind of see it better character, level models turn out to be potentially quite useful. In a number of situations, but we'll cover that in part two, the short answer is like you generally want to combine both the word level model and a connect character. Level model like if you're doing, say, translation, it's a great way to deal with unknown. Like unusual words, rather than treating it as unknown anytime, you see a word you haven't seen before. You could use a character level model for that and there's actually something in between the two quarter: byte pair and coding, vpe, which basically looks at at all. Engrams of characters, but we'll cover all that in part two. If you want to look at it right now, then part two of the existing course
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Already has this stuff taught and part two of the version 1 of this course, although the NLP stuff is in flight which, by the way so you'll understand it straight away, it was actually the thing that inspired us to move to piped watch because trying to do It in chaos turned out to be a nightmare all right, so let's create the inputs to this. We're actually going to do something slightly different. What I said we're actually going to. I predict the fourth character that actually this the fifth character using the first four. So the index four character using the index zero one two and three okay, so it was exactly the same thing, but with just a couple more layers. So that means that we need a list of the zeroth first, second and third characters. That's why I'm just cutting every character from the start from the one from two from three skipping over three at a time? Okay, so hmm! This is I, I said this wrong, so we're going to predict the third character, the fourth character from the third for the first story. Okay, the fourth character is history, all right, so our inputs will be these three lists right, so we can just use n P dot stack to pop them together, all right, so here's the zero one and two characters that are going to feed into a model and Then here is the next character in the list, so, for example, X, 1, X, 2, X, 3 and Y all right, so you can see. For example, we start off the first.
The very first item would be 40, 42 and 29 right, so that's characters naught 1 and 2 and then we'd be predicting 30. That's the fourth character, which is the start of the next row, all right. So then, 30. 25. 27. We need to predict 29, which is the start of next row and so forth, so we're always using three characters to predict the fourth. So there are 200,000 of these that we're going to try and model right. So we're going to build this, which means we need to decide how many activations, so I'm going to use 256 okay and we need to decide how big our embeddings are going to be, and so I decided to use 42 so about half the number of characters. I have - and you can play around these, so you can come up with better numbers. It's just a kind of experimental and now we're going to build our model. Now I'm gon na change my model slightly, and so here is the the full version so predicting character for using characters. 1 2 & amp 3. As you can see, it's the same picture as a previous page, but I put some very important coloured arrows here. All the arrows of the same color are going to use the same matrix, the same weight matrix right, so all of our input embeddings are going to use the same matrix all of our layers that go from one layer to the next they're going to use the Same orange arrow weight matrix and then our output will have its own matrix. So we're going to have one two three weight, matrices right and the idea here is the reason.
I'm not gon na have a separate one, but every everything here is that, like. Why would kind of semantically a carrot to have a different meaning, depending if it's the first or the second or the third item, in a sequence like it's not like we're, even starting every sequence, at the start of a sentence we're just arbitrarily chopped it into groups Of three right, so you would expect these to all have the same kind of conceptual mapping and ditto like when we're moving from claritin or character, one you know to kind of say, build up some state here. Why would that be any different kind of operation to moving from character, wonder character to so that's the basic idea. So let's create a three character model and so we're going to create one linear layer for our Green Arrow, one linear layer, fat, orange arrow and one linear layer for our blue arrow and then also one embedding okay. So the embedding is going to bring in something with size, whatever it was 84, I think vocab size and spit out something with an factors in the embedding. Well then put that through a linear layer, and then we've got our hidden layers before the output layer. So when we call forward they're going to be passing in one two, three characters, so if each one will stick it through an embedding we'll, stick it through a linear layer and we'll stick it through a value just to do it the character, one character and character.
Three: okay, then I'm going to create this circle of activations here: okay and that matrix I'm going to call H right, and so it's going to be equal to my input, activations. Okay, after going through the value and the linear layer and the embedding right and then I'm going to apply this l hidden so the orange arrow and that's going to get me to here. Okay! So that's what this layer here does and then to get to the next one. I need to reply the same thing and it apply the orange arrow to that okay, but I also have to add in this second input right so take my second input and add in okay, my previous layer, your neck. Could you pass it back three rows? I don't really see how these dimensions are the same from eight and in2 from literature which, from yeah okay, let's go through. So, let's figure out the dimensions together, so self dot E is gon na be of length, 42, okay and then it's gon na go through L in I'm just gon na make it of size, n, hidden, okay, and so then we're going to pass that which is Now size n hidden through this, which is also going to return something of size, n, hidden, okay. So it's a really important to notice that this is square. This is a square weight matrix okay, so we now know that this is of size n hidden into it's. Going to be exactly the same size as in one was, which is n hidden, so we can now sum together. Two sets of activations, both the size n, hidden passing it into here and again it returns something of size, n hidden.
So basically, the trick was to make this a square matrix and to make sure that it's square matrix was the same size as the output of this hidden. Well, thanks for the great question, can you pass that out to you now? Jeremy is summing. The only thing people can do in these cases I'll come back to that in a moment. That's great point: okay, um! I don't like it when I have like three bits of code that look identical and then three bits of codes that look nearly identical but aren't quiet because it's harder to refactor. So I'm going to put a make H into a bunch of zeros so that I can then put H here, and these are now identical. Okay, so that the hugely complex trick that we're going to do very shortly is to replace these three things. With a for loop, okay and it's going to loop through one two and three: that's that's going to be the for loop or actually zero, one, two! Okay! At that point, we'll be able to call it a recurrent neural network, okay, so just to skip ahead a little bit alright, so we create that that model make sure I've run all these, so we can actually run this thing. Okay, so we can now just use the same columnar model data class that we've used before and if we use from arrays, then it's basically it's going to spit back the exact arrays. We gave it right. So if we pass, if we stack together those three arrays, then it's going to feed us those three things back to our forward method.
So if you want to like play around with training models using like you know as roar approach as possible, but without writing lots of boilerplate, this is kind of how to do it. Here's column, Namit model data from arrays and then, if you pass in whatever you pass in here, right you're going to get back here, okay, so I've passed in three things, which means I'm going to get sent three things: okay, so that's how that works! Batch size! 512, because this is you know, this data is tiny, so I can use a bigger batch size, so I'm not using really much faster, i stuff at all, I'm using fastai stuff, just to save me fiddling around with data loaders and data sets and stuff. But I'm actually going to create a standard ply torch model, I'm not going to create a loner okay. So this is a standard paper model and because I'm using ply towards that means, I have to remember to write CUDA okay, let's tick it on the GPU. So here is how we can look inside at what's going on right, so we can say it er MD train data loader to grab the iterator to iterate through the training set. We can then call next on that to grab a mini batch and that's going to return all of our X's and why tensor? And so we can then take a look at you know: here's our X's, for example, all right, and so you would expect have a think about what you would expect for this length three, not surprisingly, because these are the three things: okay and so then XS 0, Not surprisingly, okay is of length 512 and it's not actually one hot encoded, because we use an embedding to pretend it is okay. And so then we can use a model as if it's a function.
Okay, by passing to it the variable eyes, version of our tensors and so have a think about what you would expect to be returned here. Okay, so not surprisingly, we had a mini batch of 512, so we still have 5 12 and then 85 is the probability of each of the possible vocab items and of course, we've got the log of them because that's kind of what we do in pytorch. Okay, you can see here the softmax alright, so that's how you can look inside alright, so you can see here how to do everything really very much by hand. So we can create an optimizer again using standard pipe torch, so with pytorch, when you use a plate or optimizer, you have to pass in a list of the things to optimize, and so, if you call m dot parameters that will return that list for you And then we can fit and there it goes okay, and so we don't have learning rate finders and sttr and all that stuff, because we're not using a learner so we'll have to manually do learning rate annealing so set the learning rate a little bit lower and Fit again, okay, and so now we can write a little function to to test this thing out. Okay, so here's something called getnext where
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
We can pass in three characters like why full top space right, and so I can then go through and turn that into a tensor with capital T of an array of the character index for each character in that list. So basically turn those into the integers turn. Those into variables pass that to our model right and then we can do an Arg max on that to grab which character number is it, and in order to do stuff in none pile, and I use two NP to turn that variable into a lumpy array. Right and then I can return that character and so, for example, a capital T, because what it thinks would be reasonable. After seeing why space that seems like a very reasonable way to start a sentence if it was ppl a that sounds reasonable space, th, a that's bouncer e, small, a and D space. That sounds reasonable, so it seems to reflect created something sensible. Alright, so you know, the important thing to note here is our character. Model is a totally standard, fully connected model right. The only slightly interesting thing we did was to kind of do this addition of each of the inputs one at a time. Okay, but there's nothing new conceptually here, we're training it in the usual way, all right, let's now create an errand in so an iron in is when we do exactly the same thing that we did here all right, but I could draw this more simply by saying You know what, if we've got a green arrow going to a circle, let's not draw a green arrow, go into a circle again and again and again so, let's just draw it like this green arrow, going to a circle right and rather than drawing an orange arrow Going to a circle, let's just draw it like this okay, so this is the same picture, exactly the same picture as this one right, and so you just have to say how many times to go around this circle right.
So in this case, if we were to predict character, number n from characters, one through n minus one, then we can take the character. One input get some activations feed that to some new activations that go through remember orange is the hidden to hidden weight, matrix right and each time, we'll also bring in the next character of input through its embeddings okay, so that picture and that picture. I have two ways of writing the same thing, but this one is more flexible because, rather than me having to say hey, let's do it for H. I don't have to draw eight circles right. I can just say I'll, just repeat this, so I could simplify this a little bit further by saying you know what, rather than having this thing as a special case, let's actually start out with a bunch of zeros right and then, let's have all of our characters. Inside here yes yeah, so I was wondering if you can explain it be better. Why are you reusing those why you think oh they're, the same yeah? Where are you you kind of seem to be reusing? The same same weight, matrices weight, matrices yeah? Maybe this is kind of similar to what we did in convolution, your Nets like, if somehow no, I don't think so, at least not that I can see. So the idea is just kind of semantically speaking like this arrow here. This this arrow here is saying, take a character of import and represented, as some says, some set of features right, and this arrow is saying the same thing, take some character and represent as a set of features, and so is this one okay so like.
Why would the three be represented with different weight matrices, because it's all doing the same thing right and this orange arrow is saying kind of transition from character, 0 state to character 1 state 2 characters to state again it's it's the same thing it's like. Why would the transition from character 0 to 1 be different to character from transition from one or two, so the idea is like, but is to like say hey if if it's doing the same conceptual thing, let's use the exact same white matrix. My comment on convolution neural networks is that a filter, or so this apply to multiple places. I think something like a convolution is almost like: a kind of a special dot product with shared weights. Yeah. No, that's. Okay, that's very good point and in fact one of our students actually wrote a good blog post about that last year. We should dig that up. Okay, I totally see where you're coming from, and I totally agree with you all right. So, let's, let's implement this version so this time we're going to do eight as eight sees okay and so let's create a list of every eighth character from zero through seven and then our outputs will be the next character, and so we can stack them together and So now we've got six hundred thousand by eight, so here's an example so, for example, after this series of eight characters right, so this is characters north through eight. This is characters one through nine. This is two through ten.
These are all overlapping, okay, so after characters one north through eight, this is going to be the next one, okay and then, after these characters, this will be the next one all right. So you can see that this one here has 43. Is its Y value right because after those the next one will be 43? Okay? So so this is the first eight characters. This is two through nine, three through ten and so forth right. So these are overlapping groups of eight characters and then this is the the next one. Okay, so let's create that model. Okay, so again we use from arrays to create a model data class and so you'll see here we have exactly the same code as we had before. There's our embedding Linea hidden output. These are literally identical, okay and then we've replaced our value of the linear input of the embedding with something that's inside a loop, okay and then we've replaced the cell hidden thing, okay, also inside the loop. I just realize didn't mentioned last time. The use of the hyperbolic tan, hyperbolic tan, looks like this okay. So it's just a sigmoid, that's offset right and it's very common to use a hyperbolic tan inside this trend. This state to state transition because it kind of stops it from flying off too high or too low. You know it's nicely controlled back in the old days we used to use hyperbolic tanh or the equivalent sigmoid a lot as most of our activation functions.
Nowadays, we tend to use value, but in these hidden state to here in the hidden state transition weight matrices, we still tend to use hyperbolic tanh. Quite a lot so you'll see I've done that also yeah hyperbolic tanh okay. So this is exactly the same as before, but I've just replaced it with a Pollard and then here's my output. Yes, you know so a does. He have to do anything with convergence. These networks yeah we'll talk about that a little bit over time. Let's, let's, let's come back to that, though, for now we're not really going to do anything special at all. You know recognizing. This is just a standard fully connected Network.  Eight of these things now we've now got a deep, eight layer Network, which is why units starting suggest we should be concerned. As you know, as we get deeper and deeper networks, they can be harder and harder to train, but let's try training this all right. So when it goes as before, we've got a batch size of 512 we're using Adam and where it goes so we will sit there watching it. So we can then set the learning rate down back to 20 neg 3. We can fit it again and yeah. It's actually, it seems to be training, fun, okay, but we're gon na try something else, which is we're going to use this a trick that your net rather hinted at before, which is maybe we shouldn't be adding these things together, and so the reason you might want To be feeling a little uncomfortable about adding these things together is that the input state and the hidden state are kind of qualitatively different kinds of things right.
The input state is the is the encoding of this character. For us H represents the encoding of the series of characters so far, and so adding them together is kind of potentially going to lose information. So I think what your net was going to prefer that we might do is maybe to concatenate these instead of adding them. So it sound good to you, you know she's, not it okay, so let's now make a copy of the previous cell all the same right rather than using plus, let's use cat okay. Now, if we can cat, then we need to make sure now that our input layer is not from n fac-2 hidden, which is what we had before, but because we're concatenated it needs to be in fact, plus and hidden to end hidden, okay, and so now that's Going to make all the dimensions work nicely, so this now is of size, n fact, plus and hidden. This now makes it back to size n hidden again, okay and then this is putting it through the same square matrix as before. So it's still a size n here, okay, so this is like a good design. Heuristic, if you're designing an architecture is if you've got different types of information that you want to combine. You generally want concatenate it. Okay, you know adding things together, even if they're the same shape is losing information, okay and so once you've concatenated things together, you can always convert it back down to a fixed size by just tracking it through a matrix product.
Okay. So that's what we've done here again, it's the same thing, but now we're concatenating instead, and so we can fit that and so last time we got one point: seven two, this time you go at one point: six six! So it's not setting the world on fire, but it's an improvement and the improvements of it. Okay, so we can now test that with get next and so now we can pass in eight things right. So it's no before those let's go to a part of that sounds good as well so Queens, and that sounds good too. All
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Right so great, so that's enough manual, hackery, let's see if pytorch couldn't do some of this for us, and so basically what pytorch will do for us is. It will write this loop automatically, okay and it will create these linear input layers automatically, okay and so to ask it to do that. We can use the n n, dot, R and n plus so here's the exact same thing in less code, by taking advantage of height choice and again, I'm not using a conceptual analogy to say player torches doing something like it. I'm saying play torch is doing it now. This is just the code. You just saw wrapped up a little bit, reflect it a little bit for your convenience right, so where we say we now want to create an era. Ten call our it n. Then what this does is it does that for live now, notice that our for loop needed a starting point. You remember why? Right because otherwise our for loop didn't quite work, we couldn't quite refactor it out and because this is exactly the same, this needs our starting point to, and so let's give it a starting point, and so you have to pass in your initial hidden State for reasons That will become apparent later on. It turns out to be quite useful to be able to get back that here in the state. At the end, and just like we could here, we could actually keep track of the hidden state.
We get back to things we get back, both the output and the hidden state right, so we pass in the input in the hidden State when we get back the output and the hidden state. Yes, could you remind us what the hint state represents? The hidden state is H, so it's the it's, the orange circle, ellipse of activations, okay, and so it is of size, 256. Okay, all right, so we can okay, there's one other thing too to know which is in our case, we were replacing H with a new hidden state. The one minor difference in pytorch is they append the new hidden state to a list or to a tensor which gets bigger and bigger, so they actually give you back all of the hidden states. So, in other words, rather than just giving you back the final ellipse, they give you back all the ellipses stacked on top of each other, and so because we just want the final one. I was got indexed into it with minus one here, okay, other than that. This is the same code as before, put that through our output layer to get the correct, vocab size, and then we can train that alright, so you can see here, I can do it manually. I can create some hidden state. I can pass it to that area and I can see the stuff I get back. You'll see that the dimensionality of H, it's actually a rank 3 tensor. Where else in my version it was a. Let's see it was a rank. Two tensor, okay and the difference is here: we've got just a unit axis at the front, we'll learn more about why that is later, but basically it turns out.
You can have a second R and n that goes backwards, alright, one that goes forwards, one that goes backwards from the idea is neck, and then it's going to be better at finding relationships that kind of go backwards. That's quite a bi-directional eridan. Also, it turns out, you can have an error in feed to an iron in that's got a multi-layer eridan. So basically, if you have those things, you need an additional access on your tensor to keep track of those additional layers of hidden state, but for now we'll always have a one yeah and we'll always also get back a one at the end. Okay, so if we go ahead and fit this now, let's actually trade it for a bit longer. Okay, so last time we only kind of did a couple of epochs this time, we're due for a pox. What have we sit at one in egg three and then we'll do another to epochs at one in egg four, and so we've now got our lost down to one point: five, so getting better and better. So here's our get next again. Okay - and you know - let's just it was the same thing, so what we can now do is we can look through like forty times calling get next each time and then each time will replace our input by removing the first character and adding the thing that we Just predicted, and so that way we can like feed in a new set of eight characters that get them again and again, and so that way, we'll call that get next in so here are 40 characters that we've generated.
So we started out with four th OS. So we got four those of the same. The same the same. You can probably guess what happens if you can't predicting the same the same all right so it's you know it's doing. Okay, we we now have something which you know, we've basically built from scratch, and then we've said: here's how high torture effected it for us. So if you want to like, have an interesting little homework assignment this week, try to write your own version of an RNN plus all right like try to like literally like create your like. You know Jeremy's aren't in and then like type in here, Jeremy's aren't in or in your case maybe your name's, not Jeremy, which is okay too, and then get it to run. Writing your implementation, that's fast from scratch. Without looking at the piped water source code, you know like basically it's just a case of like going up and seeing what we did back here right and like make sure you get the same answers and confirm that you do so. That's kind of a good little test, simply simple at all assignment, but I think you'll feel really good when you seem like. Oh I've, just reimplemented an end alone in alright. So I'm going to do one other thing when I switched from this one when I've moved the car one input inside the dotted line right, this dotted rectangle represents the thing I'm repeating. I also watch the triangle, the output. I moved that inside as well.
Now that's a big difference, because now what I've actually done is I'm actually saying spit out an output after every one of these circles, so spit out an output here and here and here alright. So, in other words, if I have a three character input, I'm going to spit out a three character: output, I'm saying half the character 1. This will be next after character to this, be next after character. 3. This will be next, so again, nothing different and again this you know if you want to go a bit further with the assignment. You could write this by hand as well, but basically, what we're saying is in the for loop would be saying, like you know, results equals some empty list right and then would be going through and rather than returning that we're instead be saying, you know, results dot, Append that right and then like return whatever torch, dot, stat, something like that right that it made me right in my question. So now you know we now have like every step. We've created an output okay, so which is basically this picture, and so the reason was lots of reasons. That's interesting, but I think the main reason right now that's interesting is that you probably noticed this. This approach to dealing with our data seems terribly inefficient, like we're grabbing the first eight right, but then this next set all, but one of them overlap the previous one right. So we're kind of like recalculating the exact set of embeddings.
Seven out of eight of them are going to be exact, same embeddings, right, exact, same transitions, it kind of
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Seems weird to like do all this calculation to just predict one thing and then go back and recalculate seven out of eight of them and add one more to the end to calculate the next thing all right. So the basic idea, then, is to say: well, let's not do it that way, instead, let's taking non overlapping sets of characters, all right so like so here is our first eight characters here is the next day. Characters here are the next day characters so like. If you read this top left to bottom right, that would be the whole nature right, and so then, if these are the first eight characters then offset this by one starting here, that's a list of outputs right. So after we see characters zero through seven, we should predict characters 1 through 8. The XS so after 40 should come 42 as it did after 42 should come 29 as it did. Okay, and so now that can be our inputs and labels for that model, and so it shouldn't be any more or less accurate. It should just be the same right pretty much, but it should allow us to do it more efficiently. So let's try that all right. So I mentioned last time that we had a minus 1 index here, because we just wanted to grab the last triangle. Okay, so in this case we're going to grab all the triangles. So this this is actually the way it end on. Rnn creates things we we only kept the last one, but this time we're going to keep all of them. So we've made one change, which is to remove that minus one other than that.
This is the exact same code as before. Okay, so, but there's nothing much to show you here I mean, except of course, at this time, if we look at the labels, it's now 512 by eight factors we're trying to predict eight things every time through. So there is one complexity here, which is that we want to use the negative log likelihood loss function as before, right, but the ligand, if lost likelihood, loss function just like our MSE expects to receive to rank one tensors actually with the mini-batch access to rank two Tensors all right so two to mini-batches of vectors problem is that we've got eight-time steps. You know it characters in an RNN. We call it a time step right. We have eight time steps and then for each one we have 84 probabilities. We have the probability for every single one of those eight times deaths, and then we have that for each of our 512 items in the mini batch. So we have a rank 3 tensor, not a rank two tensor um. So that means that the negative log likelihood loss function is going to spit out an error. Now, frankly, I think this is kind of dumb. You know, I think it would be better if pytorch had written the loss functions in such a way that they didn't care at all about rank and they just applied it to whatever rank you gave it, but for now at least it does care about rick. But the nice thing is, I get to show you how to write a custom loss function. Okay, so we're going to create a special negative log likelihood loss function for sequences, okay, and so it's going to take an input in the target and it's got a call.
F, dot negative log likelihood lost so the pipe launched one all right, but what we're going to do is we're going to flatten our input and we're going to flatten our targets right and so, and it turns out these are going to be the first two axes That I have to be transposed so the way pytorch handles are and end data by default is the first axis is the sequence length in this case eight right, so the sequence length of an R and n is how many times deaths. So we have eight characters, so a sequence length of eight. The second axis is the batch size and then, as would expect, the third axis is the actual hidden state itself. Okay. So this is going to be eight by 512 by n, hidden, which I think was 256 yeah okay, so we can grab the size and unpack it into each of these sequence, length batch size and I'm hidden now, target mighty dot size is 512 by 8. Where else this one here was 8 by 512, so to make them match we're going to have to transpose the first two axis: okay, pytorch, when you do something like transpose, doesn't generally actually shuffle the memory order, but instead it just kind of Keeps some internal metadata to say like hey, you should treat this as if it's transposed and some things in pytorch will give you an error if you try and use it when it has these like this internal state, and I basically say error, this tensor is Not contiguous, if you ever see that error at the word contiguous after it and it goes away, so I don't know they can't do that for you apparently.
So in this particular case I got that error, so I wrote the code contiguous after it, okay and so then. Finally, we need to flatten it out into a single vector, and so we can just go a dot view, which is the same as non PI: dot reshape and minus one means as long as it needs to be okay and then the input again. We also reshape that right, but remember the input. Sorry, the the the predictions also have this axis of length. 84. All of the predicted probabilities. Okay, so so here's a custom. These are custom lost function. That's it right! So if you ever want to play around with your own loss functions, you can just do that like so and then pass that to fit okay. So it's important to remember that Fitch. Is this like lowest level, fastai abstraction, that's --! It's that this is the thing that implements the training, look, okay and so like you're, the stuff you pass it in is all standard pytorch stuff, except for this, this is our model data object. This is the thing that wraps up the test set.
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
The training set and the validation set to get that okay, your neck. Could you pass that back? So when we pull the triangle into the replicator structure right, so the the first n minus one iterations of the sequence length, we don't see the whole sequence length yeah. So does that mean that the batch size should be much bigger so that be careful? You don't mean that size, you main sequence length right, because the batch size is like some firing. Yeah. Okay! So yes, yes, if you have a short sequence length like eight yeah, the first character has nothing to go on. It starts with an empty hidden state of zeros, okay. So what we're going to start with next week is we're going to learn how to avoid that problem right, and so it's a really insightful question or concern right, and but if you think about it, the basic idea is: why should we reset this to zero? Every time you know like, if we can kind of line up these mini batches somehow so that the next mini batch joins up correctly. It represents like the next letter in leaches works. Then we'd want to move this up into the constructor right and then like pass that here and then store it here right and now we're not resetting the hidden state each time we're actually we're actually keeping the hidden state from call to call, and so the only Time that it would be failing to benefit from learning state would be like literally at the very start of the document.
So that's where, but that's where we're going to try and ahead next week. I feel like this lesson. Every time I've got a punch line coming. Somebody asks me a question where I have to like: do the punch line ahead of time. Okay, so we can fit that and we can fit that, and I want to show you something interesting, and this is coming to the punch line that another punch line that you net try to spoil, which is when we're you know remember. This is just doing a loop right, applying the same matrix multiply again and again, if that matrix multiply tends to increase the activations each time then effectively we're doing that to the power of eight right.  So this is what we call a gradient explosion right, and so we really want to make sure that the initial H naught H the initial. But if we call it, the initial L hidden that we create is is
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Like oversize, that's not going to cause our activations on average to increase or decrease right and there's actually a very nice matrix. That does exactly that called the identity matrix, so the identity matrix for those that don't quite remember their linear algebra is this. This would be a size, 3 identity, matrix all right, and so the trick about an identity matrix is anything times. An identity. Matrix is itself right, and so, therefore you could multiply it by this again and again and again and again and still end up with itself right, so there's no gradient explosion, so what we could do is instead of using whatever the default random in it is for This matrix we could instead, after we create our errand in, is we can go into that Erol in right and notice. This right, we can go m, dot, RN, n right and if we now go like so, we can get the docs for m dot, R and M right and as well as the arguments for constructing it, it also tells you the inputs and outputs for calling the Layer - and it also tells you the attributes, and so it tells you there's something called weight, H H and these are the learn about hidden to hidden weights. That's that square matrix right. So after we've constructed our M, we can just go in and say all right, m dot, R and n dot weight, h, HL, dot data, that's the tensor dot copy underscore in place torch. I that is I, for identity. In case you are wondering.
So this is an identity matrix of size n hidden, so this both puts into this weight matrix and returns the identity matrix, and so this was like. Actually, a Geoffrey Hinton paper was like hey, you know, after it was it's 2015, so after recurrent neural Nets have been around for decades. Here's like hey gang. Maybe we should just use the identity matrix to initialize this and like it actually turns out to work really. Well, and so that was a 2015 paper believe it or not, from the father of neural networks - and so here is the here is our implementation of his paper, and this is an important thing to know right when very famous people, like Geoffrey Hinton, write a paper. Sometimes in entire implementation of that paper looks like one line of code. Okay, so let's do it before we got point six one, two, five, seven we'll fit it with exactly the same parameters, and now we get 0.5 1 and in fact, can keep training 0.50. So like this tweak really really really helped. Okay, now one of the nice things about this tweak was before I could only use a learning rate of one in x3 before it started going crazy, but after identity matrix I found I could use one in egg too, because it's you know it's better behaved weight. Initialization I found I could use a higher learning rate. Okay and honestly, these things, you know increasingly we're trying to incorporate into the defaults in first day.
I you know you don't necessarily personally need to actually know them, but you know at this point we're still at a point where you know most things in most libraries, most of the time don't have great defaults, it's good to know all these little tricks. It's also nice to know if you want to improve something, what kind of tricks people have used elsewhere, because you can often borrow them yourself all right. Well, that's the end of the lesson today and so next week we will look at this idea of a stateful RNN, that's going to keep this hidden state around and then we're going to go back to looking at language models again and then. Finally, we're going to go all the way back to computer vision and learn about things like rez nets and batch norm and all the tricks that were in figured out in cats versus dogs, see you, then [ Applause, ]