fast.ai: Deep Learning Part 1 (v2) (2018)
You will learn about collaborative filtering through the example of making movie recommendations, and talk about key developments that occurred during the Netflix prize.
We will dig into some lower level details of deep learning: what happens inside the training loop, how optimizers like momentum and Adam work, and regularization using weight decay. You will learn how to think spatially about math concepts like the ‘chain rule’, ‘jacobian’, and ‘hessian’.
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Welcome back so we had a busy lesson last week and I was really thrilled to see. Actually, one of our masters, students here at USF, actually actually took what we learned took, what we learned with structured, deep learning and turned it into a blog post which, as I suspected, has been incredibly popular because it's just something people didn't know about, and so it Actually ended up getting picked up by the towards data science publication, which I quite liked actually, if you're interested in keeping up with. What's going on a data science, it's quite good, medium publication, and so Karen talked about structured, deep learning and basically introduced. You know the the the basic ideas that we learned about last week and it got picked up quite quite widely. One of the one of the things I was pleased to say actually sebastian ruder, who actually mentioned in last week's class. As being one of my favorite researchers tweeted it and then somebody from stitch fix said: oh yeah we've actually been doing that for ages, which is kind of cute. I I kind of know that this is happening in industry. A lot and I've been telling people. This is happening in industry, a lot but nobody's been talking about it and now the Karen's kind of published a blog saying, hey check out this cool thing and they all stitch fixes like yeah we're doing that already. So so that's been great great to see, and I think there's still a lot more that can be dug into with this structured people.
Learning stuff you know to build on top of Karen's post would be that maybe like experiment with some different datasets, maybe find some old, careful competitions and see like there's some competitions that you could now win with this, or some which doesn't work for would be equally Interesting and also like just experimenting a bit with different amounts of dropout, different layer sizes. You know, because nobody much is written about this. I don't think there's been any blog posts about this before that I've seen anywhere there's a lot of unexplored territory. So I think, there's a lot we could. We could build on top of here and there's definitely a lot of interest as well. One person on Twitter saying this is what I've been looking for for ages. Another thing which I was pleased to see is Nikki or who we saw his cricket versus baseball predictor, as well as his a currency predictor after less than one went on to download something a bit bigger, which was to download a couple of hundred of images of Actors - and he manually went through and checked which well, I think, first of all, he like used Google to try and find ones with glasses and ones. Without then, he manually went through and checked that that they put in the right spot, and this was a good example of one where vanilla ResNet didn't do so well with just the last layer, and so what Nikhil did was he went through and tried on freezing The layers and using differential learning rates and got up to a hundred percent accuracy, and the thing I like about these things that Nikhil was doing is the way he's he's not downloading a kegel data set he's like deciding on a problem that he's going to try And solve he's going from scratch from google and he's actually got a link here even to the suggested way to help you download images from Google. So I think this is great and actually gave a talk just this afternoon at singularity University to an executive team of one of the world's largest telecommunications companies and actually show them this post.
Because the folks there were telling me that that all the vendors that come to them and tell them they need, like millions of images and huge data. Centers will have hardware, and you know they have to buy a special software that only these vendors can provide. And I said like actually, this person has been doing it, of course, for three weeks now and look at what he's just done with a computer that cost him 60 cents an hour and they were like. They were so happy to hear that, like okay they're, you know this actually is in the reach of normal people. I'm assuming Nikhil is a normal person I haven't actually and if your proudly abnormal Nicole, I apologize, I actually went and actually had a look at his cricket classifier, and I was really pleased to see that his code actually is the exact same code that were used In Lesson one I was hoping that would be the case. You know the only thing he changed was the number of epochs. I guess so. This idea that we can take those four lines of code and reuse it to do other things. That's definitely turned out to be true, and so these are good things to show like it yeah your organization, if you're anything like the executives of this big company I spoke to today, there'll be a certain amount of like not to surprise but almost like pushback like If this was true, somebody does it all that message. She said if this was true, somebody would have told us so like.
Why isn't everybody doing this already so we'd like it? I think you might have to actually show them. You know, maybe you can build your own there's. Some internal data you've got at work or something like here. It is, you know, didn't cost me anything. It's all finished fiddly or badly. I don't know how to pronounce his name correctly, has done another very nice post on just an introductory post on how we train neural networks, and I've wanted to point this one out as being like. I think this is one of the participants in this course who has got a particular knack for technical communication, and I think we can all learn from you know from his post about about good technical writing. What I really like particularly, is that he he assumes almost nothing like he has a kind of a very chatty tone and describes everything, but he also assumes that the reader is intelligent. But you know so like he's not afraid to kind of say here's a paper or here's an equation or or whatever, but then he's going to go through and tell you exactly what that equation means. So it's kind of like this nice mix of like writing for respectfully for an intelligent audience, but also not assuming any particular background knowledge. So then, I made the mistake earlier this week of posting a picture of my first placing on the Carroll seedlings competition, at which point five other fastai students posted their pictures of them pass over the next few days.
So this is the current leaderboard for the cattle plant seedlings competition. I believe the product top six are all fastai students or in the worst of those teachers, and so I think this is like a really Oh James is just passed. He was first. This is a really good example of like what you can do, but this is trying to think it was like a small number of thousands of images, and most of the images were only were less than a hundred pixels by a hundred pixels and yet week. You know, I bet my approach was basically to say: let's just run through the notebook, we have pretty much default, took the I don't know an hour and I'm, I think the other students doing a little bit more than that, but not a lot more and basically What this is saying is yeah these these techniques work pretty reliably to a point where people that aren't using the fast - I know, libraries, you know literally really struggling. Let's just pick off. These are fastaid. A students might have to go down quite a way. So I thought that was very interesting and really really cool. So
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Today we are going to start what I would kind of call like the second half of this course, so the first half of this course is being like getting through like these are the applications that we can use this for here's kind of the code you have To write, here's a fairly high level ish description of what it's doing and we're kind of we're kind of done for that bit and what we're now going to do is go in reverse we're going to go back over all of those exact same things again, but This time we're going to dig into the detail of every one and we're going to look inside the source code of the first idea library to see what it's doing and try to replicate that. So in a students like there's not going to be a lot more best practices to show you, like I've kind of shown you the best best practices. I know, but I feel like for us to now build on top of those to debug those models to come back to part two, where we're going to kind of try out some new things. You know it really helps to understand. What's going on behind the scenes? Okay, so the goal here today is we're going to try and create a pretty effective collaborative filtering model almost entirely from scratch. So we'll use the kind of we'll use pytorch as a automatic differentiation tool and there's a GPU programming tool and not very much else. We'll try not to use its neural net features we'll try not to use fastai library anymore than necessary. So that's the goal.
So let's go back and you know we only very quickly know collaborative filtering last time. So, let's, let's go back and have a look at collaborative filtering, and so we're going to look at this movie lens data set. So the movie lens data set basically is a list of ratings.  It also has a timestamp. I haven't actually ever tried to use this. I guess this is just like what what time did that person read that movie? So that's all we're going to use for modelling is three columns. User ID movie, ID and rating, and so thinking of that in kind of structured data terms, user ID and movie ID would be categorical variables. We have two of them, and rating would be a with the independent variable, we're not going to use this for modeling, but we can use it for looking at stuff later. We can grab a list of the names of the movies as well and reproduce. This genre information I haven't tried to be interested if, during the week, anybody tries it and finds it helpful. My guess is, you might not find it helpful, we'll see. So in order to kind of look at this better. I just grabbed the users that have watched the most movies and the movies that have been the most watched and made a crosstab of it right.
So this is exactly the same data, but it's a subset and now, rather than being user movie rating, we've got user movie rating, and so some users haven't watched some of these movies. That's why some of these okay, then I copied that into Excel and you'll, see. There's a thing called collab your XLS. If you don't see it there now I'll make sure I put it there back tomorrow, and here is where I've copied that table okay. So as I go through this like setup of the problem and kind of how its described and stuff, if you're ever feeling lost, feel free to ask either directly or through the forum, if you ask through the forum and somebody answers there, I want you to answer It here, but if somebody else asks a question you would like answered, of course, just like it and your network keep an eye out for that, because kind of that's we're digging in to the details of what's going on behind the scenes, it's kind of important that At each stage you feel like okay, I can see
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
What's going on okay, so we can actually not going to build a neural net to start with. Instead we're going to do something called a matrix, factorization, the reason we're not going to build a neural net to start with is that it so happens. There's a really really simple kind of way of solving these kinds of problems which I'm going to show you, and so, if I scroll down I've, basically what I've got here is the same, the same thing, but this time these are my predictions rather than my actuals And I'm going to show you how I created these predictions? Okay, so here my actuals right here, my predictions and then down here we have our score, which is the sum of the different squared average square root? Okay, so this is. I are MSE down here. Okay, so on average we're randomly initialized model is out by 2.8. So let me show you what this model is and I'm going to show you by saying: how do we guess how much user ID number 14 likes movie ID number 27 and the prediction here? This is just at this stage. This is still random is 0.9 1. So how we calculate 0.9 1 - and the answer is we're taking it as this vector here, dot product with this vector here so dot product means 0.71 times, 0.1, 9 plus 0.8. 1 times. Point 6, 3 plus point 7, volt plus point 3, 1 and so forth and in you know, linear algebra speak because one of them is a column and one of them is a row. This is the same as a matrix product, so you can see here.
I've used the Excel fashion, matrix multiplier and that's my prediction. Having said that, if the original rating doesn't exist at all, then I'm just going to set this to 0 right because, like there's, no error in predicting something that hasn't happened. Okay, so what I'm going to do is I'm basically going to say alright, everyone of my right rate, my predictions is not going to be a neural net. It's going to be a single matrix multiplication all right now, the matrix multiplication that it's doing is basically in practice is between like this matrix and this matrix right. So each one of these is a single part of that, so I randomly initialize these. These are just random numbers that I've just pasted in here, so I've basically started off with two random matrices, and I've said, let's assume, for the time being, that every rating can be represented as the matrix product of those two. So then in Excel. You can actually do a gradient descent. You have to go to your options to the add-ins section and check the box to say turn it on and once you do, you'll see, there's something there called solver and if I go solver it says: okay, what's your objective function and you just choose the cell? So in this case we chose the cell that contains that repeats, grade error, and then it says: okay, what do you want to change? And you can see here, we've selected this matrix and this matrix, and so it's going to do a gradient descent for us by changing these matrices to try and in this case minimize this min minimize this Excel so right, GRG nonlinear is a gradient just yet so I'll say solve and you'll see it starts at 2.8 and then down here you'll see that numbers drain down.
It's not actually showing us what it's doing, but we can see that the numbers going down. So this has kind of got a near or nettie feel to it in that we're doing like a matrix product and we're doing a gradient descent. But we don't have a nonlinear layer and we don't have a second linear layer on top of that. So we don't get to call this deep learning so things where people do like deep learning, each things where they have kind of matrix products and gradient descents. But it's not deep. People tend to just call that shallow learning. Okay, so we're doing this chattering yeah all right, so I'm just going to go ahead and press escape to stop it because I'm sick of waiting - and so you can see - we've now got down to the 0.39 all right. So, for example, it guessed that movie 72 for sorry movie, 27 for user. Seventy two would get 4.4 for rating 2772 and actually got a four ready. So you can see like it's it's it's doing something quite useful. So why is it doing something quite useful? I mean something to note here is the number of things we're trying to predict here? Is there's 225 of them right and the number of things we're using to predict is that times two so hundred and fifty of them? So it's not like we can just exactly fit. We actually have to do some kind of machine learning here.
So basically, what this is saying is that there does seem to be some way of making predictions in this way, and so for those of you that have done some linear algebra, and this is actually a matrix decomposition normally in linear algebra. You would do this using a analytical technique or using some techniques that are specifically designed for this purpose, but the nice thing is that we can use gradient descent to solve pretty much everything, including this. I don't like to so much think of it from a linear algebra point of view, though I like to think of it from an insured point of view, which is this, let's say movie sorry, let's say movie id 27 is Lord of the Rings part 1, and, Let's say move and so let's say we're trying to make that prediction for user 2072. Are they going to like Lord of the Rings, part 1, and so conceptually that particular movie? Maybe there's like this 4, so there's 5 numbers here and we could say like well what, if the first one was like, how much is it sci-fi and fantasy, and the second one is like how recent a movie and how much special effects is there? You know, and the one at the top might be like how dialogue-driven is it right, like let's say those kind of five, these five numbers represented particular things about the movie, and so, if that was the case, then we could have the same five numbers for the User saying like ok, how much does the use of like sci-fi fantasy? How much does the user, like modern, modern, CGI, driven movies? How much does this give us a like dialogue, different movies, and so, if you then took that cross-product, you would expect to have a good model right would expect to have a reasonable reading now the problem is, we don't have this information for each user? We don't have the information for each movie, so we're just going to like assume that this is a reasonable kind of way of thinking about this system and, let's, unless stochastic gradient, descent, try and find these models right.
So so, in other words, these these factors, we call these things, factors these factors and we call them factors because you can multiply them together to create this, not they're factors and how many addresses these factors we call them latent factors because they're not actually. This is not actually a vector that we've like named and understood and like entered in manually, we've kind of assumed that we can think of movie ratings. This way, we've assumed that we can think of them as a dot product of some particular features about a movie and some particular features of to look what users, like those kinds of movies right and then we've used gradient descent to just say: okay, try and find Some numbers that work, so that's that's, basically the technique right and it's kind of the end and the entirety is in this printing right. So that is collaborative filtering using what we call probabilistic matrix factorization and, as you can see, the whole thing is easy to do in an excel spreadsheet and the entirety of it really is this single thing, which is a single matrix multiplication plus randomly initializing? If it would be better to cap this to 0 and 5 - maybe yes yeah and we're gon na do that later. Right, there's a whole lot of stuff. We can do to improve this. This is like our simple as possible, starting point all right, so so what we're going to do now is we're going to try and implement this in Python and run it on the whole data set.
Another question is: how do you figure out how many you know how it's clear: how long are the matrix five yeah yeah, so something to think about, given that this is like movie 49 right and we're looking at a rating for movie 49. Think about this. This is actually at embedding matrix, and so this length is actually the size of the embedding matrix. I'm not saying this is an analogy. I'm saying it literally. This is literally an embedding mattress. We could have a one hot encoding where 72, where a one is in the 72nd position, and so we'd like to look it up and it would return this list of five numbers. So the question is actually: how do we decide on the dimensionality of our embedding vectors and the answer to that question is we have no idea, we have to try a few things and see what was the underlying concept? Is you need to pick an embedding dimensionality, which is enough to reflect the kind of true complexity of this causal system, but not so big that you have too many parameters that it could take forever to Tehran or even
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
With regularization in my overfit, so what does it mean when the factor is negative, that the factor being negative in the movie case would mean like? This is not dialogue-driven? In fact, it's like the opposite dialogue here is terrible. A negative for the user would be like. I actually dislike modern CGI movies, so it's not from zero to whatever it's the range of score it'd be negative. This is a range of score, even like no net Maxim. No there's no constraints at all here. These are just standard. Embedding matrices questions. So first question is: why do what why can we trust this embeddings because, like if you take a number six, it can be expressed as 1 into 6 or like 6 into 1 or 22 3 & amp 3 into 2? All so you're saying like we could like reorder these higher. Hardly the value itself might be different as long as the product is something well, but you see we're using gradient descent to find the best numbers so like once, we've found a good minimum. The idea is like yeah, there are other numbers, but they don't give you as good an objective value and, of course we should be checking that on a validation set really which we'll be doing in the Python version. Okay - and the second question is when we have a new movie or a new user to be a 30 trainer model. That is a really good question and there isn't a straightforward answer to that time.
Permitting will come back, but basically you would need to have like a kind of a new user model or a new movie model that you would use initially and then over time. Yes, you would then have to retrain the model so, like I don't know if they still do it, but Netflix used to have this thing that when you were first on boarded on Netflix, it would say like what movies do you like and you'd have to go Through and let's say a bunch of movies you like, and it would then my train is moral. Just find the nearest movie yeah, you could use nearest neighbors for sure, but the thing is initially, at least in this case we have no columns to describe a movie. So if you had something about like the movies genre release date, who was in it or something you could have some kind of non collaborative filtering model and that's kind of what I meant a new movie model. You have to have some some kind of
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Predictors, okay, so a lot of this is going to look familiar and and the way I'm going to do. This is again it's kind of this top-down approach, we're going to start using a few features of pytorch and fastai, and gradually we're going to redo it a few times in a few different ways, kind of doing a little bit deeper, each time um regardless. We do need a validation set, so we can use our standard cross-validation indexes approach to grab a random set of ID's. This is something called weight decay which we'll talk about later in the course for those of you that have done some machine learning. It's l2 regularization. Basically - and this is where we choose - how big a embedding matrix do we want okay? So again, you know: here's where we get our model data object from CSV passing in that ratings file, which remember looks like that. Okay, so you'll see like stuff tends to look pretty familiar after a while, and then you just have to pass in the. What are your rows effectively, what are your columns effectively and what are your values effectively? Alright, so any any collaborative filtering recommendation system approach, there's basically a concept of like you know a user and an item now they might not be users and items like if you're doing the Ecuadorian, groceries, competition.
There are stores and items and you're trying to predict how many things are you going to sell at this store of this type, but, generally speaking, just this idea of like you've got a couple of kind of high cardinality, categorical variables and something that you're measuring and You're kind of conceptualizing and saying okay, we could predict the rating. We can predict the value by doing this this dot. For that, interestingly, this is kind of relevant to that that last question or suggestion an identical way to think about this. What I've expressed this is to say when we're deciding whether user 72 will like movie twenty-seven, it's basically saying which other users liked movies that 72 liked and which other movies were liked by people like you, user 72. It turns out that these are basically two ways of saying the exact same thing. So basically, what collaborative filtering is doing you know kind of conceptually is to say okay, this movie and this user, which other movies are similar to it in terms of like similar people enjoyed them and which people are similar to this person based on people that, like The same kind of movies, so that's kind of the underlying structure at any time. There's an underlying structure like this. That kind of collaborative filtering approach is likely to be useful. Okay, so so you yeah so there's basically two parts. The two bits of your thing that you're factoring and then the the value of the dependent variable.
So as per usual, we can take our model data and ask for a learner from it and we need to tell it what size of any matrix to use. How many sorry, what validation set index is to use what batch size to use and what optimizer to use and we're going to be talking more about optimizes? Surely we want to Adam today, Adam next week or the week after, and then we can go ahead and say fit alright and it all looks pretty similar interest is usually interestingly, I only had to do three pops, like this kind of model, seem to Train super Quickly, you can use the learning rate finder as per usual. All the stuff you're familiar with will work fine, and that was it so this talk, you know about two seconds: the Train there's no free trained anything's here this is from random from scratch. Okay, so this is our validation set and we can compare it. We have. This is a mean squared error, not a root mean squared error, so we can take a square root. So with that last time I ran it was point, seven, seven, six and that's 0.88 and there's some benchmarks available for this data set and when I scrolled through and found the bench the best benchmark. I could find here from this recommendation system specific library. They had point nine one, so we've got a better loss in two seconds already. So that's good! So, that's basically how you can do collaborative filtering with the faster I library without thinking too much, but so now we're going to dig in and try and rebuild that we'll try and get to the point that we're getting something around 0.7. Seven point seven eight from scratch, but if you want to do this yourself at home, you know without worry about the detail.
That's you know those three lines of code: here's what you need! Okay, so we can get the predictions in the usual way and you know we could. For example, plot SNS is Seabourn, see one's a really great flooding library. It sits on top of matplotlib, it actually leverages matplotlib. So anything you learn about matplotlib will help you with SIBO, and it's got a few like nice. Little plots like this joint plot here is: I'm doing predictions against against actuals. So these are my actual season. My predictions and you can kind of see the the shape here is that, as we predict higher numbers, they actually are higher numbers, and you can also see the histogram of the predictions and a histogram of the ashes. So that's kind of floating. That is to show you another interesting visualization. Would you please explain the n factors why it's set to 50? It's set to 50 because I tried a few things in the world. It's the dimensionality of the embedding images or to think for it. Another way it's like how you know, rather than five, it's fit Jeremy. I have a question about suppose that your recommendation system is more implicit, so you have zeros or ones instead of just actual numbers right. So basically, we would then need to use a classifier instead of regresa. I have to sample the negative or something like that. So if you don't have it, which is up once, let's say like just kind of implicit feedback - oh I'm not sure we'll get to that.
One in this class, but what I will say is like in the case that you just doing classification rather than regression. We haven't actually built that in the library, yet maybe somebody this week that wants to try adding it. It would only be a small number of lines of code. You basically have to change the activation function, to be a sigmoid, and you would have to change the criterion or the loss function to be cross-entropy rather than rmse, and that will give you a classifier rather than a regresar.  We've all have that working. Ok, so I said that we're basically doing a dot product right or you know a dot product is kind of the vector version I guess of this matrix product, so we're basically doing
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Each of these things times each of these things and then add it together. So let's just have a look at how we do that in Python, so we can create a tensor in pytorch. Just using this little capital. T thing you can just say: that's the first day I version the full version is torch dot from I'm pie or something, but I've got to set up, so you can possibly pass in even a list of lists, so this is going to create a torch tensor With one two three four and then here's a torch tensor with two to ten ten. Ok, so here are two more chances, I didn't say doc, CUDA so they're, not on the GPU they're sitting on the CPU, just FYI. We can multiply them together, right and so anytime. You have a mathematical operator between tensors in numpy or pipe torch. It will do element wise assuming that they're the same dimensionality, which they are they're both to about two okay, and so here, we've got 2 by 2. Is 4 3 by 10 is 30 and so forth? Ok, so there's a a times B. So, if you think about basically what we want to do here is we want to take ok, so I've got 1 times. 2 is 2 2 times. 2 is 4, 2 plus 4 is 6, and so that is actually the dot product between 1 2 & amp, 2 4, and then here, we've got 3 by 10 is 34 by 40. Sorry, 4 by 10 is 40, 30 and 40 and 70 so in other words a times B, dot some along the first dimension. So that's summing up the columns, in other words, across a row.
Okay, this thing here is doing the dot product of each of these rows with each of these rows, so it makes sense - and obviously we could do that with you know some kind of matrix multiplication approach, but I'm trying to really do things with this little special Case stuff as possible: ok, so that's what we're going to use for our dot products from now on. So basically all we need to do now is remember. We have the data we have is not in that crosstab format, so in excel. We've got it in this crosstab format, but we've got it here in this listed format: here's our movie rating user movie revenue. So conceptually we want to be like looking up this user into our embedding matrix to find their 50 factors looking up that movie to find their 50 factors and then take the dot product of those two 50 long vectors. So, let's do that to do it, we're going to build a layer, our own custom, neural net layer? That's not right! So the the the more generic vocabulary we call. This is we're going to build a high torch module. Okay, so a pytorch module is a very specific thing. It's something that you can use as a layer and a neural net. Once you've created your own height watch module, you can throw it into a mirror on it and a module works by assuming we've already got once a cordon model.
You can pass in some things in parentheses and it will calculate it right so, assuming that we already have a modular product, we can instantiate it like so to create our product object, and we can basically now treat that like a function right. But the thing is it's not just a function because we'll be able to do things like take derivatives of it stack them up together into a big stack of neural network layers, blah blah blah. So it's basically a function that we can kind of compose. Very conveniently so here, how do we define a module which, as you can see here, returns a dot product? Well, we have to create a Python class, and so, if you haven't done oo before you're going to have to learn because all my torch modules are written in Python oo and that's one of the things I really like about pytorch - is that it doesn't reinvent Totally new ways of doing things by tensorflow does all the time in pytorch that you know really tend to use pythonic ways to do things. So in this case, how do you create? You know some kind of new behavior? You create a Python plus, it's so Jeremy suppose that you have a lot of data, not just a little bit of data you can have in memory. Will you be able to use fossae I to solve glory filtering? Yes, absolutely it's! It uses mini-batch stochastic gradient descent, which does have a batch at a time. The this particular version is going to create a pandas data frame and pandas data frame has to live in memory.
 You know that would be impressive. If that did happen, I guess you would have to instead save that as a be calls array and create a slightly different version that reads from a because array just streaming in or maybe from a desk data frame, which also so it would be easy to do. I don't think I've seen real-world situations where you have 512 gigabyte collaborative filtering matrices, but yeah. We can do it. Okay now this is pytorch specific. This next bit is that when you define like the actual work to be done, which is here return user times movie dot, some you have to put it in a special method called forward. Okay - and this is this idea that, like it's very likely you're on that right in a neural net, the thing where you calculate the next set of activations is called the the forward pass and so that's doing a forward calculation. The gradients is called the backward calculation. We don't have to do that because pytorch calculates that automatically, so we just have to define forward. So we create a new class. We define forward, and here we write in our definition of dot product. Ok, so that's it so now that we've created this class definition, we can instantiate our model right and we can call our model and get back the numbers be expected.
Okay, so that's it! That's how we create a custom, pytorch layer, and if you compare that to like any other library around pretty much, this is way easier. Basically, I guess because we're leveraging what's already in person, so let's go ahead and now create a more complex module and we're going to basically do the same thing. We've got to have a forward again we're going to have our users x, movies, dot sum, but we're
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Going to do one more thing before hand, which is we're going to create two embedding matrices and then we're going to look up our users and our movies in those inventing matrices. So let's go through and and do that. So the first thing to realize is that the uses, the user IDs and the movie IDs may not be contiguous. You know like they're, maybe they start at a million and go to a million in 1000. So right. So if we just used those IDs directly to look up into an embedding matrix, we would have to create an embedding matrix of size, 1 million 1000 right, which we don't want to do so. The first thing I do is to get a list of the unique user IDs, and then I create a mapping from every user ID to a contiguous integer. This thing I've done here where I've created a dictionary which maps from every unique thing to a unique index is well worth studying during the week because, like it's is super super handy, it's something you very very often have to do in all kinds of machine learning. All right - and so I won't go through it here - it's easy enough to figure out if you can't figure it out, just ask on the forum anyway. So once we've got the mapping from user to a contiguous index, we then can say: let's now replace the user ID column with that contiguous index right, so pandas dot apply applies an arbitrary function and python lambda is how you create an anonymous function on the fly And this anonymous function simply returns the NS through the same thing for movies, and so after that we now have the same ratings table we had before, but our IDs have been mapped to contiguous integers.
Therefore, they're things that we can look up into an embedding matrix. So let's get the count of our users in our movies and let's now go ahead and try and create our Python version of this okay. So earlier on, when we created our simplest possible pytorch module, there was no like state. We didn't need a constructor, because we weren't like saying how many users are there or how many movies are there or how many factors do we want or whatever right anytime. We want to do something like this, where we're passing in and saying, we want to construct our module with this number of users and this number of movies. Then we need a constructor for our class and you create a constructor in Python by defining a dunder init underscore underscore init underscore underscore yet special name, so this just creates a constructor, and if you haven't done over before you wanted to do some study during the Week, but it's pretty simple idea: this is just the thing that when we create this object, this is what gets wrong. Okay, again special python thing. When you create your own constructor, you have to call the parent class constructor and if you want to have all of the cool behavior of a PI porch module, you get that by inheriting from an end, module neural net module. Okay. So, basically, by inheriting here and calling the superclass constructor, we now have a fully functioning pytorch layer.
Okay, so now we have to give it some behavior, and so we give us some behavior by storing some things in it all right. So here we're going to create something called self dot you users, and that is going to be an embedding layer. A number of rows is an user's number of columns is in factors. So that is exactly this right. The number of rows is M users. Number of columns is inventors and then we'll have to do the same thing for movies. Okay, so that's going to go ahead and create these two randomly initialized arrays. However, when you randomly initialize over an array, it's important to randomly initialize it to a reasonable set of numbers like a reasonable scale right. If we randomly initialize them from like naught to a million, then we would start out - and you know these things would start out.  So I just kind of manually figured here like okay about what size numbers are going to give me about the right readiness, and so we don't. We know we did ratings between about normal five. So if we start out with stuff between about naught and 0.05, then we're going to get ratings of about the right level. You can easily enough, like that, calculate that in in neural nets, there are standard algorithms for basically doing doing that calculation.
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
And the basic the key algorithm is something called initialization from climbing her, and the basic idea is that you take the yeah. You basically set the weights equal to a normal distribution with a standard deviation, which is basically inversely proportional to the number of things in the previous layer and so in our previous layer. So in this case, we basically if you basically take that nor to 0.05 and multiply it by the fact that you've got 40 things with a 40 or 50 things coming out of it. 50 50 things coming out of it and then you're going to get something about the right size. pytorch has already has like her initialization class they're like we don't in normally in real life, have to think about this. We can just call the existing initialization functions, but we're trying to do this all like from scratch here: okay without any special stuff going on so there's quite a bit of pytorch, notation here, so self dot. U we've already set to an instance of the embedding class. It has a dot weight attribute which contains the actual the actual embed images, so that contains this. The actual embedding matrix is not a tensor. It's a variable. A variable is exactly the same as a tensor. In other words, it supports the exact same operations as a tensor, but it also does automatic differentiation. That's all a variable is basically to pull the tensor out of a variable. You get its data attribute, okay, so this is so.
This is now the tensor of the weight matrix of the self dot you're inventing, and then something that's really handy to know is that all of the tensor functions in pytorch. You can stick an underscore at the end, and that means do it in place all right, so this is say, create a random, uniform, random number of an appropriate size for this tensor and don't return it but actually fill in that matrix unless okay. So that's a super handy thing to know about. I mean it wouldn't be rocket science. Otherwise we would have to have gone , okay, here's the non in-place version. That's what saves us! Some typing saves us some screen noise. That's all! Okay! So now we've got our randomly initialized, embedding weight, matrices and so now the forward, I'm actually going to use the same columnar model data that we used for Russman, and so it's actually going to be passed, both categorical variables and continuous variables. And in this case there are no continuous variables, so I'm just going to grab the 0th column out of the categorical variables and call it users and the first column and call it movies. Okay.  I've lots to do about too lazy out that we do have a special class with this, but I'm trying to avoid creating a special class so just going to leverage this columnar model data plus okay. So we can basically grab our user and movies mini-batches right and remember.
This is not a single user in a single movie. This is going to be a whole mini batch of them. We can now look up that mini batch of users in our embedding matrix. U and the movies in are embedding matrix. Okay, so this is like exactly the same is just doing an array lookup to grab the user ID numbered value, but we're doing that a whole mini batch at a time right, and so it's because pytorch can do a whole mini batch at a time with Pretty much everything that we can get really easy speed up. We don't have to write any loops on the whole to do everything through our mini batch and in fact, if you do ever loop through your mini batch manually, you don't get GPU acceleration. That's really important to know right, so you never want to loop. Have a for loop going through your mini batch. You always want to do things in this kind of like whole mini batch at a time, but pretty much everything imply torch. Does things are holding events at a time, so you shouldn't have to worry about it and then here's our product just like before right so having to find that I'm now going to go ahead and say alright, my X values is everything except the rating and the Timestamp in my writings table, my Y is my rating and then I can just say: okay, let's grab a model data from a data frame using that X and that Y - and here is our list of categorical variables: okay and then so, let's now instantiate that pytorch object.
Alright, so we've now created that from scratch, and then the next thing we need to do is to create an optimizer. So this is part of pytorch. The only fastai thing here is this line right because that's like, I don't think showing you how to build data sets and data load is interesting enough. Really, we might do that in part two of the course and it's actually so straightforward. Like a lot of, you are already doing it on the forums. So I'm not going to show you that in this part, but if you're interested feel free to talk on the forums about it, but I'm just going to basically take the thing that feeds us. Data is a given particularly cuz. These things are so flexible right. You know if you've got stuff enough data frame, you can just use this, you don't have to rewrite it. So that's the only fastai thing we're using. So this is a pytorch thing, and so optiom is the thing and pytorch that gives us an optimizer will be learning about that very shortly. So it's actually the thing. That's going to update our weights. pytorch, calls them the parameters of the model. So earlier on, we set model, equals embedding dot, blah blah right and because embedding dot derives from NN module. We get all of the pytorch module behavior and one of the things we got for free is the ability to say got parameters. So that's pretty.
That's pretty any right, that's the thing that basically is going to automatically give us a list of all of the weights in our model that have to be updated, and so that's what gets passed to the optimizer. We also passed the optimized at the learning rate, the weight decay which we'll talk about later and momentum that we'll talk about later. Okay, one other thing that I'm not going to do right now, but we will do later, is to write a training loop. So the training loop is a thing that lives for each mini batch and updates the weight to subtract the gradient times. The moment there's a function in fastai, which is the training loop, and it's it's pretty simple here. It is right for a POC in epochs. This is just the thing that shows a progress bar so ignore this for X, comma Y, in my training data loader calculate the loss print out the lots you know in a progress bar call any callbacks you have and at the end call the call the metrics On the validation, alright, so there's there's just eh Apoc go through each mini batch and do one step of optimizer step is basically going to take advantage of this optimizer, but we'll be writing that from scratch shortly. So this is notice we're not using a learner. Okay, we're just using a hi book module, so this this fit thing, although it's passed to a part of fastai, it's like lower down the layers of abstraction. Now this is the thing that takes a regular high torch model.
So if you ever want to like skip as much faster eye stuff as possible, like you've, got some high torch model, you've got some code on the internet. You basically want to run it that you don't want to write your own training loop. Then this is. This. Is what you want to do? You want to call fast, a high speed version, and so what you'll find is like the library is designed so that you can kind of dig in at any layer abstraction you like right and so at this layer of abstraction you're. Not going to get things like stochastic gradient descent with restarts you're not going to get like differential learning rates like all that stuff. That's in the learner like you could do it, but you'd have to write it all about by hand yourself. Alright and that's the downside of kind of going down to this level of abstraction, the upside is that, as you saw, the code for this is very simple: it's just a simple training loop. It takes a standard 5 torch model, so this is like this is a good thing for us to use here. We can, we just call it, and it looks exactly like what we used to see all right. We got our validation and training loss for the 3 e bus now you'll notice that we wanted something around 0.76, so we're not there. So, in other words, the the the default fastai collaborative, dory rhythm is doing something smarter than this.
So we're going to try and do that, one thing that we can do since we're calling our you know this lower level fifth function, there's no learning rate and kneeling. We could do our own learning rate annealing, so you can hear it see here. There's a first day I function called set learning rates, you can pass in a standard, height, watch optimizer and pass in your new learning rate and then call fit again, and so this is how we can let manually do a learning rate schedule, and so you can See, we've got a little bit better 1.13, where you still got a long way to go okay, so I think what we might do is we might have a seven minute break and then we're going to come back and try and improve this core of it. For those who are
Pause
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Interested somebody was asking me the break for a kind of a quick walkthrough, so this is totally optional. But if you go into the first day, I library there's a model py file and that's where fit is which we're just looking at, which goes through each epoch. In epochs and then goes through each x and y in the mini batch, and then it calls this step function. So the step function is here and you can see the key thing is it calculates the output from the model, the models for M right, and so, if you remember our dot product, we didn't actually call model dot forward. We just called model parentheses and that's because the N n dot module automatically. You know when you call it as if it's a function, it passes it along to forward okay. So, that's that's what that's doing there right and then the rest of this world will learn about shortly, which is basically doing the the loss function and the backward pass. Okay. So, for those who are interested, that's that's kind of gets you a bit of a sense of how the cone it's structured. If you want to look at it and, as I say like the the faster I code is designed to both be world-class performance, but also pretty easy to read so like feel free, like take a look at it and if you want to know what's going on. Just ask on the forums and if you you know, if you think, is anything that could be clearer, let us know because yeah the code is definitely now we're going to be digging into the code or in law.
Okay, so
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Let's try and improve this a little bit and, let's start off by improving it in Excel, so you might have noticed here that we've kind of got the idea that use a 72. You know like sci-fi, modern movies, with special effects. You know whatever and movie number 27 is sci-fi and that special effects so much dialogue, but we're missing an important case which is like user 72 is pretty enthusiastic on the hall and on average rates things higher and Highland. You know and movie 27. You know it's just a popular movie, you know it's just on average its higher. So what would really like is to add a constant for the user and a constant for the movie and remember in neural network terms. We call that a bias, that's what we want to add a bias, so we could easily do that and if we go into the bias tab here, we've got the same data as before, and we've got the same latent factors as before, and I've just got one Extra row here and one extra column here - and you won't be surprised here - that we now take these same matrix multiplication as before, and we add in that - and we add in that - okay - so that's bias so other than that we've got exactly the same loss function Over here and so just like before, we can now go ahead and solve that, and now our changing variables include the bias and we can say solve, and if we leave that for a little while it will come to a better result than we had before.
Okay, so that's the first thing: we're
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Going to do to improve our model and there's really very little show just to make the code a bit shorter. I have to find a function called get embedding which takes a number of inputs and a number of factors. So the number of rows and the embedding matrix, Nomos they're both medications creates the embedding and then randomly initializes it. I don't know why I'm doing negative to positive here and it zeroed. Last time, honestly, it doesn't matter much as long as it's in the right ballpark and then we return that initialized emitting. So now we need not just our users by factors which are Chuck into you, our movies, by factors which I've shocked into M. But we also need users by one which will put into UV user bias and movies by one which will put into the movie bias okay. So this is just doing a list. Comprehension going through each of the tuples create an embedding for each of them and putting them into these things. Okay, so now our forward is exactly the same as before: u times M sub, I mean this is actually a little confusing because we're doing it into two steps. Maybe they make it a bit easier. Let's pull this out. Put it up here, put this in parentheses! Okay, so maybe that looks a little bit more familiar all right, you times, n dot, some that's the same dot product and then here it is going to add in our user, pious and our movie bus dot squeeze is the pytorch thing that adds an additional Unit axis, that's not going to make any sense if you haven't done broadcasting before I'm not going to do a broadcasting in this course, because we've already done it and we're doing it in the machine learning course.
But basically in in short, broadcasting is what happens when you do something like this?  How do you add a vector to a matrix and basically, what it does is it duplicates the vector so that it makes it the same size as the matrix and the particular way, whether it duplicates it across columns or down rows or how it does? It is called broadcasting the broadcasting rules are the same as numpy Pytor didn't actually used to support broadcasting, so I was actually the guy who first added broadcasting to pytorch using an ugly hack and then the pipe or authors did an awesome job of supporting it. Actually, inside the language, so now you can use the same broadcasting operations in five torches non-player. If you haven't dealt with this before, it's really important to learn it because, like it's, it's kind of the most important fundamental way to do computations quickly in the high-end paid warship. It's the thing that lets you not have to do loops. How could you imagine here if I had to look through every row of this matrix and add each did you know this vector to every row? It would be slow, the you know a lot more code and the idea of broadcasting. It actually goes all the way back to APL, which was a language designed in the 50s by an extraordinary guy called Ken Iverson, yeah APL was originally designed or written out as a new type of mathematical notation.
He has this great essay called notation as a tool for thought, and the idea was that, like really good, notation could actually make you think of better things and part of that notation. Is this idea of broadcasting I'm incredibly enthusiastic about it and we're gon na use? It plenty so either watch the machine, learning lesson or you know: google numpy broadcasting for information anyway. So basically it works reasonably intuitively we can add on. We can add the vectors to the matrix, all right.
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Having done that, we're now going to do one more trick, which is, I think it was your net asked earlier about. Could we squish the ratings to be between one and five, and the answer is we could right and specifically what we could do is we could put it through a sigmoid function? All right so remind you. A sigmoid function looks like that right, and this is that's one. Okay, we could put it through a secret function, so we could take like four point: nine six and put it through a sigmoid function and like that you know that's kind of high, so it kind of be over here somewhere right and then we could multiply that Sigmoid, like the result of that by five, for example, all right - and in this case we want it to be between one and five right. So maybe we would multiply it by four and add one instance. That's the basic idea, and so here is that trick. We take the result, so the result is basically the the thing that comes straight out of the dot product, plus the addition of the biases and put it through a sigmoid function. Now in pytorch. Basically, all of the functions you can do to tensors are available inside this thing called capital F, and this is like totally standard in pytorch. It's actually called torch and or functional, but everybody, including all of the pipe torch, Doc's import, torch, start and end, are functional as capital F, all right, so capital, F, dot, sigmoid means a function called sigmoid that is coming from tortures, functional module right, and so that's Going to apply a sigmoid function for the result, so I squish them all between zero and one using that nice little shape, and then I can multiply that by five minus one plus four right and then add on one and that's gon na give me plumbing between One and five okay, so like there's, no need to do this, I could comment it out and it'll still work right, but now it has to come up with a set of calculations that are always between one and five right.
Where else, if I leave this in then it's like makes it really easy. It's basically like. Oh, if you think this is a really good movie, just calculate a really high number. It's a really crappy movies, low number and I'll make sure it's in the right regions. So, even though this is a neural network, it's still a good example of this kind of like, if you're doing any kind of parameter, fitting, try and make it so that the thing that you want your function to return, it's like it's easy for it to return That, okay, so that's why we do that that function squishing. So we call this embedding dot bias, so we can create that in the same way as before you'll see here, I'm calling dr. to put it on the GPU because we're not using any learner stuff, normally it'll all happen for you, but we have to manually, say Put it on the GPU: this is the same as before, create our optimizer fit exactly the same as before, and these numbers are looking good, all right and again, we'll do a little change to our learning rate, learning rate schedule and we're down to 0.8. So we're actually pretty close, pretty close. So that's the key steps - and this is how this is, how most collaborative filtering is done and unit reminded me of an important point, which is that this is not strictly speaking a matrix factorization, because strictly a matrix, factorization would take that matrix by that matrix to Create this matrix and remembering anywhere that this is empty like here or here, we're putting in a zero right we're saying if the original was empty, put in a zero right now, normally you can't do that with normal matrix, factorization normal matrix factorization.
It creates the whole matrix, and so it was a real problem actually when people used to try and use traditional linear algebra for this, because when you have these sparse matrices like in practice, this matrix is not doesn't have many gaps because we picked the users that Watch the most movies and the movies that are the most watched.  That's gives terrible answers, so this probabilistic, matrix factorization approach takes advantage of the fact that our data structure actually looks like this rather than that cross tab right, and so it's only calculating the loss for the user ID movie ID combinations that actually appear that's its. If you like, use red a1 movie, I think 102 9 should be 3. It's actually three and a half sauce is 0.5. Like there's nothing here, that's ever going to calculate a prediction or a loss for a
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
User movie combination that doesn't appear in this table by definition, the only stuff that we can appear in a mini batch is what's in this table. Okay and like a lot of this happened, interestingly enough, actually in the Netflix price, so before the Netflix prize came along. There's probabilistic matrix factorization, it had actually already been invented, but nobody noticed all right and then, in the first year of the Netflix price, someone wrote this like really really famous blog post, where they basically said like hey check this out. Incredibly, simple technique works incredibly well when suddenly, all the net fix leaderboard entries - and so that's quite a few years ago now - and this is like now - every collaborative filtering approach. Does this? Not every collaborative filtering approach adds this sigmoid thing by the way. It's not like rocket science. This is this is not like the NLP thing we saw last week, which is like hey. This is a new state-of-the-art like this is, you know, not particularly uncommon, but there are still people that don't do this. It definitely helps a lot. I have to have this, and so actually you know what we could do is maybe now's a good time to have a look at the definition of this right. So the column data module contains all these definitions and we can now compare this to the thing we originally used, which was whatever came out of collaborative data set all right.
So let's go to collab filter data set here it is and we called get learner all right, so we can go down to get Elena and that created a collab filter. Learner passing in the model from get model is get model, so created an embedding bias, and so here is embedding drop bias, and you can see here here. It is like it's the same thing. There's the embedding for each of the things. Here's our forward that does the you times, I dot some plus plus sigmoid. So in fact we have just actually rebuilt. What's in the past, our library, literally okay, it's a little shorter and easier because we're taking advantage of the fact that there's a special collaborative filtering data set. So we can actually we're getting past in the users and the items and we don't have to pull them out of cat Kant's, but other than that. This is exactly the same. So hopefully you can see like the faster you have. Ivory is not some inscrutable code containing concepts. You can never understand. We've actually just built up this entire thing from scratch ourselves, and so why did we get 0.76 rather than 0.8? You know, I I think it's simply because we used stochastic gradient descent with restarts or the cycle multiplier and an atom optimizer. You know like a few little training chase, some I'm looking at this and thinking that is.
We could totally improve this small, but maybe looking at the date and doing some tricks with the date, because this this is kind of a just, a regular kind of smaller, no way yeah. You can add more features, yeah exactly exactly so like now that you've seen this you could now you know, even if you didn't have embedding dot bias in a notebook that you've written yourself through some other model, that's in fastai. You could look at it in faster and be like, oh, that does most of the things that I'd want to do, but it doesn't deal with time, and so you could just go. Oh okay, let's grab it copy it. You know pop it into my notebook and, let's create you, know the better version all right and then you can start playing that and you can now create your own model class from the open source code here and so yeah. Your that's mentioning a couple things. We could do we could try and incorporate in time stamp. So we could assume that maybe well maybe there's just like some for a particular user over time users tend to get more or less positive about movies. Also remember, there was the list of genres for each movie. Maybe we could incorporate that so one problem is it's a little bit difficult to incorporate that stuff into this? Embedding bias.
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Model because it's kind of it's pretty custom right, so what we're going to do next is we're going to try to create a neural net version of this hey. So the basic idea here is we're going to take exactly the same thing as we had before. Here's our list of users right and here is Erin Bates, alright and here's our list of movies, and here is our embedded right and so, as you can see, I've just kind of transposed the movie ones so that so that they're all in the same orientation - and Here is our user movie rating but D cross tab, okay, so in the original format. So each row is a user movie rating okay. So the first thing I do is, I need to replace user 14 with that users contiguous in this right, and so I can do that in Excel using this match. That basically says what you know: how far down this list do you have to go, and it said user 14 was the first thing in that list: okay, user 29 was the second thing in that list, so forth, okay. So this is the same as that thing that we did in our Python code, where we basically created a dictionary to master. So now we can for this particular user movie rating combination. We can look up the appropriate embedding right, and so you can see here what it's doing is it's saying all right, let's basically offset from the start of this list and the number of rows we're going to go down is equal to the user index and the Number of columns we're going to go across is one two three four or five okay, and so you can see what it does.
Is it creates point one nine point, six, three point three one here. It is point one, nine point: okay, so so this is literally modern. Embedding this but remember this is exactly the same as doing a one hot encoding right, because if instead this was a vector containing one zero, zero, zero, zero right - and we multiplied that by this matrix, then the only row it's going to return would be the first One okay, so so it's really useful to remember that embedding actually just is a matrix product. The only reason it exists. The only reason it exists is because this is an optimization. You know this, let's pipe or to know like okay. This is just a matrix multiply, but I guarantee you that you know this thing is one hard encoded. Therefore you don't have to actually do the matrix multiply. You can just do a directory of that. Okay, so that's literally all an embedding is. Is it is a computational performance thing for a particular kind of matrix multiplier all right, so that looks up that uses user and then we can look up that users movie all right. So here is movie ID movie ID four one, seven, which apparently is indexed number. Fourteen here it is here, so it should have been point. Seven, five point: four: seven! Yes, it is point, seven five point, plus it okay, so we've now got the user embedding and the movie embedding, and rather than doing a dot product of those two okay, which is what we do normally.
Instead, what? If we concatenate the two together into a single vector of length 10 and then feed that into a neural net, okay and so anytime, we've got you know a tensor of import activations or in this case a tensor of. Actually, this is a tensor of output activations. This is coming out of an embedding layer. We can chuck it in a neural net because neural Nets, we now know, can calculate anything - okay, including, hopefully collaborative filtering. So let's try that so here is our embedding net. So this time I have not bothered to create a separate bias, because instead the linear layer in pytorch already has a bias in it all right. So when we go NN Linea right, that's kind of draw this out. So we've got our! U matrix right, and this is the number of users - and this is the number of factors right and we've got our M matrix that so here's our number of movies and here's our again number of factors. Okay and so remember, we look up a single user. We look up a single movie and let's grab them and concatenate them together. Okay, so here's like the user part, here's the movie part and then let's put that through a matrix product right, so that number of rows here is going to have to be the number of users plus the number of movies right. Because that's how long that is and then the number of columns can be anything we want, because we're going to take that so in this case we're going to pick 10. Apparently. So it's pick 10 and then we're going to stick that through a rail you and then stick that through another matrix, which obviously needs to be of size 10. Here and then the number of columns is a size 1 because we want to predict a single rating.
Okay, and so that's our kind of flow chart of what's going on right, it is a standard, I'm called a one, hidden layer, neural net. It depends how you think of it like there's kind of an embedding layer, but because is linear, and this is linear. The two together is really one linear layer right this just a computational convenience, so it's really got one hidden layer because it's got one layer before this nonlinear activation. So, in order to create a linear layer with some number of rows and some number of columns you just go in and on in the machine learning class this week we learnt how to create a linear layer from scratch by creating our own weight matrix and our Own biases, so if you want to check that out, you couldn't do so there right, but it's the same basic technique, we've already seen, so we create our embeddings. We create our two linear layers, that's all the stuff that we need to start with. You know really, if I wanted to make this more general, I would have had another parameter here called like num hidden, you know equals equals 10 and then this would be a parameter, and then you could like more easily play around with different numbers of activations. So when we say like okay in this layer, I'm going to create a layer with this many activations, all I mean assuming it's a fully connected layer is my linear layer has how many columns in its weight matrix that's how many activations it creates all right.
So we grab our users and movies, we put them through our embedding matrix and then we concatenate them together. Okay, so torch cat concatenate them together on the first dimension, so in other words, we concatenate the columns together to create longer rows. Okay, so that's concatenating! On dimension, one drop out we'll come back to her in a moment. We've got that briefly. So then, having done that, we'll put it through that linear layer, we had we'll do our value and you'll notice. That value is again inside our capital F and end up optional right. It's just a function, so remember, activation function are basically things that take one activation in and spit one activation out in this case, taking something that can have negatives or positives and truncate the negatives. To zero, that's what well you does and then here's a sigmoid. So that's that that is now a genuine neural network. I don't know if I get to call it deep. It's only got one hidden layer, but it's definitely a neural network all right, and so we can now construct it. We can put it on the GPU, you can create an optimizer for it and we can fit it now. You'll notice, there's one other thing: I've been passing to fit, which is what loss function? Are we trying to minimize okay? This is the mean squared error loss and again it's inside F, okay, pretty much all the functions are inside it, okay.
So one of the things that you have to pass fit is something saying like: how do you score it's what counts as good or bad, so it should. I mean now that we have a real neural net. Do we have to use the same number of embeddings for users and that's a great question? You don't know absolutely right, you don't and so, like we've got a lot of benefits here right, because if we, you know think about, you know we're grabbing a user embedding or concatenating it with a movie embedding, which maybe is like some different size, but then also, Perhaps we looked up the genre of the movie and, like you know, there's actually a embedding matrix of like number of genres by I don't know three or something and so like. We could then concatenate like a genre embedding and then maybe the timestamp is in here as a continuous number right, and so then that whole thing we can then feed into you know and you're on it all right and then at the end, remember a final non-linearity Was a sigmoid right, so we can now recognize that thing we did where we did sigmoid x max reading vote min reading, + blah blah blah is actually just another nonlinear activation function. Alright remember in our last layer we use generally different kinds of activation functions. So, as we said, we don't need any activation function at all right. We could just do that right, but by not having any nonlinear activation function, we're just making it harder. So that's why we put the sigmoid in there as well.
Okay, so we can then fit it in the usual way and there we go. You know, interestingly, we actually got a better score than we did with our this model, so I'll be interesting to try training. This with stochastic gradient descent with restarts and see if it's actually better, you know, maybe you can play around with the number of hidden layers and the drop out and whatever else and see if you can come up with, you know, get a better answer than point. Seven: six ish, okay, so so general. So this is like if you were going deep into collaborative filtering at your workplace, whatever this wouldn't be a bad way to go. I could like I'd start out with, like oh okay: here's like a flat footed, dataset 30. In first day I get learner, there's you know not much, I can send it. Basically number of factors is about the only thing that I pass in. I can learn for a while. Maybe try a few different approaches and then you're like okay, there's like that's how I go if I use the defaults okay, how do I make it better and then I'd be like dig into the code and seeing like okay? Well, what? If Jeremy actually do here? This is actually what I want you know, and so one of the nice things about the neural net approach is that you know, as unit mentioned, we can have different numbers of embeddings. We can choose how many hidden and we can also choose, drop now right. So so what we're actually doing is we haven't just got real you that we're also going like okay, let's, let's delete a few things at random. Alright, let's drop out.
So in this case we were deleting after the first linear layer, 75 % of them all right and then after the second one in like 75 % of them, so we can add a whole lot of regularization. Yes, so you know this. It kind of feels like the this, this embedding net. You know you could you could change this again, we could like have it so that we can pass into the constructor. Well, if you're gon na make it look as much as possible like what we had before. We could surpass him. Peace, peace equals 0.75. Oh I'm not sure this is the best API, but it's not terrible. Probably what, since we've only got exactly two layers, we could say: p1 equals 0.75 v p2 v, and so then this will be P 1. This will be Peter, you know where we go and like. If you wanted to go further, you could make it look more like our structured data learner. You could actually have a thing this number of hidden, you know, maybe you could make a list, and so then, rather than creating exactly one hidden layer and one output layer, this could be a little loop that creates and hidden miners each one of the size you Want so like this is all stuff you can play with during the hearing the week if you want to - and I feel like if you've got like a much smaller collaborative children data set, you know, maybe you need like more regularization or whatever it's a much bigger One, maybe more layers would help. I don't know you know.
Iii haven't seen much discussion of this kind of neural network approach to collaborative filtering, but I'm not a collaborative filtering expert. So maybe it's maybe it's around, but that'd be
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Interesting thing to try, so the next thing I wanted to do was to talk about the training loop, so what's actually happening inside the training loop, so at the moment we're basically passing off the actual updating of the weights to pytorches optimizer. But what I'm going to do is like understand what that optimizer is, is actually good and we're. Also. I also want to understand what this Momentum's him he's doing. So you'll find we have a spreadsheet called grab disk gradient descent, and it's kind of designed to be read left to right. Sorry right to left worksheet was so the rightmost worksheet. Is some data right and we're going to implement gradient descent in Excel, because obviously everybody wants to do deep learning in it? Selman we've done collaborative filtering in Excel, we've done convolutions in Excel, so now we need SGD in Excel, so we can replace once and for all okay. So let's start by creating some data right and so here's you know, here's some independent. You know I've got one column of X's, you know and one column of wise and these are actually directly linearly related. So this is this is random right and this one here is equal to x, times, 2 plus 30. Ok, so let's try and use Excel to take that data and try and learn those parameters. Okay, that's going to be able. So, let's start with the most basic version of SGD, and so the first thing I'm going to do is I'm going to run a macro.
So you can see what this looks like so I'll hit run and it does five eight bucks under another five: eight bucks, another five, eight bucks, okay, so the first one was pretty terrible. It's hard to see so I'll just delete that first, one get better scaling. Alright, so you can see it actually, it's pretty constantly improving the loss. All right. This is the loss per pot all right. So how do we do that? So, let's reset it. So here is my X's and my y's, and what I do is I start out by assuming some intercept and some slope right. So this is my randomly initialized weights, so I have randomly initialized them both to one. You could pick a different random number if you like, but I promise that I randomly picked the number one twice there you go. It was a random number between one and one. So here is my intercept and slope. I'm just going to copy them over here right. So you can literally see this is just equal see. One here is equals c2 okay, so I'm gon na start with my very first row of data x equals 14 y equals 58 and my goal is to come up after I look at this piece of data. I want to come up with a slightly better intercept and a slightly better slope. Okay, so to do that, I need to first of all, basically figure out which direction is is down. In other words, if I make my intercept a little bit higher or a little bit lower, would it make my error a little bit better or a little bit worse? So, let's start out by calculating the error so to calculate the error.
The first thing we need is a prediction, so the prediction is equal to the interest at plus x times, so that is our zero hidden layer, neural network, okay, and so here is our era. It's equal to our prediction, our actual squared. So we could like play around with this. I don't want my error to be 18-49. I'd like it to be lower. So what if we set the intercepts to one point, one 18-49 goes to 1840. Okay, so a higher intercept would be better. Okay. What about the slope to increase that it goes from 1849 to 1730. Okay, a higher slope would be better as well, not surprising, because we know actually that there should be 30 into so one way to figure that out, you know encode, and this protein is to do literally. What I just did is to add a little bit to the intercept and the slope and see what happens and that's called finding the derivative through finite differencing right and so let's go ahead and do that. So here is the value of my error. If I add 0.01 to my intercept all right, so it's c4 plus 0.01 and then I just put that into my Lydian function and then I subtract, my actual all squared all right and so that causes my arrow to go down a bit. That's our increasing! My is that increasing will see for increasing the intercept. A little bit has caused my arrow to go down. So what's the derivative? Well, the derivative is equal to how much the dependent variable changed by divided by how much the independent variable changed by all right and so there it is right.
Our dependent variable changed by that that right and our independent variable we changed by 0.01. So there is the estimated value of the error dB. So remember when people talking about derivatives right, this is this is all they're doing, is they're saying what's this value, but as we make this number smaller and smaller and smaller and smaller as it as limits to zero, I'm not mad enough to think in terms of Like derivatives and integrals and stuff like that, so whatever I think about this, I always think about you know an actual like plus 0.01 and divided by 0.01, because, like I just find that easier, just like I'd ever think about probability density functions, I always think about Actual probabilities of that toss, a coin something happens three times, so I always think like remember it's. It's totally fair to do this, because a computer is discrete. It's not continuous, like a computer, can't do anything infinitely small anyway right, so it's actually got to be calculating things at some level of precision right and our brains kind of need that as well. So this is like my version of Jeffery Clinton's like to visualize things in more than two dimensions. You just like, say: 12 dimensions really quickly well, visualizing in two dimensions. This is my equivalent. You know to to think about derivatives, just think about division and like, although all the mathematicians say no, you can't do that. You actually can like, if you think of DX dy is being literally.
You know change in X over changing Y, like the division actually like the calculations, do work like all the time. So, okay. So, let's do the same thing now with changing my slope by a little bit and so here's the same thing right and so you can see both of these are negative. Okay, so that's saying if I increase my intercept, my loss goes down.
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
If I increase my slope, my loss goes down right, and so my derivative of my error, with respect to my slope, is, is actually pretty high and that's not surprising because it's actually you know the constant term is just being added. Where else as slope is being multiplied by 40, okay now find that differencing is all very well and good, but there's a big problem with finite difference, seeing in Hyden no spaces and the problem is this right, and this is like you - don't need to learn how To calculate derivatives or integrals, but you need to learn how to think about them spatially, right and so remember. We have some vector very high dimensional vector. It's got like a million items in it right and it's going through some weight, matrix right of size, like 1 million by size, a hundred thousand or whatever, and it's spitting out something of size. I hundred thousand, and so you need to realize, like there isn't like a gradient yeah, but it's like for every one of these things in this vector right, there's a gradient in every direction. You know in every part of the output right, so it actually has not a single gradient number, not even a gradient vector, but a gradient matrix right, and so this this is a lot to calculate right. I would literally have to like add a little bit to this and see what happens to all of these add a little bit to this see what happens to all of these right to fill in one column of this at a time.
So that's going to be horrendously slow, like that. So that's! Why, like, if you're ever thinking like how we can just do this with finite differencing, just remember like okay, we we're dealing in the with these very high dimensional vectors, where you know this. This kind of matrix, calculus, like all the concepts, are identical, but when you actually draw it out like this, you suddenly realize like okay for each number. I could change, there's a whole bunch of numbers that impacts - and I have this whole matrix of things to compute right, and so your gradient calculations can take up a lot of memory and they can take up a lot of time. So we want to find some way to do this more quickly, okay, and it's definitely well worth like spending time kind of studying these ideas of, like you know, the idea of like the gradients like look up things like Jacobian and Hessian they're, the things that you Want to search for just that, unfortunately, people normally write about them with. You know: lots of great letters and bla bla bla right, but there are some. There are some nice. You know intuitive explanations out there and hopefully you can share them on the forum. If you find them, because this is stuff, you really need to really need to understand in here. You know, because you're trying to train something - and it's not working properly and like later on we'll learn how to like look inside pytorch to like actually get the values of the gradients, and you need to know like okay. Well, how would I like what the gradients you know? What would I consider unusual? Like you know, these are the things that turn you into a really awesome. Deep learning practitioner is when you can like debug your problems by like grabbing the gradients and doing histograms of them and like knowing you know that you could like plot that all each layer, my average gradients getting worse or you know, bigger, okay, so the trick to Doing this more quickly is to do it, analytically, rather than through finite differencing, and so analytically is.
Basically, there is a list. You probably all learned it at high school. There is a literally a list of rules that for every mathematical function, there's a like this is the derivative of that function. So you probably remember a few of them, for example x, squared it's alright, and so we actually have here an x squared. So here is our two x right now, the one that I actually want you to know is not any of the individual rules, but I want you to know the chain rule right, which you've got some function of some function of something. Why is this important? I don't know, that's a linear layer, that's a rally right and then we can kind of keep going backwards, map, etc. Right. A neural net is just a function of a function of a function of a function where the innermost is. You know it's basically linear rally. Your linear rally your dot, linear, sigmoid or soft mass all right, and so it's a function of a function of a function and so therefore to calculate the derivative of the weights in your model. The loss of your model with respect to the weights of your model, you're going to need to use the chain rule and specifically, whatever layer. It is that you're up to like. I want to calculate the derivative here and got a need to use. All of these. All of these ones, because that's all that's that's the function, that's being applied right and that's why they call this back propagation because the value of the derivative of that is equal to that derivative.
Now, basically, you can do it like this. You can say: let's call you is this right: let's call that you all right, then it's simply equal to the derivative of that times. Derivative of that right. You just multiply them together, and so that's what back propagation is like it's not that back propagation is a new thing for you to learn. It's not a new algorithm. It is literally take the derivative of every one of your layers and multiply them all together. So like it doesn't deserve a new name right apply. The chain rule to my layers does not deserve a new name, but it gets one because us neural networks folk really need to seem as clever as possible. It's really important that everybody else thinks we are way outside of their capabilities. So the fact that you're here means that we've failed, because you guys somehow think that you're capable right. So remember. It's really important when you talk to other people that you say backpropagation and rectified linear unit, rather than like multiply the layers, gradients or replace negatives with zeros. Okay. So so here we go so here is so I've just gone ahead and grabbed the derivative. Unfortunately, there is no automatic differentiation in Excel yet so I did the alternative, which is to paste the formula into Wolfram Alpha and got back the derivative, so there's the first derivative and there's the second derivative analytically. We only have one layer in this infinite.
Finally, small neural network, so we don't have to worry about the chain rule and we should see that this analytical derivative is pretty close to our estimated derivative from the find out differencing. And indeed it is right, and we should see that these ones are pretty similar. As well, and indeed they are right, and if you're you know back when I implemented my own neural Nets 20 years ago, I you know had to actually calculate the derivatives, and so I always would write like had something that would check the derivatives using finite difference. In and so for those poor people that they'd have to write these things by hand, you'll still see that they have like a finite differencing checkout. So if you ever do have to implement a derivative by hand, please make sure that you have a finite differencing checker. So that you can test it, alright, so there's no derivatives. So we know that if we increase B, then we're going to get a slightly better loss. So, let's increase B by a bit. How much should we increase it by well, we'll increase it by some more for this, so the motor-pod we're going to choose is called a learning rate and so here's our learning rate. So here's one enoch 4. Ok, so our new value is equal to whatever it was before our derivative times, our learning rate. Okay, so we've gone from one to one point or one, and then a we've done the same thing. So it's gone from one to one point, one two. So this is a special kind of mini batch.
It's a mini batch of size, one okay, so we call this online grading. Does it just means mini batch of size one? So then we can go into the next one. Next is 86. Why is 202 right? This is my intercept and slope copied across from the last row. Okay, so here's my new wire prediction: here's my new era here are my derivatives. Here are my new, a and B okay, so we keep doing that for every mini batch of one. Until eventually, we run out at the end of the new pocket, okay and so then, at the end of an epoch, we would grab our intercept and slope and paste them back over here as our new values there we are, and we can now continue again all Right so we're now, starting with pops today's either in the wrong spot. It should be pasted special, transpose values, all right. Okay, so there's a new intercept, there's any slope. Possibly I got that the wrong way around, but anyway you get the idea and then we continue. Okay, so I recorded the world's tiniest macro, which literally just copies the final slope and puts it into the new slope copies. The final intercept put the new intercept and does that five times and after each time it grabs the root, mean squared error and pastes it into the next spare area. And that is attached to this Run button. And so that's going to go ahead and do that. Five times, okay, so that's stochastic, gradient descent and, if so so, to turn this into a CNN all right, you would just replace this error function right and therefore this prediction with the output of that convolutional example spreadsheet, okay, and that then, would be in CNN being Trained with with SGD okay, now the problem is that you'll see when I run this, it's kind of going very slowly right.
We know that we need to get to a slope of two and an intercept of thirty, and you can kind of see it. This rate, it's going to take a very long time right and specifically, it's like it.
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Keeps going the same direction, so it's like come on! Take a hint, that's a good direction, so they come on. Take a hint, that's a good direction. Please keep doing that, but more is called momentum right so on our next spreadsheet, we're going to implement momentum. Okay, so what momentum does is the same thing and what to simplify this spreadsheet? I've removed the finite difference cause okay other than that. This is just the same right. So it's true! What our X is our wise, A's and B's predictions. Our error is now over here: okay and here's, our derivatives, okay, our new calculation. For this particular row, our new calculation here for our new a term just like before is - is equal to whatever a was before. Okay. Now, this time I'm not taking the derivative, but I'm income other number times the loan rate. So what's this other number? Okay, so this other number is equal to the derivative times. What's this k, 1.02 plus 0.98 times the thing just above it? Okay, so this is a linear interpolation between this rows derivative for this mini-batches derivative and whatever direction we went last time right so, in other words, keep going the same direction as you were before. Right then update it, a little bit right and so in our rich. In our Python, just before we had a momentum of 0.9 okay, so you can see what tends to happen is that our negative kind of gets more and more negative right all the way up to like 2,000, where else, with our standard, SGD approach, a derivatives are Kind of all over the place right, sometimes there's 700 something negative, 7 positive. 100. You know so this is basically saying like yeah if you've been going down for quite a while, keep doing that until.
Finally, here it's like okay, that's that seems to be far enough. So that's being less and less and less negative, all right! Mister, we start being positive again, so you can kind of see why it's called momentum. It's like once. You start traveling in a particular direction for a particular weight. You're kind of the wheel start spinning and then once the gradient turns around the other way, it's like Oh slow down. We've got this kind of event, um and then finally turn back around right. So when we do it this way, all right, we can do exactly the same thing right and after five iterations we're at 89, where else before after five iterations we're at 104 right and after a few more, let's go, maybe 15, okay, so get this 102. For us here it's going right, so it's it's! It's a bit better! It's not hips better! You can still see like these numbers - they're, not zipping along right, but it's definitely an improvement and it also gives us something else to tune which is nice like. So if this is kind of a well-behaved error, surface right, in other words like, although it might be bumpy along the way, there's kind of some overall direction, like imagine, you're going down a hill right and there's like bumps - oh alright, so the mobile more momentum. You got going to skipping over the tops right, so we could say like okay, let's increase our beater up to 0.98 right and see if that like allows us to train a little faster and whoa. Look at that suddenly, what's going to okay, so one nice thing about things like momentum, is it's like another parameter that you can choose to try and make your model train better in practice? Basically, everybody does this every like you look at any like image: net winner or whatever they all use momentum, okay, and so back over here when we said here's SGD that basically means use the basic tab of our Excel spreadsheet.
But then momentum equals 0.9 means. Add in put a point nine over here, okay and so that that's kind of your like default starting point. So let's keep going and talk about Adam. So
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Adam is something which I actually was not right earlier on. In this course I said: we've been using Adam by default, we actually haven't, we've actually been. I noticed our we've actually been using SGD with momentum by default, and the reason is that Adam has had much faster as you'll see it's much much faster to learn with, but there's been some problems, which is people who haven't been getting quite as good. Like final answers with Adam as they have with std with momentum, and that's why you'll see like all the you know, image net winning solutions and so forth, and all the academic papers always use SGD with momentum and I'll Adam seems to be a particular problem in Nlp people really haven't got Adam working at all. Well, the good news is this was I built. It looks like this was solved two weeks ago it. Basically it turned out that the way people were dealing with a combination of weight. Decay in Adam had a nasty kind of bargainer, basically, and that's that's kind of carried through to every single library and one of our students, and then Sahara has actually just completed a prototype of adding. Is this new version of Adam has called Adam W into fastai and he's confirmed that he's getting much faster, both the faster performance and also the better accuracy? So hopefully, we'll have this Adam W in faster, ideally before next week, we'll see how we go very very soon so so it is worth telling you about about Adam. So, let's talk about it.
It's actually incredibly simple, but again, you know make sure you make it sound, really complicated when you tell people so that you can so here's the same spreadsheet again right and here's our randomly selected a and B again somehow it's still one, here's a prediction: here's our Derivatives, okay, so now how we count letting on you hey, you could immediately see it's looking pretty hopeful because, even by like row, ten we're like we're seeing the numbers move a lot more right, so this is looking pretty encouraging. So how are we calculating this? It's equal to our previous value, with B minus j h, we're gon na have to find out what that is times our learning rate divided by the square root of LH okay, so I'm gon na have to dig it and see. What's going on, one thing to notice here is that my learning rate is way higher than it used to be, but then we're dividing it by this big number. Okay. So, let's start out by looking and seeing what this day-out thing is: okay, j8 is identical to what we had before. J8 is equal to the linear interpolation of the derivative and the previous direction. Okay, so that was easy, so one part of atom is to use momentum in the way we just defined it. Okay, the second piece was to divide by square root: L 8. What is that square root? L? 8. Okay is another linear interpolation of something and something else, and specifically it's a linear, interpolation of F, 8 squared okay, it's a linear interpolation of the derivative squared, along with the derivative squared last time; okay, so in other words, we've got two pieces of momentum going on Here one is calculating the momentum version of the gradient.
The other is calculating the momentum version of the gradient squared, and we often refer to this idea as a exponentially weighted moving average. In other words, it's basically equal to the average of this one and the last one in the last one in the last one that we're like multiplicatively decreasing the previous ones right because we're multiplying it by 0.9 times what 999. And so you actually see that, for instance, in the faster I code, if you look at fish, we don't just calculate the average loss right, because what I actually want. We certainly don't just report the loss for every mini match because that just bounces around so much so instead I say, average loss is equal to whatever the average loss was last time times: 0.98 plus the loss this time times 0.02 right. So, in other words, the faster you library, the thing that it's actually when you do, like the learning rate, finder or plot loss, it's actually showing you the exponentially weighted moving average of the loss. Okay. So it's like a really handy concept. It appears quite a lot right. The other in handy concept know about is this idea of like you've got two numbers: one of them is multiplied by some value. The other is multiplied by one minus that value. So this is a linear interpolation of two values. You'll see it all the time and for some reason, deep learning people nearly always use the value alpha when they do this.  None of us like read every thing in the equation.
We look at it, we go. Oh linear, interpolation, right - and I said something I was just talking to Rachel about yesterday - is like whether we could start trying to find like a a new way of writing papers where we literally refactor them right, like it'd, be so much better to have written like Linear interpolate bla bla, bla bla bla right, because then you don't have to have that pattern. Recognition right, but until we convince the world to change how they write papers. This is what you have to do. Is you have to look? You know know what to look for right and once you do suddenly the huge page with formulas that at all, like you often notice like, for example, the two things in here like they might be totally identical. But this might be a time T, and this might be at like time, t minus y or something right like it's very often, these big ugly formulas turn out to be really really simple, if only they had ripped out them. Okay. So what are we doing with this gradient squared? So what we were doing with the gradient squared is, we were taking the square root and then we were adjusting the learning rate by dividing the learning rate by that okay, so gradient squared is always positive right and we're taking the exponentially waiting move moving average of A bunch of things that are always positive and then we're taking the square root of that right. So when is this number going to be high, it's going to be particularly high.
If there's like one big, you know if the gradients got a lot of variation. That's! Oh there's a high variance of gradient, then this G squared thing is going to be a really high number for us. If it's like a constant amount right, it's going to be smaller. That cuz, when you add things that are squared the squared slight jump out much bigger for us, if there wasn't, if there wasn't much change, it's not going to be as big. So basically, this number at the bottom here is going to be high. If our Brady -- nt is changing a lot now, what do you want to do if you've got something which is like first negative and then positive and then small and then high right? Well, you probably want to be more careful right. You probably don't want to take a big step, because you can't really trust it right. So when the when the variance of the gradient is high, we're going to divide our learning rate by a big number, we also found learning rate is very similar kind of size. All the time, then, we probably feel pretty good about the step, so we're dividing it by a small amount yeah, and so this is called an adaptive learning rate yeah and, like a lot of people, have this confusion about atom. I've seen it on the forum actually like there's some kind of adaptive learning rate where somehow you like, setting different learning rates for different layers or something it's like. No, not really right.
All we're doing is we're just saying, like this, keep track of the average of the squares of the gradients and use that to adjust the learning rate, so there's still one learning rate. Okay, in this case, it's one okay, but effectively every parameter at every epoch is being kind of like getting a bigger jump if the learning rate, if the gradients been pretty constant for that wait and a smaller jump. Otherwise, okay and that's Adam - that's the entirety of Adam in in Excel right. So there's now no reason at all why you can't train imagenet in Excel, because you've got you've got access to all of the pieces you need, and so let's try this out run. Okay, that's not bad right five and we straight up to twenty nine and two right. So the difference between, like you know, standard SGD and this is is huge and basically that you know the key difference was that it figured out that we need to be. You know moving this number much faster, okay and so, and so it do, and so you can see, we've now got like two different parameters. One is kind of momentum for the gradient piece. The other is the momentum for the gradient squared piece and there I think they're called like, I think, there's just a couple of the beta. I think when you, when you want to change it in PI tortes, is, I think, what beta, which is just a couple of two numbers. You can change Jeremy, so so you set the yeah.
I think I understand this concept of you know one day when a gradient is, it goes up and down then you're not really sure which direction should should go, so you should kind of slow things down. Therefore, you subtract that gradient from the learning rate, so, but how do you implement? How far do you go? I guess maybe I miss something early on you. Do you set a number somewhere, we divide yeah, we divide the learning rate divided by the square root of the moving average gradient squared. So that's where we use it. Oh I'm sorry, can you be a little more sure? So d2 is the learning rate, which is one yeah m27, is our moving average of the squared gradients. So we just go D 2, divided by square root and preserve. That's it. Okay thanks. I have one question yeah, so the new method that you just mentioned, which is in the process of getting implemented in yes, how different is it from here? Okay, let's do that, so to understand Adam W. We have to understand, wait, okay and maybe we'll learn more about that later. Let's see how we go now with great okay, so the idea is that when you have lots and lots of parameters like we do with you know most of the neural Nets. We train. You very often have like more parameters and data points, or you know, like regularization, becomes important and we've learnt how to avoid overfitting by using dropout right, which
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Randomly deletes some activations in the hope, that's going to learn some kind of more resilient set of weights. There's another kind of Ritter ization. We can use called weight, decay or l2 regularization and it's actually comes kind of as a kind of classic statistical technique, and the idea is that we take our loss function right. So we take out like arrow, squared loss function and we add an additional piece to it. Let's add weight decay right now. The additional piece we add is to basically add the square of the weights, so we'd say, plus, B, squared plus a squared okay. That is now wait, 2 K or L tree regularization, and so the idea is that now the the loss function wants to keep the weight small, because increasing the weights makes the loss worse, and so it's only going to increase the weights if the loss improves by More than the amount of that penalty and, in fact to make this weight to get to proper weight decay, we then need some multiplier yeah right. So, if you remember back in our here, we said weight. Decay equals W d5e, neg, 4, okay, so to actually use the same way to K, I would have to multiply by 0.005 all right, so that's actually now the same weight. Okay. So if you have a really high weight decay that it's going to set all the parameters to zero, so it'll never over fit right because it can't set any parameter to anything.
And so, as you gradually decrease the weight decay, a few more weights can actually be used right, but the ones that don't help much it's still going to leave at zero or close to zero right. So that's what that's, what weight decay is is is literally to change the loss function to a D in this sum of squares of weights times some parameter, some hyper parameter. I should say the problem is that if you put that into the loss function as I have here, then it ends up in the moving average of gradients and the moving average of Squared's of gradients for atom right, and so basically we end up when there's a Lot of variation, we end up decreasing the amount of weight decay and, if there's very little variation, we end up increasing the amount of weight decay. So we end up basically saying penalize parameters. You know weights that are really hi unless their gradient varies a lot, which is never what we intended right. That's just not not the plan at all, so the trick with Adam W is we basically remove weight decay from here? So it's not in the last function. It's not in the G, not in the G squared, and we move it so that instead it's it's. It's added directly to the when we update with the learning rate it's out of there instead, so in other words it would be, we would put the weight decay or I should a gradient of the weight decay in here when we calculate the new, a mu V. So it never ends up in our G M G squared.
So that was like a super fast description, which will probably only make sense if you listen to a three or four times on the video and then talk about it on the forum yeah. But if you're interested, let me know - and we can also look at Ann Ann's code - that's implemented. Yes and you know the the idea of using weight decay. Is it's a really helpful regularizer, because it's basically this way that we can kind of stay like you know? Please don't increase any of the weight values unless the you know, improvement in the loss is worth it, and so, generally speaking, pretty much all state of the art models have both dropout and weight decay, and I don't claim to know like how to set each one And how much of H to use to say, like you, it's worth trying both to go back to the idea of embeddings? Is there any way to interpret the final to reduce it? Embeddings, like absolutely we're gon na look at that next week, I've it's super fun! It turns out that you know we'll learn what some of the worst movies of all time. It's Letham, it's that John Travolta Scientology once my battleship earth or something I think that was like the worst movie of all time. According to our beds, to many recommendations for scaling the l2 penalty, or is that kind of based on how how wide the notes are? How many notes about III have no suggestion at all, like I, I kind of look for like papers or cackle competitions or whatever similar and try to set up. Frankly, the same, it seems like in a particular area like computer vision, object, recognition, it's like somewhere between one in neck, four or one in egg five seems to work.
You know actually, in the Adam W paper, the authors point out that, with this new approach it actually becomes like it seems to be much more stable as to what the right way to K amounts are so hopefully now, when we start playing with it, we'll be Able to have some definitive recommendations by the time we get to part two all right. Well, that's nine o'clock! So this week you know practice the thing that you're least familiar with. So if it's like jacobians and Hessians read about those. If it's broadcasting read about those, if it's understanding python ooo read about that, you know try to implement your own custom layers, read the faster higher layers you know and and talk on the forum about anything that you find weird or confusing. Alright see you next week,