fast.ai: Deep Learning Part 1 (v2) (2018)
We complete our work from the previous lesson on tabular/structured, time-series data, and learn about how to avoid overfitting by using dropout regularization. We then introduce natural language processing with recurrent neural networks, and start work on a language model.
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Okay, hi everybody! Welcome back. Let's see you all here, it's been another busy week of deep learning lots of cool things going on and like last week. I would have to highlight a few really interesting articles that some of some of you folks have have written. Vitaliy wrote one of the best articles I've seen for a while, I think, actually talking about differential learning rates and stochastic gradient descent with restarts be sure to check it out. If you can, because what he's done, I feel, like he's done a great job of kind of positioning, it a place that you can get a lot out of it. You know, regardless of your background, but for those who want to go further. He's also got links to like the academic papers. It came from and kind of rests of showing examples of all of all the things he's talking about, and I think it's a it's a particularly nicely done article so good kind of role model for technical communication. One of the things I've liked about you know seeing people post these post these articles during the week is the discussion on the forums have also been like really great. There's been a lot of a lot of people helping out like explaining things. You know, which you know, maybe those parts of the post period where people have said actually that's not quite how it works, and people have learnt new things that way.
People have come up with new ideas as a result as well. These discussions of stochastic gradient descent with restarts and cyclic or learning rates just being a few of them actually Anand sahar has written another great post talking about a similar, similar topic and why it works so well and again: lots of great pictures and references to papers And most importantly, perhaps code showing how it actually works. Mark Hoffman covered the same topic at kind of a nice introductory level. I think really really kind of clear intuition. Many Cantor talk specifically about differential learning rates and why it's interesting and again providing some nice context to people not familiar with transfer learning you're, not going right back to saying like or what is transfer learning. Why is that interesting? And, given that why good differential learning rates be helpful and then one thing I particularly liked about arjen's article was that he talked not just about the technology that we're looking at, but also talked about some of the implications, particularly from a commercial point of view. So thinking about like based on some of the things we've learned about so far, what are some of the implications that that has you know in real life and lots of background lots of pictures and then discussing some of the yeah some of the implications.
So there's been lots of great stuff online and thanks to
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Everybody for all the great work that you've been doing as we talked about last week, if you're kind of vaguely wondering about writing something but you're feeling a bit intimidated about it, because you've never really written a technical post before just jump in you know it's it's! It's it's a really welcoming and encouraging group I think, to to work with so we're going to have a kind of an interesting lesson today, which is we're going to cover a whole lot of different applications. So we've we've spent quite a lot of time on computer vision and today we're going to try if we can to get through three totally different areas: structured learning. So looking at kind of how you look at so we're going to start out looking at structured learning or structured data learning, by which I mean building models on top of things, look more like database tables, so kind of columns of different types of data. There might be financial or geographical or whatever we're going to look at using deep learning for language, natural language processing and we're going to look at using deep learning for recommendation systems. And so we're going to cover these at a very high level. And the focus will be on here's: how to use the software to do it more. Then here is what's going on behind the scenes and then the next three lessons we'll be digging into the details of what's been going on behind the scenes, and also coming back to looking at a lot of the details of computer vision that we've kind of skipped Over so far, so the focus today is really on like how do you actually do these applications and we'll kind of talk briefly about some of the concepts involved before we do? I did want to talk about one key new concept, which is
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Dropout - and you might have seen dropout mentioned a bunch of times already and got there got the impression that this is something important and indeed it is so look at dropout. I'm going to look at the dog breeds current cable competition. That's going on and what I've done is I've gone ahead and I've created a pre train network as per usual, and I've passed in pre compute equals true, and so that's going to pre compute, the activations that come out of the last convolutional layer remember an activation Is just a number it's a number just to remind you. An activation like here is one activation. It's a number, and specifically the activations, are calculated based on some weights, also called parameters that make up kernels or filters, and they get applied to the previous layers activations. But it could well be the inputs or they could themselves be the results of other calculations. Okay, so when we say activation, keep remembering we're talking about a number that's being calculated, so we've pre compute some activations and then what we do is we put on top of that a bunch of additional, initially randomly generated fully connected layers, so we're just going to Do some matrix multiplications on top of these, just like in our Excel worksheet at the very end, we had this matrix that we just did a matrix multiplication, but so what you can actually do is, if you just type the name of your loner object, you can Actually see what's in it, you can see the layers in it.
So when I was previously been skipping over a little bit about, are we add a few layers to the end? These are actually the layers of yet we're going to do batch norm. In the last lesson, so don't worry about that. For now a linear layer simply means a matrix multiply. Okay, so this is a matrix which has a 1024 rows and 512 columns, and so in other words, it's going to take in 1024 activations and spit out 512 activations. Then we have a rail unit which remember is just replace the negatives with 0 we'll skip over the batch norm. We'll come back drop out, then we have a second linear layer that takes those 512 activations from the previous linear layer and puts them through a new matrix multiply: 5 12 by 120. It spits out a new 120 activations and then finally put that through soft mats and for those of you that don't remember, softmax. We looked at that last year last week. It's this idea that we basically just take the the activation. Let's say the dog go e to the power of that and then divide that into the sum of e to the power of all the intermissions. So that was the thing that adds up to one all of them add up to one and each one individually is between 0 and 1. Ok, so that's that's what we added on top and that's the thing when we have pre computed calls. True, that's the thing we trained, so I wanted to talk about what this dropout is and what this key is, because it's a really important thing that we get to choose so a dropout layer with P equals 0.5 literally. Does this we go over to our spreadsheet and let's pick any layer with some activations and let's say: ok, I'm going to apply dropout with a P of 0.5 to con true what that means is I go through and with a 50 % chance. I pick a cell right pick an activation, so I kept like half of them randomly and I delete them.
Okay, that's that's! What dropout is right, so it's so the P equals 0.5 means. What's the probability of deleting that cell all right. So when I delete those cells, if you have a log like look at the output, it doesn't actually change by very much at all just a little bit, particularly because remember it's getting through a Mac spalling layer right. So it's only going to change it at all. If it was actually the maximum in that group of four and furthermore it's just one piece of you know if it's going into a convolution rather than into a max Paul, is just one piece of that that filter. So, interestingly, the idea of like randomly throwing away half of the activations in a layer has a really interesting result, and one important thing to mention is each mini batch. We throw away a different random half of activations earlier, and so what it means is it forces it to not over fit right. In other words, if there's some particular activation, that's really learnt just that exact, that exact dog or that exact cat right then, when that gets dropped out the whole thing now isn't going to work as well. It's not going to recognize that image right, so it has to. In order for this to work, it has to try and find a representation that that actually continues to work even as random half of the activations get thrown away every time all right.
So it's a it's it's, I guess about four years old, now, three or four years old and it's been absolutely critical in making modern, deep learning work and the reason why is it really just about solve the problem of generalization for us before drop out came along? If you try to train a model with lots of parameters and you were overfitting and you already tried all the imitation you could and you already had as much data as you could you, there were some other things you could try, but to a large degree you Were kind of stuck okay, and so then Geoffrey Hinton and his colleagues came up with this. This dropout idea that was loosely inspired by the way the brain works and also loosely inspired by Geoffrey Hinton's experience in bank teller, Hugh's, apparently and yeah. Somehow they came up with this amazing idea of like hey: let's, let's try throwing things away at random, and so, as you could imagine, if your P was like point O one then you're throwing away 1 % of your activations for that layer at random. It's not gon na randomly change things up very much at all, so it's not really going to protect you from overfitting much at all. On the other hand, if your pain was 0.99, then that would be like going through the whole thing and throwing away nearly everything right and that would be very hard for it to overfit.
So that would be great for generalization, but it's also going to kill your accuracy, so this is kind of play off between high p-values generalized well, but will decrease your training accuracy and low p-values will generalize less well. That will give you a less good training accuracy. So for those of you that have been wondering, why is it that particularly early in training, my validation losses better than my training losses, but which seems otherwise, really surprising? Hopefully, some of you have been wondering why that is because on a data set that it never gets to see, you wouldn't expect the losses to ever be that's better and the reason why is because, when we look at the validation set, we turn off dropout right. So, in other words, when you're doing inference, when you're trying to say is this or cat, or is this a dog we certainly don't want to be, including random drop out there right we want to be using the best model. We can okay. So that's why? Early in training in particular, actually see that our validation, accuracy and loss tends to be better if we're using dropout. Okay. So yes, you know you have to do anything to accommodate for the fact that you are throwing away. Some that's a great question, so we don't that pytorch does so pytorch behind the scenes. Does two things: if you say P equals point five. It throws away half of the activations, but it also doubles all the activations that are already there.
So when average, the kind of the average activation doesn't change, which is pretty pretty neat trick so yeah you don't have to worry about it. Basically, it's it's done for you. So if we say so, you can pass in peas. This is the this. Is the p value for all of the added layers to say with fastai? What dropout do you want on each of the layers in these these added layers? It won't change. The dropout in the pre trained network, like the hope, is that that's already been pretty trained with some appropriate level of dropout. We don't change it. Put on these layers that we add you can say how much, and so you can see here as a T's equals 0.5, so my first dropout has 0.5. My second dropout has 0.5. I remember coming to the input of this was the output of the last convolutional layer of pre-trained network and we go over it and we actually throw away half of that before you can start go through our linear layer, throw away the negatives throw away half the Result of that go through another linear layer and then pass it to our softness for minor numerical precision region reasons. It turns out to be better to take the log of the softmax, then softmax directly, and that's why you'll have noticed that when you actually get predictions out of our models, you always have to go npx both the predictions, but again the details as to. Why aren't important? So if we want to try removing dropout, we could go.
Peas, equals zero, all right and you'll see where else before we started with the point: seven six accuracy in the first epoch. Now you could have point eight accuracy in the first debug. Alright, so by not doing drop out, our first teapot worked better, not surprisingly, because we're not throwing anything away but by the third epoch. Here we had eighty four point eight, and here we have eighty four point one. So it started out better and ended up worse. So, even after three epochs, you can already see where master for overfitting right we've got point three loss on the train and point five loss on the validation yep. And so, if you look now, you can see in the resulting model. There's no drop out at all. So if the P is zero, we don't even add it to the model. Another thing to mention is you might have noticed that what we've been doing is we've been adding two linear layers right in our additional layers. You don't have to do that by the way. There's actually a parameter called extra fully connected layers that you can basically pass a list of. How long do you want or how big do you want, each of the additional fully connected layers to be, and so by default? Well, you need to have at least one right, because you need something that takes the output of the convolutional layer, which, in this case is of size thousand twenty-four and turns it into the number of classes you have.
Cats versus dogs would be two dog breeds would be 120 planet satellite, seventeen, whatever that's. You always need one linear layer, at least, and you can't pick how big that is, that's defined by your problem, but you can choose what the other size is or if it happens at all. So if we were to pass in an empty list - and now we're saying don't add any additional mini layers, just the one that we have to have right so here, we've got P - is equals zero extra fully connected layers is empty. This is like the minimum possible kind of top model we can put on top and again like if we do that, you can see above we actually end up with, in this case, a reasonably good result, because we're not training it for very long, and this particular Pre-Trained Network is really well suited to this particular problem yesterday so Jeremy. What kind of piece should we were using by default? So the one that's there by default for the
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
First layer is 0.25 and for the second layer is 0.5. That seems to work pretty well for most things right, so like it's, it's it, you don't necessarily need to change it at all. Basically, if you find it's overfitting just start bumping it up so try. First of all, setting it to 0.5 that'll set them both to 0.5. If it still overfitting a lot, try 0.7, like you, can you can narrow down and like it's, not that many numbers change right and if you're under fitting, then you can try and making it lower. It's unlikely. You would need to make it much lower because, like even in these dogs versus cats situations, you know we don't see they have to make it lower. So it's more likely to be increasing at about 0.6 0.7, but you can fiddle around. I find these the ones that are there by defaults in work pretty well most of the time so one place I actually did increase. This was in the dog breeds one I did set it them both to 0.5, when I used a bigger model, so like ResNet 34 has less parameters, so it doesn't over fit as much. But then, when I started bumping pumping it up to like a resonate 50, which has a lot more parameters and noticed it started overfitting. So then I also increased my drop out. So as you use like bigger models, you'll often need to add more drama. Can you pass it over there, please, you know if we set B to 0.5, roughly what percentage is it 50 %? We say, RP, pasta? Is there a particular way in which you can determine if the data is being all fitted yeah? You can see that the like here, you can see that the training error is a loss, is much lower than the validation list.
You can't tell if it's like to over fitted like zero overfitting is not generally optimal, like the only way to find that out is remember. The only thing you're trying to do is to get this number low right, the validation loss number low. So, in the end, you kind of have to play around with a few different things and see which thing ends up getting the validation loss low, but you kind of get a feel overtime for your particular problem. What does overfitting? What does too much River fitting? Look like great so so that's dropout and we're going to be using that a lot and remember it's there by default service here. Another question: oh so I have two questions. So one is so when it says the dropout rate is 0.5. It does it like. You know I delete each cell with a probability of 0.5 or does it just pick 50 % randomly I mean I know both effectively. Is the 4-month yeah? Okay, okay, a second question is: why does the average activation matter? Well, it matters because the remember, if you look the Excel spreadsheet, that the result of this cell, for example, is equal to these nine multiplied by each of these nine right and add it up. So if we deleted half of these, then that would also cause this number to half, which would cause like everything else after that to change, and so, if you change what it means you know like you, then you're changing something that used to say, like Oh fluffy Ears are fluffy if this is greater than point six now, it's only fluffy if it's greater than point three like we're changing the meaning of everything, so you here is to delete things without changing. Where are you using a linear activation for one of the earlier activations? Why are we using when you yeah why that particular activation, because that's what this set of layers is so we've with the the pre-trained network is or is the convolutional net work and that's pre computed, so we don't see it.
So what that spits out is it's a vector, so the only choice we have is to use linear layers at this point. Okay, can we have different level of dropout by layer? Yes, absolutely how to do that great. So so you can absolutely have different dropout by layer and that's why this is actually called peas, so you could pass in an array here. So if I went 0, comma 0.2 for example, and then extra fully connecting it, I might add - 512 right, then that's going to be 0 drop out before the first of them and point to drop out before the second of them. Yes, requests, and I must admit I don't have a great intuition even after doing this, for a few years for like when should earlier or later layers have different amounts of dropping out. It's still something I kind of play with, and I can't quite find rules of thumb. So if some of you come up with some good rules of thumb, I'd love to hear about them. I think if in doubt you can use the same drop out and every fully connected layer. The other thing you can try is often
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
People only put drop out on the very last linear layer, so there'd be the two things to try so Jeremy. Why do you monitor the log loss? The loss instead of the accuracy going up? Well, because the loss is the only thing that we can see for both the validation set in the training set, so it's nice to be able to compare them. Also, as we learn about later, the loss is the thing that we're actually optimizing. So it's it's kind of a little more. It's a little easier to monitor that and understand what that means. Can you pass it over there so with the drop out we're kind of adding some random noise every iteration right, you know. So that means that we don't do as much learning yeah that's right, so it doesn't seem to impact the learning rate enough thrive ever noticed it - I I would say you're, probably right in theory it might, but not enough that it's ever affected me. Okay. So let's talk about
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
This structured data problem and so to remind you, we were looking at kegels rossmann competition, which is a German chain of supermarkets. I believe - and you can find this in lesson - three Russman and the main data set is the one where we were looking to say at a particular store. How much did they sell?  One is what was the date another was: were they open? Did they have a promotion on? Was it a holiday in that state and was it a holiday? As for school, a state holiday, there wasn't a school holiday yeah, and then we had some more information about stores like what for this store, what kind of stuff did they tend to sell? What kind of store are they how far away the competition and so forth? So, with the data set like this there's really two main kinds of column: there's columns that we think of as categorical. They have a number of levels, so the assortment column is categorical and it has levels such as a B and C. Where else something like competition distance, we will call continuous. It has a number attached to it where differences or ratios, even if that number have some kind of meaning, and so we need to deal with these two things quite differently. Okay, so anybody who's done any machine. Learning of any kind will be familiar with using continuous columns if you've done any linear regression.
For example, you can just like modify them by parameters, for instance categorical columns, we're going to have to think about a little bit more we're not going to go through the data cleaning, we're going to assume that that's a feature, Engineering we're going to assume all that's Been done, and so basically, at the end of that we have a list of columns and the in this case I didn't do any of the thinking around the feature, engineering or dedicating myself. This is all directly from the third-place winners of this competition, and so they came up with all of these different columns that they found useful and so you'll notice. The list here is a list of the things that we're going to treat as categorical variables numbers like year, a month and day, although we could treat them as continuous like they, the different you know, differences between 2000 and 2003 is meaningful. We don't have to right and you'll see shortly how how categorical variables are treated, but basically, if we decide to make something, a categorical variable, what we're telling our neural net down the track is that for every different level of say year, you know 2000. 2001. 2002. You can treat it totally differently where else, if we say it's continuous, its have to come up with some kind of like function, some kind of smooth ish function right and so often, even for things like a year that actually are continuous, but they don't actually have Many distinct levels: it often works better to treat it as categorical, so another good example, day of week, right so like day of week between naught & amp 6, it's a number and it means something motifs between 3 & amp, 5 is two days and has meaning, But if you think about like how word sales in a strawberry buy a day of week, it could well be that, like you know, Saturdays and Sundays, are over here, and Fridays are over here and Wednesdays. Are over here, like each day is going to behave.
Kind of qualitatively differently right so by saying this is the categorical variable as you'll see we're going to let the neural-net do that right. So this thing where we get where we say which are continuous in which a categorical to some extent, this is the modeling decision.  There's no way to treat that directly as a continuous variable. On the other hand, if it starts out as a continuous variable like age or day of week, you get to decide whether you want to treat it as continuous or categorical. Okay, so summarize, if it's categorical and data it's going to have to be categorical in the model, if it's continuous in the data, you get to pick whether to make it continuous or categorical in the model. So in this case again, what I just did, whatever the third-place winners of this competition did, these are the ones that they decided to use as categorical. These were the ones they decided to use as continuous, and you can see that basically, the continuous ones are all of the ones which are actual floating-point numbers like competition. Distance actually has a decimal place to it.
Right and temperature actually has a decimal place to it. So these would be very hard to make categorical because they have many many levels right like if it's like five digits of floating-point, then potentially there will be as many levels as there are, as there are roads and by the way, the word we use to say How many levels are in a category we use the word cardinality right. So if you see me say cardinality example, the cardinality of the day of week variable is 7 because there are 7 different days of the week. Do you have a heuristic for one to have been continuous variables, or do you ever in variables? I don't ever been continuous variables, so yeah. So one thing we could do with like max temperature is group it into nought to 10 10 to 20 20 to 30 and then call that categorical. Interestingly, a paper just came out last week in which a group of researchers found that sometimes bidding can be helpful, but it literally came out in the last week and until that time I haven't seen anything in deep learning. Saying that so I haven't. I haven't looked at it myself until this week. I would have said it's a bad idea now. I have to think differently. I guess maybe it is sometimes so if you're using year as a category, what happens when you run the model of a year? It's never seen so your training will get there yeah.
The short answer is: it will be treated as an unknown category and so pandas, which is the underlying data frame. Thinking we're using with categories as a special category called unknown and if it stays a category it hasn't seen before it gets treated as unknown so for AB deep learning model unknown will just be another category. If our data set training, the data set, doesn't have a category and test has unknown. How will it did you know just paper, this unknown category? It's still predict, it will predict something right like it will just have the value 0 barn scenes and if there's been any unknowns of any kind in the training set, then it off learnt a way to predict unknown. If it hasn't it's going to have some random vector - and so that's a interesting detail around training that we probably want to talk about in this part of the course, but we can certainly talk about on the forum. Okay, so we've got our categorical and continuous variable lists defined. In this case there was eight hundred thousand rows, so eight hundred thousand dates basically by Storz, and so you can now take all of these columns. Look through each one and replace it in the data frame where the version where you say, take it and change its type to category okay, and so that just that just a pandas things. So I'm not going to teach you pandas, there's plenty of books so particularly with McKinney's books.
Book on python for data analysis is great, but hopefully it's intuitive as to what's going on, even if you haven't seen the specific syntax before so we're going to turn that column into a categorical column and then for the continuous variables we're going to make them all. 32-Bit floating-point and for the reason for that is that pipe torch expects everything to be 32-bit, floating-point. Okay, so, like some of these include like 1 0 things like, I can't see them straight away, but anyway, so much yeah like was there. A promo was, was a holiday and so that'll become the floating point: values 1 and 0, for instance. Ok, so I try to do as much of my work as possible on small data sets for when I'm working with images that generally means resizing the images to like 64 by 64 or 128 by 128. We can't do that with structured data, so instead I tend to take a sample. So I randomly pick a few rows, so I start running with a sample and I can use exactly the same thing that we've seen before for getting a validation set. We can use the same way to get some random random row numbers to use in a random sample okay, so this is just a bunch of random numbers and then okay, so that's going to be a size 150,000 rather than 800 40,000, and so my data that Before I go any further, it basically looks like this. You can see I've got some boolean x' here. I've got some integers here of various different scales. Here's my year 2014 and I've got some letters here.
So, even though I said, please call that a pandas category pandas still displays that in the notebook as strings right, it's just stored in internally differently. So then
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
The first day, our library has a special little function, called processed data frame and process data frame takes a data frame, and you tell it what's my dependent variable right and it does a few different things. The first thing is, it's pulled out that dependent variable and puts it into a separate variable, okay and deletes it from the original data frame. So DF now does not have the sales column in where else Y just contains a sales column. Something else that it does is it does scaling, so neural nets really like to have the input data to all be somewhere around zero, with a standard deviation of somewhere around one all right. So we can always take our data and subtract the mean and divide by the standard deviation to make that happen. So that's what do see a littles. True, that's and it actually returns a special object which keeps track of what mean and standard deviation did it use for that normalizing. So you can then do the same thing to the test set later. It also handles missing values, so missing values and categorical variables just become the ID 0 and then all the other categories become 1. 2. 3. 4. 5. 4, that categorical variable for continuous variables. It replaces the missing value with the median and creates a new column. That's a boolean and just says: is this missing or not, and I'm gon na skip over this pretty quickly because we talked about this in detail the machine learning course? Okay. So if you've got any questions about this part, that would be a good place to go.
It's nothing deep learning specific there, so you can see afterwards year 2014, for example, has become year, two okay, because these categorical variables have all been replaced with with contiguous integers starting at zero, and the reason for that is later on we're going to be putting them Into a matrix right - and so we wouldn't want the matrix to be 2014 rows long when it could just be two rows one there. So that's the basic idea there and you'll see that the AC, for example, has been replaced in the same way with one and three okay. So we now have a data frame which does not contain the dependent variable and where everything is a number okay. And so that's that that's where we need to get to to do deep learning and all of the stage about that. As I said, we talked about in detail in the machine learning course nothing deep learning specific about any of it. This is exactly what we throw into our random forests as well. So another thing we talk about a lot in the machine learning core, of course, is validation, sets in this case. We need to predict the next two weeks of sales right. It's not like pick a random set of sales, but we have to pick the next two weeks of sales. That was what the cattle competition folks told us to do, and therefore I'm going to create a validation set, which is the last two weeks of my training, set right to try and make it as similar to the test set as possible, and we just posted actually Rachel wrote this thing last week about creating validation sets.
So if you go too fast at AI, you can check it out, we'll put that in the lesson wiki as well, but it's basically a summary of a recent machine learning lesson that we did. The videos are available for that as well, and this is kind of a written, a written summary of it, okay, so yeah so Rachel, and I spend a lot of time thinking about kind of you know. How do you need to think about validation, sets and training sets and test sets and so forth, and that's all there, but again
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Nothing deep learning specific, so, let's get straight to the deep learning action. Okay, so in this particular competition, as always with any competition or any kind of machine learning project, you really need to make sure you have a strong understanding of your metric. How are you going to be judged here, and in this case you know Carol makes it easy. They tell us how we're going to be judged, and so we're going to be judged on the roots mean squared percentage error right, so we're gon na say like. Oh, you predicted three. It was actually three point three, so you were can sent out and then we're gon na average, all those percents right and remember. I warned you that you are gon na need to make sure you know logarithms really well right, and so in this case from you know, we're basically being saying your prediction divided by the actual. The mean of that right is the thing that we care about, and so we don't have a metric in play. Torch called root mean squared percent error. We could actually easily create it by the way. If you look at the source code, you'll see like it's, you know a line of code, but easiest deal would be to realize that that, if you have that right, then you could replace a with like log of a dash and be with like log of B Dash and then you can replace that whole thing with a subtraction, that's just the rule of loaves right, and so, if you don't know that rule then don't make sure you go look it up, because it's super helpful, but it means in this case all we need To do is to take the log of our data, which I actually did earlier in this notebook, and when you take the log of the data, getting the root mean squared error will actually get you.
There means great percent error for free okay, but then, when we want to like print out our it means percent error, we actually have to go e ^ it again right and then we can actually return the percent difference. So that's all that's going on here. It's again not really deep learning specific at all, so here we finally get to the deep learning alright. So, as per usual, like you'll, see everything we look at today looks exactly the same as everything we've looked at so far, which is first, we create a model data object, something that has a validation, set, training set and optional test set built into it. From that we will get a learner, we will then optionally called learner. Dot LR find real, then called learner, dot, fetch it'll, be all the same parameters and everything that you've seen many times before. Okay, so the difference, though, is obviously we're not going to go image. Classify a data dot from CSV or dot from paths, we need to get some different kind of model data and so for stuff, that is in rows and columns. We use columnar model data, but this will return an object with basically the same API that you're familiar with, and rather than from paths or from CSV. This is from data frame. Okay, so this gets passed a few things. The path here is just used for it to know where, should it store like model files or stuff like that right? This is just basically saying: where do you want to store anything that you saved later? This is the list of the indexes of the rows that we want to put in the validation set we created earlier.
Here's our data frame, okay and then look here's. This is where we did the log right. So I took the the Y that came out of property F, our dependent variable. I logged it and I call that yl all right. So we tell it when we create our model data, we need to tell it that's our dependent variable. Okay. So so far, we've got most of the stuff from the validation set, which is what's our independent variables, how dependent variables and then we have to tell it which things do we want traded as categorical right because remember by this time, everything's a number right. So it could do the whole things it's continuous. It would just be totally meaningless right, so we need to tell it which things do we want to treat as categories, and so here we just pass in that list of names that we used before.  For example, you can set the batch size yeah. So after we do that, we've got a little standard model. Data object, but there's a trained DL attribute. There's a Val DL attribute a trained es attribute of LDS attribute. It's got a length, it's got all the stuff exactly like it did in all of our image. Based data objects. Okay, so now we need to create the the model or create the learner and so to skip ahead a little bit. We're basically going to pass in something that looks pretty familiar.
We're going to be passing thing from our model from our model data create a learner that is suitable for it and will basically be passing in a few other bits of information which will include how much dropout to use at the very start. How many? How many activations
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
To have in each layer how much dropout to use at the later layers, but then there's a couple of extra things that we need to learn about, and specifically it's this thing called embeddings. So this is really the key new concept. We have to learn about all right, so all we're doing basically is we're going to take our let's forget about categorical variables for a moment and just think about the continuous variables for our continuous variables. All we're going to do is we're going to grab them all. Okay, so for our continuous variables, we're basically going to say like okay, here's a big list of all of our continuous variables like the minimum temperature and the maximum temperature and the distance to the nearest competitor and so forth. Right and so here's just a bunch of floating-point numbers, and so basically what the neuron that's going to do is going to take that that 1d array, or or vector or to be very DL like rank one tensor or means the same thing. Okay, so we're going to take our egg one tensor and let's put it through a matrix multiplication. So let's say this has got like I don't know 20 continuous variables and then we can put it through a matrix which must have 20 rows. That's how matrix multiplication works and then we can decide how many columns we want right. So maybe we decided 100 right and so that matrix model captions going to spit out a new length. 100 rank 1 tensor.
Okay, that's that's what that's! What a linear! That's? What a matrix product does and that's the definition of a linear layer, indeed what okay and so then the next thing we do is we can put that through a rail you right, which means we throw away the negatives okay, and now we can put that through Another matrix product, okay, so this is going to have to have a hundred rows by definition and we can have as many columns as we like, and so let's say, maybe this was the last layer. So the next thing we're trying to do is to predict sales. So there's just one value: we're trying to predict for sales, so we could put it through a matrix product that just had one column and that's going to spit out a single number all right. So that's like that's kind of like a one layer neural net. If you like now in practice, you know we wouldn't make it one layer, so we would actually have leg. You know maybe we'd have 50 here, and so then that gives us a 50 long vector and then maybe we then put that into our final 50 by one and that's if it's out a single number and one reason I would have to change that there was To point out, you know rally, you would never put rally you in the last layer. I could never want to throw away the negatives because that the softmax - let's go back to the softness, the soft max needs negatives in it, because it's the negatives that are the things that allow it to create low probabilities. That's minor detail, but it's useful to remember.
Okay, so basically, so, basically, a simple view of a fully connected euro net is something that takes in as an input a rank, one tensor. It's bits: it's through a linear layer, an activation layer, another linear layer, softmax and that's the output, okay, and so we could obviously decide to add more linear layers. We could decide, maybe to add, dropout all right, so these are some of the decisions that we need. We get to make right, but we there's not that much. We can do right, there's not much really crazy architecture stuff to do so when we come back to image models later in the course we're going to learn about all the weird things that go on and like resonates and inception networks, and but in these fully connected Networks, they're, really pretty simple they're, just in dispersed linear layers, that is matrix products and activation functions like value and a soft mix at the edge, and if it's not classification, which actually ours is not classification in this case we're trying to predict sales, there isn't even A soft mix right: we don't want it to be between 0 and 1. Ok, so we can just throw away the last activation altogether. If
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
We have time we can talk about a slight trick. We can do there, but for now we can think of it that way. So that was all assuming that everything was continuous right. But what about categorical right? So we've got like day of week right and we're going to treat it as categorical practice like Saturday, Sunday Monday, that should be 6 ready. Okay, how do we feed that in? Because I want to find a way of getting that in so that we still end up with a wreck. One tends to refloat, and so the trick is this. We create a new little matrix of with seven rows and as many columns as we choose right. So, let's pick four all right, so here's our seven rows and four columns right and basically what we do is, let's add our categorical variables to the end. So let's say the first row was Sunday right. Then what we do is we do a lookup into this matrix. We say: oh here's sunday, we do and look up into here and we grab this row and so this matrix we basically fill with floating-point numbers. So we're going to end up grabbing little subset of for floating-point numbers at Sunday's particular for floating-point numbers, and so that way we convert Sunday into a rank 1 tensor of for floating-point numbers and initially those four numbers are random. All right - and in fact this whole thing - we initially start out random, okay, but then we're going to put that through our neural net right. So we basically then take those four numbers and we remove sunday. Instead, we add our four numbers on here right, so we've turned our categorical thing into a floating-point vector, and so now we can just put that throughout neural net just like before and at the very end we found out the loss and then we can figure out Which direction is down and do gradient descent in that direction, and eventually that will find its way back to this little list of four numbers and it'll, say: okay, those random numbers, weren't very good.
This one needs to go up a bit that one is to go up a bit, that one is to go down a bit that one is to go up a bit and so will actually update our original those four numbers in that match and we'll do this Again and again and again, and so this this matrix will stop looking random and it will start looking more and more like, like the exact four numbers that happen to work best for Sunday. The exact four numbers that happen to work best for Friday and so forth, and so in other words, this matrix, is just another bunch of weights in our neural net. All right, and so matrices of this type are called embedding matrices. So an embedding matrix is something where we start out with an integer between zero and the maximum number of levels of that category. We literally index into a matrix to find a particular row. So if it was, the level was one we take the first row. We grab that road and we append it to all of our continuous variables, and so we now have a new vector of continuous variables and when we can do the same thing, so let's say zip code right. So we could like have an embedding matrix. Let's say there are 5,000 zip codes. It would be 5,000 rows long as wide as we decide. Maybe it's 50 wide and so we'd say: ok, here's 9, 4. 0. 0. 3. That zip code is index number 4. You know matrix, ordered out and we'd find the fourth row regret those 50 numbers and append those on to our big vector and then everything after that is just the same. We just put it through our linear layer, a linear layer, whatever.
What are those 4 numbers represent? That's a great question and we'll learn more about that when we look at collaborative filtering, but now they represent no more or no less than any other parameter in a neural net. You know they're just they're, just parameters that we're learning that happen to end up giving us a good loss. We will discover later that these particular parameters, often, however, are human interpretive. All and quote can quite interesting, but that's a side effect of them. It's not fundamental. They're, just for random numbers for now that we're that we're learning or sets of four random numbers to have a good heuristic for at the dimensionality of embedding matrix. So why four here, I sure do so. What I first of all did was I made a little list of every categorical variable and its cardinality, okay, so they're they allow so there's a hundred and there's a thousand plus different stores, apparently in Rothman's Network. There are eight days of the week. That's because there are seven days of the week, plus one left over for unknown, even if there were no missing values in the original data. I always still set aside one just in case there's a missing or an unknown, or something different in the test set again for years, but there's actually three plus room for an unknown and so forth right.
So what I do my rule of thumb is this: take the cardinality with the variable divide it by two, but don't make it bigger than 50 okay. So these are my embedding matrices, so my store matrix, so there has to have a thousand one hundred and sixteen rows cuz. I need to look up right to find his store number three and then it's been a return back a rank, one tensor of length. Fifty day of week, it's going to look up into which one of the eight and returning the thing of length four. So what you typically build on embedding metrics for each categorical feature: yes yeah! So that's what I've done here. So I've said for see in categorical variables see how many categories there are and then for each of those things create one of these. And then this is called embedding sizes. And then you may have noticed that. That's actually the first thing that we pass to get learner, and so that tells it for every categorical variable. That's the embedding matrix to use for that variable. That is behind you, listen yes, traffic aggression! So, besides our random initialization and there are other ways to actually initialize, embedding yes or no there's two ways: one is random. The other is pre-trained and we'll probably talk about pre-trained more later in the course. But the basic idea, though, is if somebody else at Rossmann had already trained a neural net. Just like you, you would use a pre trained net from imagenet to look at pictures of cats and dogs.
If somebody else is pre-trained a network to predict cheese sales in ruspin, you may as well start with their embedding matrix of stores to predict liquor sales in Rossmann, and this is what happens, for example, at Pinterest and Institute. They both use this technique. Instacart uses it for routing their shoppers, Pinterest uses it for deciding what to display on a web page when you go there and they have embedding matrices of products in instigates case of stores that get shared in the organization. So people don't have to train you once so for the embedding sighs. Why wouldn't you just use like open hot scheme and just well? What is the advantage of doing this they're supposed to just? Do it well good question? So so we could easily, as you point out, have instead of passing in these four numbers record instead of passed in seven numbers, all zeroes, but one of them is one, and that also is a list of floats and that would totally work and that's how, generally Speaking, categorical variables have been used in statistics for many years. It's called dummy variables. The problem is that in that case, the concept of sundae could only ever be associated with a single floating-point number right, and so it basically gets this kind of linear behavior. It says, like sunday is more or less of a single thing: yeah worth noticing directions. It's saying like now.
Sunday is a concept in four dimensional space right, and so what we tend to find happen is that these embedding vectors tend to get these kind of rich semantic concepts. So, for example, if it turns out that weekends kind of have a different behavior you'll tend to see that Saturday and Sunday will have like some particular number higher or more likely. It turns out that certain days of the week are associated with higher sales of certain kinds of goods that you kind of can't go without. I don't know like gas or milk see where else there might be. Other products like like wine, for example, like wine, that tend to be associated with like the days before weekends or holidays right. So there might be kind of a column which is like to what extent is this day of the week kind of associated with people going out? You know so basically yeah by by having this higher dimensionality dektor, rather than just a single number. It gives the deep Learning Network a chance to learn these rich representations, and so this idea of an embedding is actually what's called a distributed representation, it's kind of the fun most fundamental concept of neural networks. This is the idea that a concept in a neural network has a kind of a a high dimensional representation, and often it can be hard to interpret because the idea is like each of these numbers in this vector doesn't even have to have just one meaning.
You know it could mean one thing if this is low and that one's high and something else if that one's high and that one's low, because it's going through this kind of rich nonlinear function right, and so it's this. It's this rich representation that allows it to learn such such such interesting relationships, I'm kind of oh another, question sure I'll speak louder. So are there he's in a meeting, so I get the the fundamental of be like the word vector were to Vic, vector algebra even run on this thing: are the embedding suited suitable for certain types of variables like or are these only suitable, for there are different Categories that that the embeddings are suitable for an embedding is suitable for any categorical variable. Okay, so so the only thing it it can't really work well at all. Four would be something that is too high cardinality, so I'm like, in other words we had like whatever it was six hundred thousand rows. If you had a variable with six hundred thousand levels, that's just not a useful categorical variable. You could packetize it. I guess but yeah in general, like you, can see here that the the third place getters in this competition really decided that everything that was not too high cardinality, they put them all as categorical, very and I think that's a good rule of thumb. You know if you can make a categorical variable you may as well, because that way it can learn this rich distributed representation.
Where else, if you leave it as continuous, you know, the most it can do is to kind of try and find a know. A single functional form that fits it well after question, so you were saying that you are kind of increasing the dimension, but actually in most cases we will use a one holding column which has even a bigger dimension that so in a way you are also reducing, But in the most reach, I think that's very good yeah it like yes, you know you can figure this one hot encoding which actually is high dimensional, but it's not meaningfully high dimensional, because everything set one is easy right. I'm saying that also because even this will reduce the amount of memory and things like this, that you have to write, you're, absolutely right and, and so we may as well go ahead and actually destroyed like what's going on with the matrix algebra. Behind the scenes see this: if this doesn't quite make sense, you can kind of skip over it, but for some people I know this really helps if we started out with something saying this is Sunday right. We could represent this as a one, hot, encoded, vector right and so Sunday, you know maybe was position here, so that would be a 1 and then the rest of zeros, okay and then we've got our embedding matrix right with eight rows and in this case four Columns, one way to think of this actually is a matrix product right, so I said you could think of this as like.
Looking up the number one you know and finding like its index in the array, but if you think about it, that's actually identical to doing a matrix product between a one-pot, encoded vector and the embedding matrix like you're, going to go zero times this row. One times this row zero times this row, and so it's like a one hot, embedding matrix product is identical to during the lookup, and so some people in the bad old days actually implemented embedding matrices by doing a one, hot encoding and then a matrix product. And in fact, a lot of like machine learning methods still kind of do that. But, as you know, that was kind of alluding to it's. That's terribly inefficient. So all of the modern libraries implement this as taking take an integer and do a lookup into an array. But the nice thing about realizing that is actually a matrix product mathematically is. It makes it more obvious how the gradients are going to flow. So when we do stochastic gradient descent, it's we can think of it as just another linear layer. Okay, does it say: that's like somewhat minor detail, but hopefully for some of you it helps. Could you touch on using dates and times this category course and how that affects seasonality? Yeah, absolutely that's. A great question did I cover dates. It all remember no! Okay. So I cover dates in a
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Lot of detail in the machine learning course, but it's worth briefly mentioning here, there's a fastai function called add date. Part which takes a data frame and column in that column. Name needs to be a date. It removes unless you squat drop, equals false. It optionally removes the column from the data frame and replaces it with lots of column, representing all of the useful information about that date like day of week, day of month month of year year, is at the start of the quarter. Is at the end of the quarter, basically everything that pandas gives us, and so that way we end up when we look at our list of features where you can see them here, right, yeah month week, data etc. So these all get created for us by a date pad. So we end up with you know this eight long embedding matrix. So I guess eight rows by four column embedding matrix for day of week and conceptually that allows us or allows our model to create some. Pretty interesting time series models all right like it can, if there's something that has a seven day period cycle, that kind of goes up on Mondays and down on Wednesdays, but only for dairy and only in Berlin. It can totally do that, but it has all the information it needs to do that. So this turns out to be a really fantastic way to deal with time series. So I'm really glad you asked the question. You just need to make sure that that the the cycle indicator in your time series exists as a column.
So if you didn't have a column there called day of week, it would be very, very difficult for the neural network to somehow learn to do like a divide, mod, seven and then somehow look that up in an omitting matrix like it not impossible but really hard. It would use lots of computation, wouldn't do it very well. So an example of the kind of thing that you need to think about might be holidays. For example, you know - or if you are doing something in you know sales of beverages in San Francisco. You probably want a list of like when weather that when is the ball game on at AT & amp T Park, because that's going to impact how many people that are drinking beer in Soma right. So you need to make sure that the kind of the basic indicators or or periodicity x' or whatever there and your data and as long as they are the neuron it's going to learn to use them. So I'm kind of trying to skip over some of the non deep learning parts alright. So the key thing here is that we've got our model data that came from the data frame. We tell it how big to make the embedding matrices. We also have to tailor of the columns in that data frame, how many of those categorical variables or how many of them are continuous variables, so the actual parameter is number of continuous variables, so you can here, you can see we just pass in how many columns Are there how many categorical variables are there? So then, that way, the the neural net knows how to create something that puts the continuous variables over here and the categorical variables over there. The embedding matrix has its own drop out alright.
So this is the dropout to apply to the embedding matrix. This is the number of activations in the first linear player, the number of activations in the second linear layer, the dropout in the first linear player, the drop out for the second linear layer. This bit, we won't worry about for now, and then, finally, is how many outputs do we want to create okay, so this is the output of the last mini layer and obviously it's one, because we want to predict a single number, which is sales okay. So, after that, we now have a learner where we can call LR find and we get the standard looking shape and we can say what amount do we want to use and we can then go ahead and start training using exactly the same API we've seen before. So this is all identical you can pass in, I'm not sure if you've seen this before custom metrics. What this does is, it just says: please print out a number at the end of every epoch. By calling this function. This is a function we defined a little bit earlier, which was the root means bread percentage error, first of all, going either the power of our sales, because our sales were originally logged. So this doesn't change the training at all it just it's just something to print out, so we trained that for a while, and you know, we've got some benefits that the original people that built this don't have. Specifically. We've got things like cyclic, all muscle, learning rate, stochastic, gradient descent with restarts, and so it's actually interesting to have a look and compare, although our validation set isn't identical to the test set, it's very similar, it's a two-week period that is at the end of the Training data, and so our numbers should be similar, and if we look at what we get point, oh nine, seven and compare that to the leaderboard public leaderboard, you can see we're kind of sort of look in the top.
Actually, that's interesting. There is a big difference between the public and private leaderboard. It would have, it would have been right at the top of the private leaderboard, but only in the top thirty or forty on the public leaderboards. So not quite sure, but you can see like we're. Certainly, in the top end of this competition, I actually tried running the third place to get his code and their final result was over a point one. So I actually think that we're trippy compared to private leaderboard, but I'm not sure so anyway, so you can see they're, basically there's a technique for dealing with time series and structured data, and you know, interestingly, the group that that used this technique. They actually wrote a paper about it, that's linked in this notebook when you compare it to the folks that won this competition and came second, they did the other folks did way more feature. Engineering like the winners of this competition, were actually subject matter. Experts in logistics, sales forecasting, and so they had their own code to create lots and lots of features and talking to the folks at Pinterest who built their very similar model for recommendations for Pinterest. They say the same thing, which is that when they switched from gradient boosting machines to deep learning, they did like way way way less feature engineering.
It was a much much simpler model and requires much less maintenance, and so this is like one of the big benefits of using this approach to deep learning. You can get state of the at results, but with a lot less work. Yes, are you using any time series in any of these fits indirectly absolutely using what we just saw? We have a day of week, month of year, all that stuff, our columns and most of them are being treated as categories, so we're building a distributed representation of January we're building a distributed representation of Sunday we're building a distributed representation of Christmas. So we're not using any plastic time series techniques. All we're doing is true, fully connected layers in a neural net, better metrics, that's what exactly exactly yeah, so the embedding matrix is able to deal with this stuff like day of week, periodicity and so forth. In a way, richer way than any standard time series technique, I've ever come across one last question: the matrix in the earlier models we did CNN did not pass it during the fig. We passed it when the data was when we got the data, so we're not passing anything to fit just the learning rate and the number of cycles. In this case, we're passing in metrics is not a printout some extra stuff. There is a difference in the we're calling data get learner.
So, with the imaging approach, we just go learner, dot, trained and pass at the data, but in for these kinds of models, in fact, for a lot of the models, the model that we build depends on the data. In this case, we actually need to know like what embedding matrices do. We have and stuff like that. So in this case it's actually the data object that creates the learner so yeah it is. It is a bit upside down to what we've seen before yeah. So just to summarize - or maybe I'm confused so in this case, what we are doing is that we have some kind of structured data did feature engineering. We got some columnar database or something embedding matrix for the categorical variables, so the continuous we just put them straight feature engineering, yeah, then, to map it to deep learning. I just have to figure out which one I can great question. So yes, exactly if you want to use this on your own data set step, one is list the categorical variable names list. The continuous variable names put it in a data frame. Pandas dataframe step two is to create a list of which row indexes. Do you want, in your validation, set step? Three is to call this line of code using this, except like these exact you can just copy and paste it step. Four is to create your list of how big you want each embedding matrix to be and then step 5 is to call get loner. You can use these exact parameters to start with and if it over fits or under fits, you can fiddle with them and then the final step is to call fit so yeah.
Almost all of this code will be nearly identical. Have a couple of questions. One is: how is data element ation can be used in this case and the second one is why, whatever dropouts doing in here, okay, so data augmentation - I have no idea - I mean that's a really interesting question. I think it's got ta be domain-specific. I've never seen any paper or anybody in industry, doing data augmentation with structured data and deep blow, so I don't think it can be done. I just haven't seen it done. What is dropout doing exactly the same as before. So at each point we have the output of each of these. Linear layers is just a rank. One tensor and so dropout is going to go ahead and say: let's throw away half of the activations, and the very first dropout imbedding drop out literally goes through the embedding matrix and says: let's throw away half the activations, that's it: okay, let's
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Take a break and let's come back at five past eight: okay, thanks everybody! So now we're gon na move into something equally exciting. Actually, before I do, I just been sure that I had a good question during the break, which was what's the downside like like almost no one's using this. Why not and - and basically I think the answer is like, as we discussed before. No one in academia almost is working on this because it's not something that people really publish on and as a result, there haven't been really great examples where people could look at and say: oh here's a technique that works well. So, let's have our company implement it, but perhaps equally importantly, until now, with this fastai library, there hasn't been any way to do it. Conveniently if you wanted to implement one of these models, you had to write all the custom code yourself. Where else now, as we discussed it's, you know sick, it's basically a six step process you know involving about you know not much more than six lines of code. So the reason I mentioned this is to say, like I think there are a lot of big commercial and scientific opportunities to use this to solve problems that previously haven't been solved very well before so, like I'll, be really interested to hear. If some of you try this out, you know maybe on like old cattle competitions you might find like.
Oh, I would have won this if I'd use this technique, that would be interesting or if you've got some data set. You work with at work without some kind of model that you've been doing to the GBM or a random forest. Does this help? You know the thing I I'm still somewhat new to this. I've been doing this for basically, since the start of the year was when I started working on these structured, deep learning models, so I haven't had enough opportunity to know where might it fail? It's worked for nearly everything I've tried it with so far, but yeah. I think this class is the first time that there's going to be like more than half a dozen people fulfilled who actually are working on this. So I think you know, as a group, we're gon na hopefully learn a lot and build some interesting things, and this would be a great thing if you're thinking of writing a post about something or here's, an area that there's a couple of that. There's a poster in staccato about what they did. Pinterest has a an O'reilly, a a video about what they did. That's about it and there's two academic papers both about Carroll competition victories one from Yoshi, Joshua Ben geo and his group. They won a taxi destination forecasting. Competition and then also the one linked for this rossmann competition, so yeah
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
There's some background on that. Alright, so language, natural language processing is the area which is kind of like the most up-and-coming area moaning, it's kind of like two or three years behind computer vision in deep learning. It was kind of like the the second area that deep learning started getting really popular in and you know, computer vision got to the point where it was like clear state of the art for most computer vision, things. Maybe in like 2014, you know and in some things in like 2012 in NLP, we're still at the point where, for a lot of things, deep learning is now the state of the art, but not quite everything, but as you'll see the state of kind of the Software and some of the concepts is much less mature than it is for computer vision, so in general, none of the stuff we talked about after computer vision is going to be as like settled as the computer vision and stuff was so NLP. One of the interesting things is in the last few months, some of the good ideas from computer vision have started to spread into NLP for the first time and we've seen some really big advances. So a lot of the stuff you'll see in NLP is is pretty new, so I'm going to start with a particular kind of NLP problem and one of the things refined in NLP is like. There are particular problems you can solve and they have particular names and so there's a particular kind of problem in NLP called language modeling and language.
Modeling has a very specific definition. That means build a model. We're given a few words of a sentence. Can you predict what the next word is going to be so, if you're, using your mobile phone and you're typing away and you press space and then it says like this is what the next word might be like SwiftKey? Does this like really well and SwiftKey, actually uses deep learning for this? That's that's a language model. Okay, so it has a very specific meaning. When we say language modeling, we mean a model that can predict the next word of a sentence. So let me give you an example: I downloaded about 18 months worth of papers from archive, so for those of you that don't know what archive is the most popular preprint server in this community and various others, and has you know lots of academic papers? And so I grabbed the abstracts and the topics for each and so here's an example. So the category of this particular paper, what computer CSMA is computer science and networking and then the summary let the abstract of the paper they're, seeing the exploitation of mm-wave bands is one of the key enabler for 5g mobile bla bla bla, okay. So here's like an example piece of text from my language model, so I trained a language model on this archived data set that I downloaded and then I built a simple little test, which basically you would pass it. Some like priming text, so you'd say like.
Oh imagine you started reading a document that said category is computer science, networking and the summary is algorithms that and then I said, please write an archive abstract. So it said that if it's networking algorithms that use the same network as a single node, I'm not able to achieve the same performance as a traditional network based routing algorithms. In this paper we propose a novel routing scheme, but okay, so it it's learnt by reading archive papers that somebody who is playing algorithms, that where the word cat CSM ie came before it is going to talk like this and remember it started out not knowing English. At all right, it actually started out with an embedding matrix for every word in English, that was random. Okay, and by reading lots of archive papers, it weren't what kind of words followed others. So then, I tried what, if we said: cat computer science, computer vision, summary algorithms that use the same data to perform image specification are increasingly being used to proves the performance of image classification, algorithms, and this paper. We propose a novel method for image specification using a deeper convolutional neural network parentheses CNN. So you can see like it's kind of like almost the same sentence as back here, but things have just changed into this world of computer vision rather than networking.
So I tried something else which is like okay category computer vision and I created the world's shortest ever abstract that words and then I said title on, and the title of this is going to be on that performance. Object. Learning for image classification in OS is end of string. So that's like end of title. What if it is networking summary algorithms title on the performance of wireless networks as opposed to towards computer vision towards a new approach to image specification networking towards then you approach to the analysis of wireless networks. So, like I find this mind-blowing right, I started out with some random matrices, which had like literally no no pre-trade anything. I fed at 18 months worth of archived articles and it learnt not only how to write English pretty well but also, after you say, something's a convolutional neural network. You should then use parentheses to say what it's called and, furthermore, that the kinds of things people talk could say: create algorithms for in computer vision are performing image. Classification and in networking are achieving the same performance as traditional network based routing algorithms. So, like a language model is, can be like incredibly deep and subtle right and so we're going to try and build that, but actually not because we care about this at all. We're going to build it because we're going to try and create a pre-trained model.
What we're actually going to try and do is take IMDB movie reviews and figure out whether they're, positive or negative. So if you think about it, this is a lot like cats vs. dogs, that's a classification algorithm, but rather than an image, we're going to have the text of a review. So I'd really like to use a pre-trained Network like I would at least my connect to start with a network that knows how to read English right, and so my view was like okay to know how to read English means you should be able to like predict The next word of a sentence, so what if we pre train a language model and then use that pre-trained language model and then just like in computer vision? Stick some new layers on the end and ask it instead of predicting the next word in the sentence. Instead predict whether something is positive or negative. So when I started working on this, this was actually a new idea. Unfortunately, in the last couple of months I've been doing it. You know a few people have actually couple people started publishing this, and so this has moved from being a totally new idea to being a you know, somewhat new idea. So so this idea of creating a language model making that
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
The pre-trained model for a classification model is what we're going to learn to do now, and so the idea is we're really kind of trying to leverage exactly what we learnt in our computer vision work, which is: how do we do fine tuning to create powerful classification Models, yes, you know, so why don't you
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Think that doing just directly, what you want to do doesn't work better well because it doesn't just turns out, it doesn't empirically and the reason it doesn't is a number of things. First of all, as we know, fine-tuning a pre-trained network is really powerful right. So if we can get it to learn some related tasks first, then we can use all that information to try and help it on the second task. The other reason is IMDB movie reviews. You know up to a thousand words long, they're, pretty big, and so after reading a thousand words knowing nothing about how English is structured or even what the concept of a word is or punctuation or whatever at the end of this thousand integers. You know they end up being integers. All you get is a 1 or a 0 positive or negative, and so trying to like learn the entire structure of English and then how it expresses positive and negative sentiments from a single number is just too much to expect. So, by building a language model first, we can try to build a neural network, that kind of understands the English of movie reviews, and then we hope that some of the things that's learnt about are going to be useful in deciding whether something's a positive or a Negative nutrition. That's a great question! You can pass that thanks! Is this similar to
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
The car RNN yeah - this is somewhat similar to our Olympic apathy, so the famous car, as in CH AR AR and in try to predict the next letter, given a number of previous letters, language models generally work at a word level. They don't have to and doing things that a word level turns out to be can be quite a bit more powerful and we're going to focus on word level. Modeling in this course. To what extent are these generated words actually copies of what it found in the in the training data set, or are these completely random things that it actually learned, and how do we know how to distinguish between those two yeah? I mean these are awkward questions. The the words are definitely words, we've seen before the work, because it's not at a character level, so it can only give us the word it seen before the sentences there's a number of kind of rigorous ways of doing it. But I think the easiest is to get a sense of like well. Here are two like different categories, where it's kind of created very similar concepts, but mixing them up in just the right way like it. It would be very hard to to do what we've seen here just by like speeding back things at scene before, but you could of course, actually go back and check. You know. Have you seen that sentence before or like a stream distance? Have you seen a similar sentence before, in this case? Oh and of course, another way to do it is the length, most importantly, when we train the language model as we'll see we'll have a validation set, and so we're trying to predict the next word of something.
That's never seen before, and so, if it's good at doing that, it should be good at generating text. In this case, the purpose, the purpose is not to generate text. That was just a fun example, and so I'm not really gon na study that too much. But you know you're during the week turtley can like you, can totally build your you know Great American Novel generator or whatever. There are actually some tricks to to using language models, to generate text that I'm not using here, they're, pretty simple. We can talk about them on the forum if you like, but my focus is actually on classification, so I think that's the thing which is incredibly powerful. Like text classification, I don't know you're a hedge fund. You want to like read every article as soon as it comes out through writers or Twitter or whatever, and immediately identify things which in the past have caused. You know massive market drops, that's a classification model or you want to recognize all of the customer service queries which tend to be associated with people who who leave your you know who cancel their contracts in the next month's. That's a classification problem so like it's a really powerful kind of thing for data journalism, Activision that activism more promise so forth, right, like I'm, trying to class documents into whether they're part of legal discovery or not part of legal discovery. Okay, so you get the idea.
So in terms of stuff we're importing we're importing a few new things here, one of the bunch of things we're importing is torch text torch text is PI, torches like LP library and so fast. Ai is designed to work hand in hand with torch text as you'll, see and then there's a few text specific sub bits of faster, fastai that we'll be using. So we're going to be working with the IMDB large movie review data set. It's very very well studied in academia. You know lots and lots of people over the years have studied this. Data set fifty thousand reviews highly polarized reviews, either positive or negative. Each one has been classified by sentiment, okay, so we're going to try our first of all. However, to create a language model, so we're going to ignore the sentiment entirely, all right so just like the dogs and cats Cree trainer model to do one thing and then fine tune it to do something else, because this kind of idea in NLP is is so So so new there's basically no models you can download for this, so we're going to have to create their own right. So having downloaded the data, you can use the link here we do the usual stuff of saying the path to training and validation path and, as you can see, it looks pretty pretty traditional compared to a vision. There's a directory of training, there's a directory of tests. We don't actually have separate test and validation in this case and just like in envision.
The training directory has a bunch of files in it in this case, not representing images but representing movie reviews. So we could cat one of those files, and here we learn about the classic zombie garand movie. I have to say, with a name like zombie, gedan and an atom bomb on the front cover. I was expecting a flat-out chop-socky fun coup rent it. If you want to get stoned on a Friday night and laugh with your buddies, don't rent it if you're an uptight, weenie or want a zombie movie or lots of fresh eating, I think I'm going to enjoy zombie getting so alright. So we've learned something today all right, so we can just use standard UNIX stuff to see like how many words are in the data set, so the training set we've got seventeen and a half million words test set. We've got 5.6 million words, so he is. These are, this is IMDB, so IMDB is random people. This is not New York Times listed review. As far as I know, okay, so before we can do anything,
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
With text we have to turn it into a list of tokens. Token is basically like a word right, so we're going to try and turn this eventually into a list of numbers. So the first step is to turn it into a list of words. That's called tokenization in NLP NLP has a huge lot of jargon that will we'll learn over time. One thing: that's a bit tricky, though, when we're doing tokenization is here: I've, I've, tokenized that review and then joined it back up with spaces and you'll. See here that wasn't has become two tokens which makes perfect sense. Right was, is two things right: dot dot dot has become one token right. Where else lots of exclamation marks has become lots of tokens, so, like a good tokenizer, will do a good job of recognizing like pieces of it in your sentence. Each separate piece of punctuation will be separated and each part of a multi-part word will be separated as appropriate. So Spacey is, I think, it's an Australian develop piece of software. Actually, that does lots of you know, P stuff. It's got the best tokenizer, I know, and so past AI is designed to work well with the Spacey tokenizer, as is torch text, so here's an example of tokenization alright. So what we do with torch text is we basically have to start out by creating something called a field, and a field is a definition of how to pre-process some text and so here's an example with the definition of a field.
It says I want to lowercase a text and I want to tokenize it with the function called Spacey tokenize. Okay, so it hasn't done anything yet we're just telling you when we do do something. This is what to do and so that we're going to store that description of what to do in a thing called capital text, and so this is this is none of this, but this is not fastai specific at all. This is part of torch text. You can go to the torch text. Website read the docs, there's not lots of Doc's. Yet this is all very, very new, so probably the best information you'll find about it is in this lesson, but there's some more information on this site all right. So what we can now do is go ahead and create the usual fastai model data object, okay and so to create the model data object. We have to provide a few bits of information, but you have to say: what's the training set so the path to the text, files, the validation set and the test set in this case. Just to keep things simple, I don't have a separate validation and test set. So I'm going to pass in the validation set for both of those two things right so now we can create our model data object as per usual. The first thing you give it as a path. The second thing we give it is the torch text field, definition of how to pre-process that text. The third thing we give it is the dictionary or the list of all of the files. We have trained validation tests.
As per usual, we can pass in a batch size and then we've got a special special couple of extra things here. One is very commonly used in NLP minimum frequency. What this says is in a moment we're going to be replacing every one of these words with an integer which basically will be a unique index for every word, and this basically says if there are any words that occur less than 10 times just call it unknown. Right, don't think of it as a word, but we'll see that indeed more detail in a moment. We're going to see this in more detail as well. Be PTT stands for back prop through time, and this is where we define how long a sentence will we stick on the GPU at once, so we're going to break them up in this case we're going to break them up into sentences of 70 tokens or less On the whole, so we're going to see all this in a moment:
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Okay, so after building our model data object right, what it actually does is it's going to fill this text field with an additional attribute called vocab, and this is a really important and LP concept. I'm sorry there's so many NLP concepts we just have to throw at you kind of quickly, but we'll see them a few times right. The vocab is the vocabulary, and the vocabulary in NLP has a very specific meaning. It is what is the list of unique words that appear in this text, so every one of them is going to get a unique in this. So let's take a look right here. Is text vocab dot I to s this dancer. This is all torch text, not faster hide text, upper cap, dot, int 2 string Maps, the integer 0 to unknown the integer 1, the padding unit 2 to desert, then in comma dot and of 2 and so forth right. So this is the first 12 elements of the array of the vocab from the IMDB movie review and it's been sorted by frequency and except for the first two special ones. So, for example, we can then go backwards. S2I string to int here is the it's in position: 0, 1, 2, so stream to end the is 2. So the vocab lets us take a word and map it to an integer or take an integer and that a tour word right, and so that means that we can then take the first 12 tokens, for example, of our text and turn them into twelve inch. So, for example, here is of the agency 7 2, and here you can see 7/2 right, so we're going to be working in this form.
Did you have a question deputy plus that back there you know? Is it a common tyranny stemming or limit izing? Not really? No generally tokenization is is what we want like with a language model. We, you know to keep it as general as possible. We want to know what's coming next and so like whether its future tense or past tense or plural or seem to learn like we. Don't really know which things are going to be interesting in which ant, so it seems that it's generally best to kind of leave it alone as much as possible, be the short answer. You know. Having said that, as I say, this is all pretty new, so if there are some particular areas that some researcher maybe is already discovered that some other kinds of pre-processing you're helpful, you know I wouldn't be surprised not to know about it, so we Abdullah, we know You don't natural language is in context. Important context is very important, so individual words, no, no we're not looking worth this. Is this look. This is, I just don't get some of the big premises of this like they're there in order yeah. So, just because we replaced I with the number 12, these are still in that order. Yeah, there is a different way of dealing with natural language called a bag of words and bag of words. You through throw away the order in the context and in the machine learning course we'll be learning about working with bag of words, representation, z'. But my belief is that they are no longer useful or in the verge of becoming no longer useful, we're starting to learn.
How to use deep learning to use context properly now, but it's kind of for the first time it's really like only in the last few months right so I mentioned that we've got two numbers batch size and B PTT back crop through time, so this is kind Of subtle, so we've got some big long piece of text. Okay, so we've got some big long piece of text. You know here's our sentence, it's a bunch of words right and actually what happens in a language model is even though we have lots of movie reviews. They actually all get concatenated together into one big block of text right, so it's basically predict the next word in this huge long thing, which is all of the IMDB movie reviews concatenated together. So this thing is you know what do we say? It was like tens of millions of words long and so what we do is we split it up into batches first right, so these, like aerial, spits into batches right, and so, if we said we want a batch size of 64, we actually break the whatever. It was sixty million words into 64 sections right and then we take each one of the 64 sections and we move it like underneath the previous one. I didn't do a great job of that all right move it underneath. So we end up with a matrix, which is you sixty-four? Actually, I think we've moved them across twice, so it's actually, I think, just transpose it. We end up with the matrix. It's like 64 columns wide and the length. Let's say the original was 64 million right.
Then the length is like 10 million long right, so each of these represents one sixty-fourth with our entire IMDB. Refused it all right, and so that's our starting point. So then, what we do is we then grab a little chunk of this at a time and those chunk lengths are approximately equal to be PTT, which I think we had equal to 70. So he basically grab a little 70 long section and that's the first thing. We check into our GPU that's a batch right, so a batch is always of length of width, 64 or batch size and each bit is a sequence of length up to 70. So let me show you all right so here, if I go, take my train data loader. I know if you folks have tried playing with this yet, but you can take any data, loader wrap it with inner turn it into an iterator and then call next on it to grab a batch of data. Just as if you were a neural net, you get exactly what the neuron that gets and you can see here. We get back a 75 by 64 sensor right. So it's 64 wide right - and I said it's approximately 70 high and but not exactly and that's actually kind of interesting, a really neat trick that torch text does. Is they randomly change the backprop through time number every time, so each epoch it's getting slightly different bits of text. This is kind of like in computer vision. We randomly shuffle the images we can't randomly shuffle the words right because we needed to be in the right order, so instead we randomly move their breakpoints a little bit okay, so this is the equivalent. So in other words, this this here is of length 75 right. There's a there's, an ellipsis in the middle, and that represents the first 75 words of the first review right.
Where else this 75 here represents the first 75 words of this of the second of the 64 segments. Let's it have to go in, like 10 million words to find that one right and so here's the first 75 words of the last of those 64 segments. Okay, and so then, what we have down here is the next sequence right so 51, there's 51. 6. 1. 5 there's 6. 1. 5. 25 is 25 right, and in this case it actually is of the same size. It's also 75 plus 64, but for minor technical reasons, it's being flattened out into a single vector, but basically it's exactly the same at this matrix. But it's just moved down by one because we're trying to predict the next word right so that all happens for us right. If we ask for - and this is the first date I know, if you ask for a language model data object, then it's going to create these batches of batch size width by B, PTT height bits of our language corpus, along with the same thing shuffled along by One word right, and so we're always going to try and predict the next word. Yes, so why don't you, instead of just arbitrarily choosing 64? Why don't you choose like, like 64, is a large number, maybe like stood by sentences and make it a large number and then padded with zero or something?  Basically, wouldn't that make more sense, not really because remember we're using columns right, so each of our columns is of length about 10 million right. So, although it's true that those columns aren't always exactly finishing on a full stop, there's so damn long, we don't care because they're, like 10 million won and we're trying to also line, contains multiple cents incentive column contains more costumes and sorry yeah it's of length about 10 million - and it contains many many many many many sentences because remember the first thing we did was take all thing and split it into 64 groups - okay, great so um.
I found this. You know pertaining to this question this thing about like. What's in this language model, matrix a little mind-bending for quite a while, so don't worry if it takes a while, and you have to ask a thousand questions on the forum. That's fine right, but go back and listen to what I just said in this lecture again. Go back to that bit where I showed you splitting it up to 64 and moving them around and try it with some sentences in Excel or something and see if you can do a better job of explaining it than I did. Okay, because this is like how torch text works and then what fastai adds on is this idea of, like kind of how to build a a language model out of it? Well, they'll? Actually, a lot of that stolen from torch text as well like there's some times where torch text starts and fastai ends is, or vice versa, trees a little saddle. They really work closely together. Okay, so now that we have a model data object that can feed us batches, we can go ahead and create a model right, and so, in this case we're going to create an embedding matrix and our vocab. We can see how big a vocab was. Let's have a look back here, so we can see here in the model data object. There are four thousand six hundred and two kind of pieces that we're going to go through. That's basically equal to the number of the total length of everything divided by batch size times B PTT and this one I wanted to show you NT. I've got the definition up here. Number of unique tokens.
Nt is the number of tokens. That's the size of our vocab, so we've got three thirty, four thousand nine hundred and forty five unique words and notice the unique words that had to appear at least ten times. Okay, because otherwise they've been replaced with the length of the data set, is one because, as far as the language model is concerned, there's only one thing, which is the whole corpus all right and then that thing has I hear it is twenty point: six million words. You know right so those thirty four thousand hundred and forty five things are used to create an embedding matrix of number of rows is equal to thirty, four, nine, four, five right, and so the first one represents UNK. The second one represents pad the third one was dot. The fourth one was comma, with one under sketching, was there and so forth right, and so each one of these gets an embedding vector. So this is literally identical to what we did before the brick right. This is a categorical variable, it's just a very high cardinality, categorical variable and furthermore, it's the only variable right. This is pretty standard in NLP. You have a variable, which is a word right. You have a single categorical variable single column. Basically, and it's it's of thirty. Four thousand nine hundred forty five cardinality categorical variable and so we're going to create an embedding matrix for it.
So M size is the size of the omitting vector 200. Okay. So that's going to be length 200, a lot bigger than our previous embedding vectors, not surprising, because a word has a lot more nuance to it than the concept of Sunday right or Russ means Berlin's door or whatever right. So it's generally an embedding size. For a word will be somewhere between about 50 and about 600 okay, so I've kind of done some in the middle. We then have to say, as per usual, how many activations do you want in your layers, so we're going to use 500 and then how many layers do you want in your neural net, we're going to use three okay? This is a minor technical detail. It turns out that we're going to learn later about the atom optimizer that basically the defaults for it. Don't work very well with these kinds of models, so you just have to change some of these. You know basically any time you're doing NLP. You should probably include this mine because it works pretty well. So having done that, we can now again take our model data object and grab a model out of it, and we can pass in a few different things. What optimization function do we want? How big an embedding do we want how many hidden activate how many activations number of Hitler, how many layers and how much drop out of many different kinds? So this language model, we're going to use, is a very recent development called AWD LS TM by Steven marady who's, a NLP research based in San Francisco, and his main contribution really was to show like how to put drop out all over the place in in these Nlp models, so we're not going to worry now we'll do this in the last lecture is worrying about like what all that like.
What is the architecture, and what are all these dropouts for now just know is the same as per usual. If you try to build an NLP model and draw underfitting and decrease all of these dropouts, if you're overfitting then increase all of these dropouts in roughly this ratio? Okay, that's that's my rule of thumb and again this is such a recent paper. Nobody else is working on this model anyway, so there's not a lot of guidance, but I found this. These ratios worked well, it's what Stephens been using as well there's another kind of way we can avoid overfitting that we'll talk about in the last class again for now, this one actually works totally reliably. So all of your NLP models probably want this particular line of code, and then this point we're going to talk about at the end last lecture as well. You can always improve this.   Oh too, big right with gradient clipping.
It kind of goes this far and it's like, oh, my goodness, I'm going too far I'll stop and that's basically what gradient flipping does so anyway.  The details, don't matter too much right now. You can just deal these and then we can go ahead and call fit with exactly the same parameters as usual, so Jeremy um there are all this other work. Embedding things like like worked vague and glow. So I have two questions about that. One is how are those different from these and the second question: why don't you initialize them with one of those yeah? So so, basically, that's a great question. So basically, people have pre trained these embedding matrices before to do various other tasks. They're not called pre-trained models. They're, just a pre trained, embedding matrix and you can download them and as unit says they have names like word to Veck and love and they're, literally just a matrix, there's, no reason we couldn't download them really it's just like kind of. I found that building a whole pre-trained model in this way didn't seem to benefit much if at all, from using pre trained word vectors, where else using a whole pre-trained language model made a much bigger difference.  It made a big splash when it came out.
I mean I'm finding this technique of pre-trained language models seems much more powerful basically, but I think we can combine both to make them a little better still what what is the model that you have used like? How can I know that architecture of the model so we'll be learning about the model architecture? In the last lesson, and for now it's a recurrent neural network using something called an LS TN long, short-term memory? Okay. So so, if they had lots of details that we're skipping over - but you know you can do all this without any of those details, we go ahead and fit the model. I found that this language model took quite a while to fit so I kind of like ran it for a while noticed it was still under fitting safer. It was up to ran it. A bit more longer cycle length saved it again. It still was kind of under fitting you know, run it again and kind of finally got to the point where it's like kind of honestly, I kind of ran out of patience, so I just like saved it at that point and I did the same kind of Tests that we looked at before so I was like - oh, it wasn't quite expecting, but I realized it anyway, the best and the most like. Okay, let's see how that goes, the best performance with one movie were, I say: okay, it looks like the language models working pretty well, so I've pre-trained a language model, and so now I want to use it fine tune it to do. Classification sentiment classification now, obviously, if I'm gon na use a pre trained model, I need to use exactly the same vocab but the the word the still needs to map for the number two so that I can look up the vector that right.
So that's why I first of all load back up my my field object the thing with the vocab in right now, in this case, if I ran it straight afterwards, this is unnecessary, it's already in memory, but this means I can come back to this later right. In a new session, basically, I can then go ahead and say: okay, I've never got one more field right. In addition to my field, which represents the reviews, I've also got a field which represents the label. Okay and the details are too important here now this time I need to not treat the whole thing as one big piece of text, but every review is separate because each one has a different sentiment attached to it, but it so happens that torch text already has A data set that does that for IMDB, so I just used IMDB built into torch text. So basically, once we've done all that we end up with something where we can like grab for a particular example, or you can grab its label positive and here's. Some of the text - this is another great Tom, Berenger movie, all right, so this is all not nothing faster. I specific here we'll come back to it in the last lecture, but torch text Docs can help understand. What's going on, all you need to know is that once you've used this special tox torch text, thing called splits to grab a Spitz object. You can passed it straight into faster, a text data from splits and that basically converts a torch text object into a fastai object.
We can train on so as soon as you've done that you can just go ahead and say get model right and that gets us our learner and then we can load into it. The pre trained model, the language model right, and so we can now take that pre-trained language model and use the stuff that we're kind of familiar with right. So we can make sure all that you know all its at the last layers frozen training a bit unfreeze. It train it a bit and the nice thing is once you've got a pre trained language model. It actually trains super fast. You can see here it's like a couple of minutes for epoch and it only took me to get my is my best one here and he took me like 10 a box, so it's like 20 minutes to train this bit. It's really fast and I ended up with 94.5 %, so how gone is 94.5 %? Well, it so happens that actually one of Steven verities colleagues James Bradbury, recently created a paper looking at the state at like they tried to create a new state of the art for a bunch of NLP things, and one of the things that looked at was IMDB And they actually have here a list of the current world's best for IMDB and even with stuff that is highly specialized for sentiment analysis. The best anybody had previously come up with 94.1, so in other words, this technique getting 94.5, it's literally better than anybody has created in the world before as far as we know, or as far as James Bradbury knows so so when I say like, there are big Opportunities to use this I mean like this is a technique that nobody else currently has access to which you know you could, like you know, whatever iBM has in what CERN or whatever any big company has.
You know that they're advertising, unless they have some secret sauce, that they're not publishing, which they don't right, because people get you know if they have a better thing, they publish it. Then you now have access to a better text. Classification method, then, has ever existed before. So I really hope that you know you can try this out and see how you go. There may be some things it works really well on and others that it doesn't work as well, and I don't know I think, this kind of sweet spot here that we had about 25,000. You know short to medium size documents if you don't have at least that much text, it may be hard to train a different language model. But, having said that, there's a lot more. We do here right and we won't be able to do it in part. 1 of this course, but in part 2, that, for example, we could start like training language models that look at like you know, lots and lots of medical journals, and then we could like make a downloadable medical language model that then anybody could use to like fine Tune on like a prostate cancer, subset of medical literature, for instance like there's so much, we could do it's kind of exciting, and then you know to the next point. We could also combine this with like pre-trained word vectors, it's like even without trying that hard.
Like you know, we even without use like we could have pre-trained a Wikipedia, say corpus language model and then fine-tuned it into a IMDB language model and then fine tune that into an IBM. Imdb sentiment analysis model and we would have got something better than this. So like this, I really think this is the tip of the iceberg, and I was talking there's a really fantastic researcher called Sebastian Reuter, who is basically the only NLP researcher. I know who's been really really writing a lot about pre-training and fine tuning and transfer learning and NLP, and I was asking him like: why isn't this happening more and his view was it's because there isn't the software to make it easy? You know so I'm actually going to share this lecture with with him tomorrow, because you know it feels like there's. You know,
The rest of the video covers the ins and outs of the notebook ‘lesson4-imdb’, don’t forget to use ‘J’ and ‘L’ for 10 sec backward/forward on YouTube videos.
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Hopefully, gon na be a lot of stuff coming out now that we're making it really easy to do this. Ok we're kind of out of time. So what I'll do is I'll quickly look at collaborative filtering introduction and then we'll finish it next time, but collaborative filtering. There's very very little new to learn. We've basically learned everything. We're gon na need, so collaborative filtering will will cover this quite quickly next week and then we're going to do a really deep dive into collaborative filtering next week, where we're going to learn about like we're. Actually, going to from scratch, learn how to do mr. plastic gradient descent, how to create loss functions, how they work exactly and then we'll grow from there and will gradually build back up to really deeply understand. What's going on in the structured models and then what's going on in confidence and then finally, what's going on in recurrent neural networks and hopefully we'll be able to build them all from scratch? Ok, so this is kind of a gon na be really important. This movie lens data set because we've got a user to learn a lot of like really foundational theory and kind of math behind it, so the movie lens data set.  It says user number one watched movie number 31 and they gave it a rating of two and a half at this particular time, and then they watched movie 102 nine and they gave it a rating of three and they watched reading one one's really one one.
Seven two and they gave it a rating before okay and so forth. Okay, so this is the ratings table. This is really the only one that matters and our goal will be for some use that we haven't seen before so for some user movie combination we haven't seen before we have to predict if they'll like it right, and so this is how recommendation systems are built. This is how like Amazon, besides what books, to recommend how Netflix decides, what movies to recommend and so forth, to make it more interesting, we'll also actually download a list of movies, so each movie we're actually gon na, have the title, and so for that question earlier About like, what's actually going to be in these embedding matrices, how do we interpret them? We're actually going to be able to look and see how that's working. So, basically, this is kind of like what we're creating. This is kind of crosstab of users by movies, alright and so feel free to look ahead during the week. You'll see basically, as per usual collaborative data set from CSP model data. Docket learner learn it and we're done and don't be surprised to hear when we then take that and we can kick the benchmarks, it seems to be better than the benchmarks where you look at so that'll, basically be it and then next week we'll have a deep Dive and we'll see how to actually build this from scratch.
Alright see you next week, thank you. [ Applause, ],