fast.ai: Deep Learning Part 1 (v2) (2018)
You will learn more about image classification, covering several core deep learning concepts that are necessary to get good performance: what a learning rate is and how to choose a good one, how to vary your learning rate over time, how to improve your model with data augmentation (including test-time augmentation). We also share practical tips (such as training on smaller images), an 8-step process to train a world-class image classifier, and more information on your hardware setup (including crestle, paperspace, and AWS as options).
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Okay, so welcome back to deep learning lesson. 2. Last week we got to the point where we had successfully trained a pretty accurate image classifier, and so just to remind you about how we did that, can you guys see? Okay, I think the actually we can turn the phone once all right. Can you guys all see the screen? Okay, we can to adjust these ones. Can we some pictures all into darkness? But if that works then, is that okay, that's better, isn't it yeah? Can I dream the other two and maybe that one as well? Oh, but that one? Oh that's great! Sorry! I don't know your renders. Oh okay, great! That's better! Isn't it me so just to remind you the way that we built this image classifier was we used a small amount of code, basically three lines of code and these three lines of code pointed at a particular path which already had some data in it, and so The key thing for this to know how to train this model was that this path, which was data dogs, cats, and had to have a particular structure, which is that it had a train, folder and a valid folder and in each of those trained and valid folders. There was a cats folder in the dogs folder and if the cats on the docs folders was a bunch of images of cats and votes, but this is like a pretty standard. It's one of two main structures that are used to say here is the data that I want you to train an image model from so I know some of you during the week went away and tried different data sets where you had folders with different sets of Images and in credit, your own image, classifiers and generally, that seems to be working pretty well from what I can see on the forums so to make it clear at this point.
This is everything you need to get started. So if you create your own folders with different sets of images, you know a few hundred or a few thousand at each folder and run the same three lines of code. That will give you an image, classifier and you'll. Be able to see this third column tells you how accurate is so. We looked at some kind of simple visualizations to see like what was it uncertain about what was it wrong about and so forth, and that's always a really good idea, and then we learned about the one key number you have to pick. So this is this number here is the one key number is 0.01, and this is called the learning rate, and so I wanted to go over this again and we'll learn about the theory behind what this is during the rest of the course in quite a lot Of detail, and for now I just wanted to talk about the practice. Yes, you know they cannot see you in the medium turnout. I just turned it around. You tell us about the other three numbers being bad. We did these three here we're going to talk about the other other ones shortly. So the main one we're going to look at for now is is the last column, which is the accuracy. The first column, as you can see, is the epoch number. So this tells us how many times has it been through the entire dataset trying to learn a better classifier and in the next two columns is, what's called the loss which we'll be learning about either later today or next week.
The first point is the loss on the training set. These are the images that we're looking at in order to try to make a better classifier, and the second is the loss of the validation set. These are the images that we're not looking at and we're training, but we're just sitting on the side to see how accurate we are so we'll learn about littering loss in accuracy later. Okay, so so we've got the epoch number. The training loss is the second column. The validation loss is the third column, and the accuracy is the fourth column you, okay, so the basic idea of the loading
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Rate, so the basic idea of the learning rate is it's the thing that's going to decide: how quickly do we zoom do we kind of hone in on the solution, and so I find that a good way to think about this is to think about like well. What if we were trying to fit to a function that looks something like this right, we're trying to say: okay, where's, where abouts is the minimum point. This is basically what we do when we do deep learning. Is we try to find the minimum point of a function now? Our function happens to have millions or hundreds of millions of parameters, but it works the same basic way, and so when we look at it, you know we can immediately see that the lowest point is here. But how would you do that? If you are a computer algorithm and what we do is we we start out at some point at random. So you pick say here and we have a look and we say: okay, what's the what's the loss or the error at this point, and we say: what's the gradient, in other words, which way is up and which way is down, and it tells us that down Is going to be in that direction, and it also tells us how fast is it going down which is at this point is going down pretty quickly, and so then we take a step in the direction that's down and the distance we travel is going to be Proportional to the gradient sort of, unfortunately, how steep it is.
The idea is, if it's deeper, then we're probably further away. That's the general idea right and so specifically what we do is we take the gradient, which is how steep is it at this point and we multiply it by some number and that number is called the learning rate okay. So if we pick a number that is very small, then we're guaranteed that we're going to go a little bit closer and a little bit closer and a little bit closer each time right. But it's going to take us a very long time to eventually get to the bottom. If we dig a number, that's very big. We could actually step too far could go in the right direction, but we could step all the way over to here right, as result of which we end up further away than we started, and we could oscillate and get worse and worse. So if you start training a neural net and you find that your accuracy or your loss is like spitting off into infinity almost certainly your learning rates too high, so in a sense learning rate too low is, is a better problem to have because you're going to Have to wait a long time, but wouldn't it be nice if there was a way to figure out like what's the best learning rate, something where you could kind of go quickly, go like Bom, Bom, Bom right, and so that's why we use this thing. Called a learning rate finder and what the learning rate finder does is it tries each each time it looks at another.
Remember the mini-batch, how many batches a few images that we look at each time so that we're using the parallel processing power of the GPU effectively. We look generally at around 64 128 images at a time for each mini batch, which is labeled here. As an iteration, we gradually increase the learning rate, multiplicatively increase the learning rate. We started really really tiny learning rates to make sure that we don't start at something too high and we gradually increase it. And so the idea is that eventually, the learning rate will be so big that the loss will start getting worse. And so what we're going to do, then, is we're a look at the plot of learning rate against loss right. So when the learning rates tiny it increases slowly, then it's that's. Where increase a bit faster and then eventually it starts not increasing as quickly and in fact it starts getting worse right so clearly here and make sure you're. You want to be familiar with this scientific notation, okay, so ten to the negative one is 0.1 10 to 50 or is 1 10 to the negative 2 is 0.001, and when we write this in Python, we'll generally write it like this, rather than writing 10 to The negative 1 or 10 to the negative 2 we'll just write 1, a neg 1 or 1 e neg. Okay. I mean the same thing: you're going to see that all the time and remember that equals 0.1. Oh point: O one: okay, so don't be confused by this text that it prints out here this this loss here is the the final loss at the very at the end of it's not of any interest right so ignore this.
This is only interesting when we're doing regular trading. That's not interesting for the learning rate finder. The thing that's interesting for the learning rate finder is this loan shed plot and specifically we're not looking for the point where it's the lowest back to the point where it's the lowest, it's actually not getting better anymore. So that's to higher learning rate. So I generally look to see like where is it the lowest and then I go back like one for magnitude, so one enoch two would be a pretty good choice: yeah, okay! So that's why you saw when we ran our fit here. We picked 0.01 right, which is one a neg two, so important point to make here is like this. This is the one key number that we've learnt to adjust and if you just adjust this number at nothing else, most of the time you're going to be able to get pretty good results, and this is like a very different message to what you would hear or See in any textbook or any video or any course, because up until now, there's been like dozens and dozens of these they're called hyper parameters. Dozens and dozens of hyper parameters to set and they've been thought of as highly sensitive and difficult to set so inside. The fastai library, we kind of do all that stuff for you as much as we can and during the course we're going to learn that there are some more. We can quake to get slightly better results.
But it's kind of like it's kind of in a funny situation here, because for those of you that haven't done anything learning before is kind of like oh, this is that's all there is to it. This is very easy and then, when you talk to people outside this class, they'll be like deep learning so difficult as someone to say it's a real art form, and so that's why there's this as is difference right and so that the truth is that the learning Rate really is the key thing to set, and this ability to use this to figure out how to set it. Well, though, the paper is now probably 18 months old. Almost nobody knows about this paper. It was from a guy who's, not from a famous research labs. So most people kind of ignored it and, in fact even this particular technique was one subpart of a paper that was about something else. So again, this idea of like this is how you can set the learning rate. Really nobody outside this classroom. Just about knows about it, obviously the guy who wrote it Leslie Smith knows about it yeah. So it's a good thing to tell your colleagues about is like here is actually a great way to set the learning rate and there's even been papers. Caught like one of the famous papers is called no more pesky learning rates, which actually is a less effective technique than this one. But this idea that, like setting learning rates, is, is very difficult and, thirdly, is has been true for most of the kind of deep learning history. So here's the trick right go.
Look at this plot find kind of the lowest to go back about a multiple of ten and try that all right - and if that doesn't quite work, you can always try. You know going back another multiple ten, but this is always worked for me so far once why does this learning rate this method, work versus something else like momentum base or what's like the advantages, a disadvantage with just learning rate rate like technique? We're just feels. That's a great question, so we're going to learn during this course about a number of ways of improving gradient percent, like you mentioned momentum and atom and so forth. This is orthogonal. In fact. So one of the things the faster a library tries to do is figure out the right, gradient descent version and, in fact, behind the scenes, this is actually using something called atom. And so this technique is telling us. This is the best learning rate to use. Given what I thought other tweaks you're using in this case, the atom optimizer. So it's not that there's some compromise between this and some other approaches who sits on top of those approaches, and you still have to set the learning rate when you use with other approaches. So we're trying to find the best kind of optimizer to use for a problem that you still have to set the learning rate, and this is how we can do it. And, in fact, this idea of using this technique on top of more advanced optimizers.
Like Adam. Might haven't even seen mentioned in a paper before so I think this is like a I mean it's not a huge breakthrough, it seems obvious, but nobody else seems to tried it. So, as you can see, it was well when we use optimizers like Adam ditched, Harvick adaptive learning rate so, and he said this learning rate is Italy, initial learning rate because it changes during the people. So we're going to be learning about things like Adam the details about it later in the class, but the basic answer is no, even with even the Adam that there actually is a learning rate, it's just being it's being basically divided by the the gradient, the average Previous gradient and also the recent summer, Squared's of gradients, so there's still like a number called the learning rate there. There isn't a even these, so called dynamic learning rate methods still have unlearning rate. Okay, so the
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Most important thing that you can do to make your model better and is to give it more data. So the challenge that happens is that these models have hundreds of millions of parameters and, if you train them for a while, they start to do what's called overfitting. And so overfitting means that they're going to start to see. Like the specific details of the images you're. Giving them, rather than the more general learning that can transfer across to the validation set, so the best thing we can do to avoid overfitting is to find more data now. Obviously, one way to do that would just be to collect more data from where you're getting it from or label more data, but a really easy way that we should always do is to use something called data augmentation. So they don't open tuition. Is one of these things that's key in many courses, it's not even mentioned at all, or if it is it's kind of like an advanced topic right at the end, but actually it's like the most important thing that you can do to make a better model. Okay, and so it's built into the faster you library, to make it very easy to do, and so we're going to look at the details of the code shortly. But the basic idea is that, as in our initial code, we had a line that said image classified data from parts and we passed in the path to our data and for transforms. We passed in basically the sizing, the architecture. We'll look at this in more detail.
Shortly, we just add one more parameter, which is what kind of data augmentation do you want to do and so to understand data augmentation? It's may be easiest to look at some pictures of data augmentation. So what I've done here again we'll look at the code in more detail later, but the basic idea is: oh, I've run I've built a data class like multiple times, I'm going to do it six times and each time I'm going to plot the same catch and You can see that what happens is that this cap here is further over to the left. This one here is further over to the right, and this one here is fit horizontally and so forth. So data augmentation different types of image, you're going to want different types of data, augmentation right. So, for example, if you were trying to recognize letters and digits, you wouldn't want to flip horizontally because, like it's actually has a different meaning, whereas on the other hand, if you're looking at photos of cats and dogs, you probably don't want to fit vertically, because cats Aren't generally upside down all right, where else, if you're looking at there's a current Kaggle competition, which is recognizing icebergs in satellite images, you probably do want to fit them upside down, because it's really matter which area around the iceberg or the satellite was right. So one of the examples of the transform sets we have is transforms sidon.
So, in other words, if you have photos that are like generally taken from the side, which generally means you want to be able to flip them horizontally, but not vertically, this is going to give you all the transforms you need for that, so it'll flip them sideways. Rotate them by small amounts, but not too much and slightly bury their contrast and brightness and slightly zoom in and out a little bit and move them around a little. So each time it's a slightly different fight, billionaires we're getting a
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Couple of questions from people about you, explaining, in the reason why you don't think the minimum of the loss curve yeah, but it's like the higher rate, so yeah and also could you people understand if this works for every CNN for CNN every minute. There's a running right: fine done yeah, exactly yeah, okay, great um! Could you put your hand up if there's a spare seat next to you? So there was a question about the learning rate finder about. Why do we use the learning rate? That's less than the lowest point, and so the reason why is to understand what's going on with this learning rate finder? So, let's go back to our picture here like how do we figure out what learning rate to use right, and so what we're going to do is we're going to take steps and each time we're going to double the learning rate, so kind of double the amount By which we multiply the grander gradient, so, in other words, would go tiny step slightly, bigger, slightly bigger, slightly bigger, slightly bigger, slightly bigger, slightly bigger okay, and so the question is of the purpose of this is not to find the minimum. The purpose of this is to figure out what learning rate is allowing us to decrease quickly right. So the point at which the loss was lowest here is actually there right, but that learning rate actually looks like it's probably too high.
It's going to just jump like probably backwards and forwards, okay, so, instead what we do is we go back to the point where the learning rates quickly are giving us a quick increase in the loss. So here is so here is the actual learning rate increasing every single time. We look at a new mini batch, so mini-batch reiteration versus learning right and then here is learning rate versus loss. So here's that point at the bottom, where is now already too high? Okay and so here's the point where we go back a little bit and it's increasing nice and quickly we're going to learn about something called stochastic gradient descent with restarts shortly, where we're going to see like in a sense, you might want to go back to 1 Enoch 3, where it's actually even steeper still, and maybe we would actually find this book actually learn even quicker. You could try it, but we're going to see later why actually using a higher number is going to give us better generalization. So for now, let's put that aside, do you mean higher learning rate? When you say I know, I mean higher letting retina say higher yeah yeah I mean I am learning rate so as we increase the iterations from the learning rate finder, the learning rate is going up. This is iterations versus learning, ready. Okay, so as we do that as the learning rate increases - and we plot it here - the loss Goes Down, and here we get to the point where the learning rate is too high and at that point the most is nowgetting worse.
Because I asked the question because you were just indicating that you know, even though the minimum was at 10 to the minus 1, you were gon na. You suggest that we should choose 10 to the minus 2, but now you're saying I mean we should go back. The other way higher, so I didn't mean to say that I'm sorry if I said something backwards, I want to go back down to the lower learning rate. So possibly I said a higher when I meant higher into this lower OS. Do you know I'm learning right? Okay, thanks yep last class, is said that the local, all the local minima are the same, and this graph also shows the same. Is that is this something that was observed or is the logic theory behind it? That's not what this graph is showing. This graph is simply showing that there's a point where, if we increase the learning rate more, then it stops getting better than actually starts getting worse. The idea that all local minima are the same is a totally separate issue and it's actually something else, we'll see a picture of shortly. So let's come back to that Jeremy. Do we have to find the base learning rate every time we are going to run a poke third time, we're running on a poke and a pop? So how many times should I run this like? Let me write find my training. That's a great question unit um. I certainly run it once when I start later on in this class we're going to learn about unfreezing layers and after I unfreeze layers, I sometimes run it again.
If I do something to like change the thing, I'm training or change the way, I'm training it. You may want to run it again, basically, or you know, if you particularly if you've changed something about how you train, like unfreezing layers, which we're gon na soon learn about and you're finding the other training is unstable or too slow. Well again, you can run it again, there's never any harm in running it. It doesn't take very long. That's great question. Okay, so back to
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Data augmentation, so if we add to a when we run this little transforms from model function, we pass in orientation transforms. We can pass in the main to a transform side on or transforms top down later on, we'll learn about creating your own custom, transform lists as well, but for now because we're taking pictures from the side, cats and dogs will say, transform side on and now each Time we look at an image, it's going to be zoomed in or out a little bit moved around a little bit rotated a little bit, possibly flipped, okay, and so what this does is it's not exactly creating new data, but as far as the convolutional neural net Is concerned, it's a different way of looking at this thing, and it actually therefore allows it to learn how to recognize cats or dogs from somewhat different angles right. So when we do data orientation, we're basically trying to say based on our domain knowledge here here are different ways that we can mess with this image that we know still make it the same image you know and that we could expect that you might actually see That kind of image in the real world, so what we can do now is when we call this from parts function which we'll learn more about shortly. We can now pass in this set of transforms, which actually have these augmentations in now. So that's going to we're going to start from scratch here we do a fit and initially the augmentations actually don't do anything and the reason initially they don't do.
Anything is because we've got here, something that says: precompute equals true we're going to come back to these lots of times, but basically what this is doing is. Do you remember this picture? We saw where we learn each different layer has these activations that basically look for it or anything from the middle of flowers to eyeballs of birds or whatever right, and so literally, what happens is that the the later layers of this convolutional neural network have these things Called activations and activation literally it's a number. An activation is a number that says this feature like eyeball of bird is in this location, with this level of confidence with its probability right, and so we're going to see a lot of this later. But what we can do is we can say all right. Well, in this we've got a pre trained network. Remember and a pre trained network is one where it's already learned to recognize certain things. In this case, it's learnt to recognize the one and a half million images in the imagenet dataset, and so what we could do is we could take the the second last layer. So the one which is like got all of the information necessary to figure out what kind of thing a thing is and we can save those activations. So basically saving things saying you know, there's this level of eyeball nurse here in this level of dogs facing us here or in this level of fluffy, ear there and so forth, and so we save for every image these activations and that we call them the pre Computed activations, and so the idea is now that when we want to create a new classifier which can basically take advantage of these pre computed applications, we can just very quickly train when all the details there shortly, we can very quickly train a simple linear model based On those - and so that's what happens when we say pre-compute equals true, and that's why you may have noticed this week, the first time that you run a model, a new model.
It takes a minute or two. Where else you saw when I ran it, it took like five or ten seconds, took you a minute or two, and that's because it had to pre-compute these activations and just has to do that once if you're, using like your own computer or AWS, it just has To do it once ever, if you're using Crestle, it actually has to do it once every single time you rerun press all because press or uses are just for these pre computed activations. It uses a special that all had a scratch space that disappears each time. You restart your press, or instance, so other than the special case of cresol generally speak. He does have to run at once ever for a data set okay. So the issue with that is that since we pre computed for each image, you know how much does it have an EI here and how much does it have a lizard's eyeball there and so forth? That means that data augmentations don't work right. In other words, even though we're trying to show at a different version of the cat each time, we've pre computed the activations for a particular version of that cat. So in
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Order to use data augmentation, we just have to go and learn. Pre compute equals false okay, and then we can run a few more APIs right, and so you can see here that, as we run more a Potts, the accuracy isn't particularly getting better. That's the bad news. The good news is that you can see that the train loss practices like the way of measuring the error of this model. Although that's getting better the errors going down, the validation error isn't going down, and but we're not overfitting and overfitting would mean that the training loss is much lower than the validation loss and we're going to talk about that. A lot during this course. But the general idea here is, if you're doing much better job on the training set, then you are on the validation set. That means your models, not generalize, so we're not at that point, which is good, but we're not really improving. So we're going to have to figure out how to deal
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
With that before we do, I want to show you one other cool trick. I've added here cycle length equals one, and this is another really interesting idea. Here's the basic idea cycle length equals one enables a recent fairly recent discovery and deep learning called stochastic gradient descent with restarts and the basic idea is this as you as you get closer and closer as you get closer and closer to the right spot right now. Getting closer and closer, I may want to start to decrease my learning rate right because, as I get closer, I'm kind of like oh I'm, pretty close down. So, let's, let's slow down my steps to try to get executive, the right spot right, and so, as we do more iterations, our learning rate perhaps should actually go down right because, as we go along we're getting closer and closer to where we want to be - and We want to like get exactly to the right spot. Okay, so the idea of decreasing the learning rate, as you train, is called learning rate annealing and it's it's very, very common, very, very popular everybody uses it basically all the time. The most common kind of learning rate annealing is really horrendously hacky. It's basically that researchers like pick a learning rate that seems to work for a while and then when it stops learning well, they drop it down by about 10 times, and then they keep learning a bit more until it doesn't seem to be improving and they drop It down by another ten times, that's what most academic research papers and most people in industry do so this would be like stepwise annealing very manual.
Very annoying. A better approach is simply to pick some kind of functional form like a line. It turns out that a really good functional form is one half of the cosine curve right and the reason. Why is that for a while when you're, not very close, you kind of have a really high learning rate, and that is you do get close. You kind of quickly drop down and do a few iterations with a really low learning rate, and so this is called cosine annealing. So to those of you who haven't done trigonometry for a while, cosine basically looks something like this right. So we've picked one little half piece: okay, so we're going to use cosine annealing, but here's the thing when you're in a very high dimensional space right near we're only able to show three dimensions right, but in reality we've got hundreds of millions of dimensions. We've got lots of different, fairly flat points there. No, not the actual local minima, but they're fairly flat points, all of which are pretty good right, but they might differ in a really interesting way, which is that some of those flat points? Let me show you: let's imagine: we've got a surface that looks something like this right now. Imagine that, where you kind of random guest started here and our initial therefore kind of learning rate annealing schedule got us down to here now.
Indeed, that's a pretty nice low error right, but it probably doesn't generalize very well, which is to say if we use a different data set where things are just kind of slightly different in one of these directions. Suddenly is a terrible solution right where else over here is basically equally good in terms of loss right, but it rather suggests that, if you move, if you have slightly different data, sets that are slightly moved in different directions, it's still going to be good right. So, in other words, we would expect this solution here is probably going to generalize better than this by key one. So here's what we do is we've got like a bunch of different low bits. Right then, our standard loading rate, annealing approach will start of go down here or downhill downhill, downhill downhill to one spot right, but what we could do instead is use a learning rate schedule. That looks like this, which is to say, we do a cosign annealing and then suddenly jump up again into a cosign, annealing and then jump up again and so each time we jump up. It means that if they're going to spiky bit and then we subtly increase the learning rate - and it jumps now all the way over to here and so then we kind of learning right in your learning right near death down to here, and then we jump up Again to a high learning rate: oh and it stays here right, so in other words, each time we jump up the learning rate.
That means that if it's in a nasty spiky part of the surface, it's going to hop out of the spiky part, and hopefully, if we do that enough times, it'll eventually find a nice smooth Bowl. Could you get the same effect by running multiple iterations through the different ground of my starting point, so that eventually, you explore all possible minimize yeah. So, in fact, that that's a great question and before this approach, which is called stochastic gradient, descent with restarts was, was created. That's exactly what people used to do. They used to create these things called ensembles, where they would basically relearn a whole new model ten times in the hope that one of them's like, but it ended up being better, and so the cool thing about this decosta gradient descent with restarts is that the model Once we're in a reasonably good spot, each time we jump up the learning rate, it doesn't restart, it actually hangs out in this nice part part of the space and then keeps getting better. So, interestingly, it turns out that this approach, where we do this a bunch of separate cosine annealing steps, we end up with a better result as then, if we just randomly try it a few different starting points. So it's a super neat trick and it's a fairly recent development, but and again almost nobody's heard of it.
But I found like it's now, like my superpower like using this, along with the learning rate finder like I, can get better results than nearly anybody like in a casual competition. You know in the first week or two I can like jump in. It's been an arrow to and back I've got a fantastically good result, and so this is why I didn't pick the point where it's got the steepest slope. I actually trying to pick something kind of aggressively high. It's still getting down, but maybe like getting to the point where it's nearly too high, not because I want to make sure, because that's because, when we do this, stochastic gradient descent with restarts this ten to the negative two represents the a highest number that it uses. So it goes up to ten to the negative two and then goes down and then up to ten negative two and down. So if I use to lower learning rate, it's not going to jump to a different part of the function. So I have a few questions, but the first one is: how many times do you change your learning rate? You want to work, we don't change the learning rate, all three, how many times? Okay. So, in terms of this part here, where it's going down, we change the learning rate, every single mini all right and then the number of times we reset it is set by the cycle, length, parameter and so 1 means reset it up to every epoch. So if I had to there, it would reset it up to every to epochs and, interestingly this, this point that when we do the learning rate and kneeling that we actually change it, every single batch it turns out to be really critical to making this work, and It again is very different to what nearly everybody in industry in academia has done before.
What do you get a chance? Could you explain recompute? It was true because it's still yeah we're going to come back to that multiple times in this course. So the way this course has been a work is we're going to like do a really high-level version of each thing and then we're going to like come back to it in two or three lessons and then come back to it. At the end of the course and each time we're going to see like more of the math more of the code and get a deeper view, okay - and we can talk about it - also in the forums during the week. Our main goal is to generalize and we don't want to get those like narrow, demas yeah. That's a it's a very short summary. This method. Are we keeping track off to minimize and averaging them? Ah, that's that's another level of sophistication and indeed you can see there's something here called snapshot ensemble, so we're not doing it in the code right now. But yes, if you wanted to make us generalize even better, you can save the weights here and here and here and then take the average ishes. But for now we're just going to pick the last one. If you want to skip ahead, if you want to skip ahead, there's a parameter called cycle, safe name, which you can add as well as cycle them, and that will save a set of weights at the end of every learning rate cycle. And then you can ensemble them.
Ok, so we've
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Got a pretty decent model here. Ninety nine point: three percent accuracy and we've gone through of you know a few steps that is taken. You know a minute or two to run, and so from time to time I tend to save my weight. So if you go, learn, dot, save and then pass in a file name, it's going to go ahead and save that for you later on. If you go, learn load you'll be straight back to where you came from okay. So it's a good idea to do that from time to time. This is a good time to mention what happens when you do this when you go, learn dot save when you create precomputed activations. Another thing we learn about soon when you create resized images. These are all creating various temporary files, okay, and so what happens is if we go to data and we go to dogs cats. This is my data. Folder and you'll see there's a folder here called TMP or, and so this is automatically created and all of my pre computed activations end up in here. I mention this because, if, if things are, if you're getting weird errors, that might be because you've got some, Oh pre computed activations like we're only half completed or are in some way incompatible with what you're doing so. You can always go ahead and just delete this TMP, this temporary directory and see if that causes your error, to go away. This is the faster I equivalent of turning it off and then on again.
You'll also see there's a directory called models, and that's where all of these, when you say dot, save with a model. That's where that's going to go. Actually it reminds me when the stochastic gradient descent with restarts paper came out. I saw a tweet that was, somebody was like, Oh to make your deep learning work, better turn it off and then on again question it. So, if I want to see, I want to retrain my model fuselage again. Do I just do everything the 10 folder? If you want, if you want to train your model,
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
From scratch, there's generally no reason to delete the pre computed activations, because the pre computed activations are without any training. That's what the pre trained model created with the with the weights that you downloaded off the internet, the only yeah I mean. The only reason you want to delete the pre computed activations is that there was some error caused by like half creating them and crashing or some something like that, as you change the size of your import change, different architectures and so forth, they all create different sets Of activations with different file names, so you don't generally you shouldn't have to worry about it. If you want to start training again from scratch, all you have to do is create a new learn object. So, each time you go like conch learner, dot, pre-trained. That creates a new object with with new sets.
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Of weights fever train from okay, so before our break, we'll finish off by talking about about fine-tuning and differential learning rates, and so so far everything we've done has not changed any of these free trained filters right. So we've used a pre trained model. That already knows how to find at the early stages, edges, ingredients and then corners and curves, and then repeating patterns and bits of text and eventually eyeballs right. We have not retrained any of those activations any of those features. Well, specifically, any of those weights in the convolutional kernels, all we've done is: we've learnt some new layers that we've added. On top of these things, we've learned how to mix and match these pre-trained features. Now, obviously, it may turn out that your pictures have you know different kinds of eyeballs or faces or if you're, using different kinds of images like satellite images, totally different kinds of features altogether right. So if you're like training to recognize icebergs, you'll, probably want to go all the way back and learn, you know all the way back to kind of different combinations of these simple gradients and edges. In our cases, dogs, vs. cats, we're going to have some minor differences, but we still may find it's helpful to slightly tune some of these later layers, as well. So to tell the learner that we now want to start actually changing the convolutional filters themselves.
We simply say unfreeze okay, so a frozen layer is a layer which is not trained is not updated. Okay, so unfreeze unfreezes all of the layers. Now, when you think about it, it's pretty obvious that layer, one right which is like a diagonal edge or a gradient. Probably doesn't need to change by much if at all, right from the 1 and a half million images on image net, it probably already is figured out pretty well how to find like edges of gradients. It probably already knows also like which kind of corners to look for and how to find which kinds of curves and so forth, so, in other words, these early layers probably need little. If any learning, where else these later ones are much more likely to need more learning - and this is universally true, regardless of whether you're looking for satellite images of rainforests or icebergs or whether you're looking for cats versus dogs right, so what we do is we create An array of learning rates where we say okay: these are the learning rates to use for our additional layers that we've added on top. These are the learning rates to use in the middle few layers, and these are the learning rates to use for the first few layers. So these are the ones for the layers that represent like very basic, geometric features. These are the ones that are used to for the more complex kind of sophisticated convolutional features, and these are the ones that are used for the features that we've added and went from stretch right.
So you can create a array of learning rates and then, when we called up fit and pass an array of learning rates, it's now going to use those different learning rates for different parts of the model. This is not something that we've like invented, but I'd also say it's like it's, so not that common, that it doesn't even have a name as far as I know so we're going to call it differential learning rates if it actually has a name or indeed, if Somebody's actually written a paper specifically talking about it. I don't know, there's a great researcher, called Jason, your Sinskey, who who did write a paper about the kind of the idea that you might want different learning rates and showing why? But I don't think any other libraries support it and yeah. I don't know of a name for it. Having said that, though, this ability to like unfreeze and then use these differential learning rates, I found it's like the secret to taking a pretty good model and putting it into an awesome model, so just to clarify, so you have three numbers there: okay, three hyper parameters: The first one is the photo late model, so the mall that are late layers, the so with the it's. Your answer is many, many right and they're kind of in groups and we're going to learn about the architecture. This is called a ResNet for residual network. It kind of has ResNet blocks, and so what we're doing is we're grouping the blocks into three groups, and so this one is actually.
This first number is for the earliest layers, yeah they're ones closest to the pixels that represent like corners and edges and gradients. But why why do you well? I thought those layers are frozen at first, so yeah right, so we just said unfreeze the streets. Also, we so yeah, I'm freezing them because you have kind of partially trained, although lately we've trained, we've trained our added layers. Yes, now you are, we training the Oh step exactly obviously so it waits and the learning rate is particularly small for the early layers. That's right because you just find a want to find food yeah yeah. We probably don't want to change them at all, but you know if it does need to. Then it can thanks no problem so using the differential in rates a little different from like grid search. There's, no similarity to grid search, so grid search is where we're trying to find the best hyper parameter for something. So, for example, you could kind of think of the learning rate finder as a really sophisticated grid search, which is like trying lots and lots of learning rates to find which one is best, but this has nothing to do with that. This is actually for the entire training. From now on, it's actually going to use a different learning rate for each layer, and so I was wondering so you give a pre train model. Then you have to use the same input dimensions right because I was thinking.
Okay, let's say you have this big: they use like big machines to train these things and you want to take advantage of it.  You have like images that are like bigger than the ones that they used or we're going to be talking about sizes later. But the short answer is that with this library and the modern architectures were using, we can use any size we like so did I mean do we need? Can we at least just a specific layer? We can we're not doing it yet. But if you wanted to, you can learn dot, freeze, underscore two and pass into layer number much to my surprise, or at least initial my surprise. It turns out. I almost never need to do that. I almost never find it helpful and I think it's because we're using differential learning rates, the the optimizer can kind of learn just as much as it needs to so yeah. It's a little data like very little data yeah. It still doesn't seem to help the one place I have found it helpful is, if I'm using like a really big memory, intensive model and I'm like running out of GPU crazy having the the less layers you unfreeze, the less memory it takes and the less time It takes so there's that kind of practical aspect. So to me she'll say I asked the question right. Can I just like unfreezes specific layer? No, you. You can only unfreeze layers from layer n onwards. You could probably delve inside the library in phase one phase, one layer, but I don't know why you would okay.
So I'm really excited to be showing you guys this stuff, because it's like it's something. We've been kind of researching all year, it's figuring out how to train state of the art models and we've kind of found these like tiny number of tricks, and so once we do that, we now go learn about fit right and you can see. Look at this. We get right up to that 99.5 % accuracy, which is crazy. There's one other trick. You might see here that, as well as using stochastic gradient descent with restarts a cycle length equals one. We've done three cycles so earlier on. I lied to you. I said this is this: is the number of epochs it's actually the number of cyclists right. So if you said cycle length equals two, it would do three cycles of each of two epochs or do six, because so here I've said two three cycles. Yet somehow it's done seven epochs and the reason why is I've got one last trick to show you which is cycle mult equals two and to tell you what that does I'm simply going to draw you a picture you? The picture, if I go learn Dutch share top plot learning rate there. It is now you can see what cycle mode equals to is doing. Okay, it's it's doubling the length of the cycle after each cycle, and so in the paper that introduced this stochastic gradient descent. With restarts the researcher kind of said, hey, this is something that seems to sometimes work pretty well, and I've certainly found that often to be the case.
So, basically, intuitively speaking, if your cycle length is too short right, then it's kind of starts going down to find a good spot and then it pops out - and it goes to a try and photographs button pops out it never actually gets to find a good spot Right so earlier on, you want it to do that, because it's trying to find the bit that's like smoother but then later on, you want it to fight, do more, exploring and then more exploring right. So that's why this cycle mole equals two thing often seems to be a pretty good approach right, so suddenly we're introducing more and more hyper parameters. Having told you that there aren't that many, but the reason is that, like you, can really get away with just taking a good learning rate, but then adding these extra tweaks really helps get that extra level up without any effort right and so in practice. I find this kind of three cycles, starting at 1 mode, equals 2 works very, very often to get a pretty decent model if it does doesn't, then often I'll just do 3 cycles of length 2 with no molt, okay, there's kind of like two things that seem To work a lot and there's not too much fiddling, I find necessary and, as I say, even even if you just if you use this line every time, I'd be surprised if you didn't get a reasonable result. So a question here: why does a smoother services correlate to more
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Generalize networks, so it's kind of this some this intuitive explanation. I try to just kill the whole thing I try to give back here, which is that if you've got something spiky right and so what this, what this x-axis is showing is like how? How good is this at recognizing dogs versus cats, as you change, this particular parameter right, and so so something to be generalizable. That means that we wanted to work when we give it when we give it a slightly different data set, and so a slightly different data set may have a slightly different relationship between this parameter and how caddy versus dog it is. It may instead look a little bit like this right, so in other words, if we end up at this point right, then it's not going to do a good job on this slightly different data set. For else. If we end up on this point, it's still going to do a good job on this data set okay. So that's what psychomotor equals doing? Okay, so we've got one last thing before we going to take a break, which is we're now going to take this model, which has 99.5 percent accuracy and we're going to try and make it better still. And what we're going to do is we're not actually going to change the model at all right, but instead we're going to look back at the original virtual visualization. We did where we looked at some of our incorrect pictures.
Now, what I've done is I've printed out? The whole of these incorrect pictures, but the key thing to realize is that particularly in fact, when we do the the validation set, all of our inputs to our model, all the time have to be square right, and the reason for that is, it's kind of a Minor technical detail, but basically the GPU doesn't go very quickly if you have like different dimensions for different images, because it needs seems to be consistent so that every part of the GPU can do the same thing, and I think this is probably fixable, but it now. That's the state of the technology we have so our validation set when we actually say for this particular thing is it's a dog what we actually do to make it square, as we just pick out the square in the middle right, so we would take off its Two edges, and so we take the whole height and then as much of the middle as we can, and so you can see in this case we wouldn't actually see this dog's head right, so I think the reason this was actually not correctly classified was because the Validation set only got to see the body and the body doesn't look particularly doglike or cat-like. It's not at all, punctual what it is. So what we're going to do when we calculate the predictions for our validation set, is we're going to use something called test time.
Augmentation and what this means is that every time we decide is this cat or a dog not in the training but after we've trained the model is we're going to actually take four random data augmentations and remember the data augmentations move around and zoom in and out And flip okay, so we're going to take four of them at random and we're going to take the original and augmented sent a cropped image and we're going to do a prediction for all of those and then we're going to take the average of those predictions. So I'm going to say is this a cat? Is this a cat? Is this a cat? Is this a cat, but, and so hopefully in one of those random ones? We actually make sure that the face is there zoomed in by a similar amount to other dogs faces at sea, and it's rotated by the amount that it expects to see it and so forth, and so do that. All we have to do is just call tt8. Tta stands for Test time, augmentation this term of like what a? What do we call up when we're making predictions from up from a model we've trained? Sometimes it's called inference time. Sometimes it's called test time. Everybody since have a different name, so TTA, and so when we do that we go learn, TTA check the accuracy and lo and behold we're now at ninety nine point, six five percent, which is kind of crazy where's, our green box, but for every park we are Only showing one type of augmentation or for particular image right so when we are training back here, we're not doing any TTA right, so TTA is not like you could and sometimes like I've written libraries where, after a cheap up, I run TTA to see how.
Well, it's going, but that's not what's happening here. I trained the whole thing with training time organization which doesn't have a special name, because that's what we mean when we say data augmentation. We need training, time augmentation. So here every time we showed a picture, we were randomly changing it a little bit so each epoch, each of these seven epochs. It was seen slightly different versions of the picture. Having done that, we now have a fully trained model. We then said: okay, let's look at the validation set, so TTA by default, uses the validation set and said: okay, what are your predictions of? Which ones are cats and which ones are dogs, and it did 4 predictions with different random orientations, plus one on the organ. Under Augmented version, average them all together and that's what we got and that's what we can't clear the accurate. So is there a high probability of having sample in TTA that was not shown in doing trained yeah. Actually, every data, augmented for image is, is unique because the rotation could be like point. Zero. Three four degrees and zoom could be 1.0 one sixty five. So every time it's slightly different, no problem was behind you.  Oh padding's. Not yes! So, like there's lots of different types of a better orientation you can do, and so one of the things you can do is to add a border around it.
Basically adding a border around it in my experiments doesn't doesn't help it doesn't make it any less cat-like. It's not the convolutional neural network doesn't seem to find it very interesting. Basically, something that I do do we'll see later is. I do something called reflection, padding, which is where I add some borders that are the outside just reflected. It's a way to kind of make some bigger images works well with satellite imagery in particular, but yeah in general. I don't do I have a lot of padding. Instead, I do a bit of zooming, it's kind of follow-up to that last one, but rather than cropping just at white space, because when you crop you lose the dog's face. But if you added white space you wouldn't yeah. So that's that's where the kind of the reflection padding or the zooming or whatever can help. So there are ways in the faster. You know, library, when you do custom transforms of of making that happen, I find that it kind of depends on the image size. You know, but, generally speaking, it seems that using TTA plus data augmentation, the best thing to do is to try to use this larger image as possible, and so, if you kind of crop the thing down and put white borders on top and bottom, it's now quite A lot smaller and so to make it as big as it was before.
You now have to use more GPU, and if you're going to use more that multi figure, you could have zoomed in and used a bigger image. So in my playing around that doesn't seem to be generally as successful. There is a little interest on the topic of how do the domain tation in older than images. Indeed at least not images, um yeah, um, no one seems to know I actually um. I asked some of my friends in the natural language processing community about this we'll get to natural language processing. In a couple of lessons you know it seems like it'd, be really helpful. There's been a few example. I carry very few number examples of people where papers would like try replacing synonyms, for instance, but on the whole and understanding of like appropriate data. Augmentation for non image domains is under-researched in under under developed. The question was: could couldn't we just use a sliding window to generate on the images so in that dog? Thank you. Couldn't we generate three parts of it? Wouldn't that be better yeah PTI you mean just just in general, when you're creating your so training time. I would say no that wouldn't be better, because we're not gon na get as much variation. You know we want to have it like, like one degree off five, you know five degrees off ten pixels up like lots of slightly different versions, and so, if you just have three standard ways, then you're not giving it as many different ways of looking at the Data for testing augmentation having fixed cropped locations - I think, probably, would be better - and I just haven't gotten around to writing that. Yet I have a version in an old library.
I think having fixed cropped locations plus random contrast, brightness rotation changes might be better. The reason I've got around to it yet is because in my testing it didn't seem to help him practice very much and it made the code a lot more complicated. So you
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Know it's kind of it's an interesting question. I just wanted all of this last AI api's that you are using. Is it yeah? That's a great question, so the faster you go, libraries open source and let's talk about it, a bit more generally, because you know it's like the fact that the fact that we're using this library is kind of interesting and unusual and it sits on top of something Called a torch right, so pytorch is a fairly recent development and it's kind of I've noticed all the researchers that I respect pretty much are now using high torch. I found in part two of last year's course that a lot of the cutting-edge stuff I wanted to teach I couldn't do it in chaos and tensorflow, which is what we used to teach with, and so I had to switch the course to pay torch halfway through Part two: the problem was that pytorch isn't very easy to use. You have to write your own training loop from scratch. I basically write everything from scratch or the stuff you see inside the class. They are library we would have had to written it. You know to learn, and so it really makes it very hard to learn deep learning when you have to write hundreds of lines of code to do anything. So so we decided to create a library on top of pytorch, because we, you know this. Our mission is to teach world class big morning, so we wanted to show you like here's how you can be the best in the world at doing it, and we found that a lot of the world class stuff we needed to show really needed pytorch or At least with pytorch, it was far easier and but then PI thought itself just wasn't suitable as a first thing to teach with for new for new deep learning practitioners.
So we built this library on up of pytorch, initially heavily influenced by chaos, which is what we taught last year and but then we realized, we could actually make things much much much easier than care us. So in care us. If you look back at last year's course notes, you'll find that all of the code is two to three times longer and there's lots more opportunities for stakes, because there's just a lot of things you have to get right, so we ended up kind of building this. This this library, in order to make it easier to get into deep learning, but also easier to get state-of-the-art results and then, over the last year, as we started developing on top of that, we started discovering that by using this library, it made us so much more Productive that we actually started kind of developing you, state-of-the-art results and new methods ourselves, and we started realizing that there's a whole bunch of like papers that have kind of been ignored or lost, which, when you use them, it could like automate or semi-automated stuff. Like learning read, finder, that's not in any other library, so so it kind of got to the point where now not only is kind of fastai lets us do things easier, much easier than any other approach, but at the same time it actually has a lot More kind of sophisticated stuff behind the scenes than anything else, so so it's kind of an interesting mix so yeah.
So we've released this library like at this stage, it's like very early version and so through this course. By the end of this course, I hope, as a group, you know we will all a lot of people are already helping, have developed it into something. That's you know really pretty stable and rock-solid and yeah. Anybody can then can use it to build your own models under an open-source license. As you can see it's available on github behind the scenes, it's it's creating play, torch models and so apply torch models can then be exported into various different formats. Having said that, like a lot of folks like issue, if you want to do something on a mobile phone, for example, you're probably going to need to use tensorflow and so later on. In this course we're going to show like how some of the things that we're doing in the past AI library, you can do in chaos and cancel flow. So you can going to get a sense of what the different libraries look like and, generally speaking, the simple stuff is like it'll. Take you a small number of days to learn to do it and care us in tensorflow versus fastai and high torch and the more complex stuff. Often this won't be possible so that, like, if you needed to be intensive flow, you're, just kind of simplify it off in a little bit. But you know, I think the more important thing to realize is every year the kind of the libraries that are available and which ones are the best totally changes.
So, like the main thing, I hope that you get out of this course is an understanding of the concepts like here's, how you find a learning rate. Here's why differential learning rates are important? Is they do learn where the kneeling? You know here's what stochastic gradient a second's restarts does so on and so forth, because you know by the time we do this course again. Next year you know the library situations and the difference the king. That's a question of that. I was wondering if you've had an opinion on pyro, which is ubers new release. I haven't looked at it, no I'm very interested in probabilistic programming and it's really cool that's built on top of paper. So one of the things we'll learn about in this course is we'll see that pytorch is much more than just a deep learning library. It actually lets us write arbitrary gpu-accelerated algorithms from scratch, which we're actually going to do and pyro is a great example of what people are now doing with might watch outside of the deep level. Great. Ok, let's take a eight-minute break and we'll come back at 7:55. So ninety nine point: six five
PAUSE
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Percent accuracy, what does that mean? So in classification, when we do classification and machine learning, the really simple way to look at the result of a classification is, what's called the confusion matrix. This is not just deep learning, but in any kind of classifier machine learning where we say okay, what was the actual truth? There were thousand cats and a thousand dogs out of the thousand actual cats. How many did we predict were cats? This is obviously in the validation step. This is the images that we didn't use to train with. It turns out. There were nine hundred ninety-eight cats that we actually predicted as cats and two that we got wrong. Okay and then for dogs. There were nine hundred ninety-five that we predicted were dogs and then five that we got wrong and so often these confusion matrices can be helpful, particularly if you've got like four or five classes. You're trying to predict to see like which group you having the most trouble with and you can see it uses color coding to tell you you know to highlight the large. The large bits you've got to hope that the diagonal is the highlighted section. So now that we've retrained the model, it can be quite helpful now that's better to actually look back and see like okay, which ones in particular were incorrect, and we can see here there were actually only two incorrect cats.
It prints out four by default, so you can actually see these two actually less than 0.5, so they weren't they weren't wrong. Okay, so it's actually. These two were wrong. Cats and this one isn't obviously a cat at all. This one is, but it looks like it's got a lot of weird artifacts and you can't see its eyeballs at all so and then here are the how many dogs, where they're all wrong there were five wrong dogs. Here are four of them. That's not! Obviously, a dog that looks like a mistake that looks like a mistake that one, I guess doesn't
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Have enough information that I guess it's a mistake, so so we've done a pretty good job here of creating a good classifier. I would based on entering a lot of capital competitions and comparing results. I've done two various research papers. I can tell you it's a state of the art classifier. It's it's right up there with the best in the world, we're going to make it a little bit better in a moment, but here in the basic steps right. So if you want to create a world-class image classifier, the steps that we just went through was that we started our week's term data augmentation on by saying oil transforms, equals and you either say sidon or top-down, depending on what you're doing start with pre compute equals. True find a decent learning, eight. We then train just like it one or two epochs, which that takes a few seconds as we got through compute equals true, then we turn off pre compute, which allows us to use data augmentation to do another two or three epochs. Generally, with cycle length equals one, then I unfreeze all them, as I then set the earlier layers to be like either somewhere between a 3 times 2 10 times mobile learning rate in the previous. So in this case I did 10 times right. So it's like this was my learning rate that I found from the learning rate finer than I went 10 times smaller and then 10 times smaller as a rule of thumb like knowing that you're, starting with a pre trained imagenet model.
If you know, if you can see that the things that you're now trying to classify a pretty similar the kinds of things in imagenet ie pictures of normal objects in normal environments, you probably want about a 10x difference, because you want those earlier layers, like you, think That the earlier layers are probably very good already, but also, if you're doing something like satellite imagery or medical imaging, which is not at all like image net, then you probably want to be training those earlier layers, a lot more, so you might have like. Oh just a 3/8 difference all right, so that's like one change that I make is to try to make it out of 10x or 3x. Yes, so then, after unfreezing, you can now call LR find again, but at Nike didn't in this case, but like once you've unfrozen all the layers, you've turned on differential learning rates. You can then call a lot of fine again right, and so you can then check like oh does it still look like the same point I had last time is about right. Something to note is that if you call LR find having set differential learning rates, the thing that's actually going to print out is the learning rate of the last layers right, because you've got three different learning rates, so it's actually showing you the last layer. So then yeah then I trained the full
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Network with cycle more equals two and it'll either it starts with the fitting or I run out of time right so like. Let me show you all right, so let's do this again, a totally different data set. So this morning I noticed that some of you on the forums were playing around with this playground. Kegel competition very similar, called dog breed identification, so the dog breed identification cat will challenge is one where you don't actually have to decide which ones are cats and which ones the dogs, they're all dogs. But you have to decide what kind of dog it is, but there are 120 different breeds of dogs. Okay, so you know. Obviously this could be like different types of cells and pathology, slides. It could be different kinds of cancers in CT scans. It could be different kinds of icebergs and satellite images, whatever right as long as you've got some kind of labeled images. So I want to show you what I did this morning, so it took me about an hour basically to go in to end from something I'd. Never seen before so I downloaded the data from kaggle and I'll show you how to do that shortly. But the short answer is: there's something called cable CLI, which is a github project you can search for and if you read the docs to basically run cagey download, provide the competition name and it will grab all the data for you to your crystal or Amazon or Whatever instance I put in my data folder and I then went LS, and I saw that it's a little bit different to our previous data set.
It's not that there's a train folder, which has a separate folder for each kind of dog, but instead of tonette there was a CSV file and the CSV file. I read it in with pandas, so pandas is the thing we use in python to do structured data analysis like csv files, so he picked pandas. We called pd, that's pretty much universal PDR. Htsb reads in the csv file. We can then take a look at it and you can see that basically, it had like some kind of identifier and then the debris right, so this is like a different way. This is the second main way that people kind of give you image labels. One is to put different images into different folders. The second is generally to give you as some kind of file like a CSV file, to tell you here's the image name and here's the label. Okay. So what I then did was I used pandas again to create a pivot table which basically groups it up just to see how many of each breed there were, and I sorted them, and so I saw okay they've got like about a hundred some of the more Common breeds and some of the less common breeds they've got like 60 or so okay. Altogether, there are 120 rows and I've been 120 different breeds represented. Okay, so I'm going to go through the steps right so enable data augmentation so to enable data augmentation. When we call this transforms from model, you just pass in and all transformers. In this case, I chose side on again. These are pictures of dots and stuff, so this side on photos.
I we're talking about maqsuum as more detail later, but maximum basically says when you do the data augmentation. We like zoom into it by up to one point one times: okay, so but randomly between one, the original image size and one point one points. So it's not always cropping out in the middle or an edge, but it could be cropping out a smaller part. Okay. So having done that, the key step now is to graphically going from paths. So previously we went from paths, and that tells it that the the names of the folders are the names of the labels. We go from CSV and we pass in the CSV file that contains the letters so we're passing in the path that contains all of the data, the name of the folder that contains the training data, the CSV that contains the labels. We need to also tell it where the test set is, if you want to submit to cattle later talk more about that next week. Now this time the previous data set, we had had actually separated a validation set out into a separate folder right, but in this case you'll see that there is not a separate folder called validation right. So we want to be able to track how good our performance is low, so we're going to have to separate some of the images out to put it into a validation set okay, so I do that at random and so up here you can see how it Basically opened up the CSV file, turned it into a list of rows and then taken the length of that minus one, because there's a header at the top right.
And so that's the number of rows in the CSV file, which must be the number of images that we have and then this is a fastai thing get cross-validation indexes now we'll talk about cross-validation later. But basically, if you call this and pass in a number, it's going to return to you by default a random twenty percent of the rows who uses your validation set and you can pass in parameters to get different amounts right. So this is now going to grab twenty percent of the data and say all right. This is the this is the indexes the numbers of the files which we're going to use as a validation set. Okay. So now that we've got that, in fact, let's kind of run this, so you can see what that looks like so well, indexes is just a big bunch of numbers, okay and so an is 10,000 right, and so we have about twenty percent of those is going To be in a validation set, so when we call from CSV we can pass in a parameter which is talent which indexes to treat us a validation set, and so that's passed in those indexes. One thing that's a little bit tricky here is that the file names actually have. I checked, they actually have a dot jpg on the end, and these obviously don't have a dot jpg. So you can pass in when you call from CSV you can pass in a suffix. It says that the labels don't actually contain the full file names. You need to add this to them.
Okay, so that's basically all I need to do to set up my data and, as a lot of Europe noticed during the week inside that data object, you can actually get access to the data set like what the training data set by same train. Yes and inside train des is a whole bunch of things, including the file names. Okay, so train desktop file. Names contains all of the file names of everything in the training set, and so here's like one file name, okay, so here's an example of one file name. So I can now go ahead and open that file and take a look at it. That's the next thing I did was to try and understand what my file my dataset looks like, and it found an adorable puppy, so that was very nice, so feeling good about this. I also want to know like how big of these files right, like how big are the images, because that's a key issue, if they're huge and then I have to think really carefully about how to deal with huge images, that's really challenging if they're tiny. Well, that's also challenging. Most of imagenet models are trained on either 224 by 224 or 299 by 299 images. So anytime, you have images in that kind of range. That's that's really hopeful you're, probably not going to have to do too much different in this case. The first image I looked at was about the right size, so I'm thinking of pretty hopeful.
So what I did then, is: I created a dictionary comprehension now, if you don't know about list, comprehensions and dictionary comprehensions in Python, go study them they're. The most useful thing super handy you can see the basic idea here is that are going through all of the files and then putting a dictionary that map's the name of the file to the size of that file. Again, this is a handy, little Python feature which I'll, let you think learn about during the week. If you don't know about it, which is zip and using a special star, notation, is never to take this dictionary and turn it into the rows and the columns, and so I can now turn those into num pay, arrays and like okay. Here are the first five rows sizes for each of my images, and then matplotlib is something you want to be very familiar with. If you do any kind of data science or machine learning in python matplotlib, we always refer to as PLT as if this is a histogram. And so I got a histogram of the how high how many rows there are in each image. So you can see here. I'm kind of getting a sense before I start doing any modeling. I kind of need to know what I'm modeling with and I can see. Some of the images are going to be like 2500 3000 pixels high, but most of them seem to be around 500. So, given it so few of them were bigger than a thousand.
I use standard numpy slicing to just grab those at a smaller than a thousand and histogram that just to zoom in a little bit - and I can see here all right - it looks like yet. The vast majority are around 500, and so this actually also prints out the histogram, so I can actually go through and I can see here for four thousand five hundred of them are about 450, okay, so I get about that seems about anywhere so generally, how many Images should we get in the validation set is always a 20 %, so the size of the validation set like using 20 % is fine. Unless you kind of feeling like my data is my data sets really small. I'm not sure. That's enough, you know like if you've got basically think of it. This way, if you train like the same model multiple times and you're, getting very different validation set results and your validation sets kind of small but smaller than a thousand or so then it's going to be quite hard to interpret how well you're doing now. This is particularly true like if you're like, if you care about the third decimal place of accuracy and you've, got like a thousand things in your validation set. Then you bring about like a single image. Changing class is changing. You know it's what you're looking at. So it's, it really depends on my cow, accurate. You have much difference you care about.
I would say in general, like at the point where you care about difference between like out of 0.01 and 0.02, like the second decimal place, you want that to represent like 10 or 20 roads. You know like changing the class of that 10 or 20 rows. Then that's something you can be pretty confident of so like most of the time you know give them the data sizes we normally have 20 percent seems to work fine, but yeah. It's it's it's kind of a. It depends a lot on specifically what you're doing and what you care about, and it's not it's not a deep learning, specific question either you know so those who are interested in this kind of thing we're going to look into it. A lot more detail in our machine learning course, which will also be available online. Ok, so I did the same thing for the columns just to make sure that these aren't like super wide and I've got similar results and checked in and again found. They're kind of like 4 or 500 seem to be about the average size so based on all of that, I kind of thought. Ok, this looks like a pretty normal kind of image data set that I can probably use pretty normal kinds of models on. I was also particularly encouraged to see that when I looked at the that the dog like takes up most of the frame right, so I'm not too worried about like cropping problems.
 You know maybe zooming in a lot more or something like a medical imaging that happens a lot like often the tumor or the cell. Whatever is like one tiny piece and there's much more complex, so yeah based on all that and this morning I kind of
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Thought like okay, this looks pretty standard, so I I went ahead and created a little function called get data that basically had my normal two lines of code in it, but I made it so I could passed in a size and a batch size. The reason for this is that when I start working with new data set, I want everything to go super fast, and so, if I use small images, it's going to go super fast, so I actually started out with size equals 64 just to create some super small Images that just go like a second to run through and see how it later on, I started using some big images and some and some also some bigger architectures, at which point I started running out of GPU memory. So I started getting these errors saying CUDA out of memory error when you get a CUDA out of memory error. The first thing you need to do is to go kernel restart. Once you get a code, an out of memory error on your GPU. You can't really recover from it right. Doesn't matter what you do, you know you have to go restart and once I've restarted. I then just changed my batch size to something smaller. So when you call create your data object, you can pass in a batch size parameter. Okay and like i normally use 64 until i hit something that says out of memory and then i'll just have it, and if i still get out of memory I was hobbit again, okay.
So that's where I created this to allow me to like start making my size as bigger as I looked into it more and you know, as I started running out of memory to decrease my batch size. So at this point you know I went through this. A couple of iterations, but I basically found everything, was working fine. So once it's working fine set size 2 to 24 and I created my you know: pre-compute equals true first time I did that it took a minute to create the precomputed activations and then it ran through this in about 4 or 5 seconds, and you can see. I was getting eighty-three percent accuracy. Now. Remember, accuracy means it's it's exactly right, and so it's predicting out of a hundred and twenty categories. It's predicting exactly right. So when you see something with two classes, is you know 80 % accurate versus something with 120 classes? Is 80 % accurate, they're, very different levels? You know so when I saw like eighty-three percent accuracy with just a pre computed, classify and OData augmentation, though I'm freezing anything else across 120 classes of the oh. This looks good right, so um, then I just kind of kept going throughout at all standard process right. So then I turn precompute off. Okay and cycle length equals one, and I started doing a few more cycles. Few more epochs so remember an epoch is one pass through the data and a cycle is, however, many epochs, you said is in a cycle. It's one, it's the learning rate going from the top that you asked for all the way down.
So since here cycle length equals one a cycle in an epoch at the same okay, so I did. I tried a few epochs. I did actually do the learning rate finder and I found one in a two again looked. Fine. It often looks fine and I found it kind of kept improving, so
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
I tried five epochs and I found my accuracy getting better so then I saved that and I tried something which we haven't looked at before, but it's kind of cool. If you train something on a smaller size, you can then actually call learned, set data and pass in a larger size data set and that's gon na take your model. However, it's trained so far and it's going to let you can in you to train on on larger images, and I tell you something amazing. This actually is another way you can get state-of-the-art results and I've never seen this written in any paper or discussed anywhere. As far as I know, this is a new insight. Basically, I've got a pre trained model which, in this case I've trained a few epochs with the size of 224 by 224, and I'm now going to do a few more air pops with the size of 299. By 299, now I've gotten very little data cut out by deep learning standards, only about 10,000 images right so with a 224 by 224. I kind of built this these final layers to try to find things that work well to 24, but to 24. But I go to 299 by 299. I basically, if I over fit before I'm definitely not going to over fit now might have changed the size of my images, they're kind of like totally different but like conceptually they're, still picked. The same kinds of pictures are the same kinds of things, so I found this trick of like starting training on small images for a few a box and then switching to bigger images and continuing training is an amazingly effective way to avoid overfitting.
And it's like it's. So easy and so obvious, I don't understand why it's never been written about before. Maybe it's in some paper somewhere and I haven't found it, but it's I haven't seen it. Would it be possible to do the same thing on using? Let's take a resort our disposal to feed a different size yeah, I think so like as long as you use one of these more modern architectures, what we call fully convolutional architectures, which means not vgg and you'll, see we don't use vgg in this course, because it Doesn't have this property, but most of the architectures developed in the last couple of years can handle pretty much arbitrary sizes yeah be worth trying yeah. I think it ought to work okay, so I call get data again. Remember get data is the just a little function that I created back up here. Right get data is just this little function. That's, oh, I just passed a different size to it, and so I call freeze just to make sure that, but everything so the last layer is frozen. I mean it actually already was at its point that really doing a thing, and you can see now with free compute off I've now got that data augmentation working. So I kind of run a few more a pox, and what I notice here is that the loss to my training set and the loss of my validation set, my validation set loss is a lot lower than my training set. This is still just training. The last layer, so what this is telling me is I'm under fitting right and so from under fitting.
It means this cycle length equals one is too short, it means it's like finding something better popped with popping out and it's like never getting a chance to zoom. In properly so then I'd set cycle.
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Mod equals two to give it more time so, like the first time is one epoch. The second one is two epochs. The third one is for epochs and you can see now. The validation, train and training are about the same okay. So that's kind of thinking yeah. This is this is about the right track, and so then I tried using test time augmentation to see if that gets any better still didn't, actually help a hell of a lot just a tiny bit and just kind of at this point I think here this is Nearly done so, I just did it like. You know one more cycle of two to see if it got any better and it did get a little bit better and then I'm like okay, that looks pretty good. I've got a validation, set lost 0.199, and so your Lotus here, actually, you haven't tried unfreezing the reason why I was going to try to unfreezing and training more it didn't get any better, and so the reason for this clearly is that this data set is so Similar the image net that the training that convolutional layers actually doesn't help in the slightest and actually, when I loaded up into it, it turns out that this competition is actually using a subset of improve image net. So, that's okay, so that if we check this out point one nine, nine against the leaderboard, this is only a playground competition. So it's not like the best of here, but you know it's still interesting. It gets us somewhere around ten thrillers, okay and, in fact, we're competing against.
I noticed other the fastai student. This is a fastai student. These people up here, I know they actually posted that they cheated. They actually went. You downloaded the original images and train to that so, and this is why this is a playground, competition. They call it it's not it's not real right. You know it's just to allow us to try things out, but you can basically see out of two hundred and something people where you know we're getting some very good results without doing anything remotely interesting or clever, and we haven't even used the whole data set you're Going to use to eighty percent of it like to get a better result, I would go back and remove that validation set and just rerun the same steps and then submit that exact. Let's just use it under percent of the data. I have three questions. The first one is like that class in this case is very, it's not balanced instead, unbalanced, like it's, not totally balanced, but it's not bad right. It's like between sixty and a hundred like it's it's it's! It's not unbalanced enough that I would give it a second thought: okay, yeah! Let's get to that later in this course, and don't let me forget right, the short answer is that there was a recent list. The paper came out about two or three weeks ago on this, and it said the best way to deal with very unbalanced data sets is to basically make copies of the rare cases yeah. My second question is: I want to pin down a difference between creation.
He read was, and so you have these two options right. So when you beginning I did an optimization use that pre computed. It was true by not using layers right right, so it's not only they frozen their pre computed, so the data augmentation doesn't do anything at that point right before you outcries everything. What as examples you and IV only you only on freeze, so we're going to learn more about the details as we look into the the math and stuff in coming lessons. But basically what happened was we started with a pre trained network right which was kind of finding activations that had these kind of rich features and we were adding, then we add a couple of layers on the end of it, which which start out random, and so With fries equals, with with everything frozen and indeed with pre compute equals. True, all we're learning is told is those couple of layers that we've added and so with pre compute equals. True, we actually pretty calculate like how much does this image have something that looks like this? A ball one looks like this face and so forth, and therefore data augmentation doesn't do anything with pre compute equals true, because you know we're actually showing exactly the same activations. Each time we can then set pre compute equals false, which means it's still only training, those last two layers that we added it's still frozen, but data augmentations now working because it's actually going through and recalculating all of the activations from scratch and then, finally, when we Unfreeze, that's actually saying: okay now you can go ahead and change all of these earlier convolutional filters.
So well you just so the only reason to have pre compute equals true. Is it's just much faster? So it's like it is it's about.  There's no. Like companies like accuracy reason ever to use pre computed calls true. It's just a it's just a shortcut. It's also like quite handy. If you're like throwing together a quick model, you know it can take a few seconds to create my last question, which I think you answer is I don't like your suggestions to build a model. You have this aged yeah. What, if would you like? We just wanted one initial setting without these like checking after each I mean if you want it like. If your question is like, is there some shorter version of this? That's like a bit quicker and easier. I could like to lead a few things here. Okay, I think this is a kind of a minimal version to get you a very good result, which is like don't worry about pre compute equals true, because that's just saving a little bit of time. You know so so I still suggest use LR find at the start to find a good learning rate by default. Everything is frozen from the start, so then you can just go ahead and run two or three epochs or cyclic Nichols one unfreeze and then train the rest of the network with differential learning rates.
So it's basically three steps: learning rate, finder trained frozen network with cycle methods, one and then trained unfrozen network, with differential learning rates and cycle molecules too. So, like that's something you could turn into, I guess five or six lines of code at all. I think it's a question provide your own mix book by reusing the batch size. Does the only at better speed of training yeah pretty much so each batch and again we're going to see like all this stuff about precomputing batch sizes. We dig into the details of the algorithms it's going to make a lot more sense intuitively, but basically, if you're, showing it less images each time, then it's calculating the gradient with less images, which means it's less accurate, which means like knowing which direction to go and How far to go in that direction is less accurate, so, as you make the batch size, smaller you're, basically making it kind of more volatile, it's kind of like it kind of impacts, the optimal learning rate that you would need to use, but in practice, where only You know I generally find only dividing with the batch size by like 2 or 4. It doesn't seem to change things very much. Should I reveals the learning rate of quality me if you, if you change the batch size by much, you can rerun the learning rate.
Finder to see if it's changed, if I match, but it it I was in for only generally looking at like a power of 10, it probably is not going to change the it's, not that you can't, because plus back there, this is sort of a conceptual basic Questions we're going back to the previous night, where you should put in the thought behind sorry yeah. This is well one for conceptual, so a basic question: we've actually really slide where you should what the different layers were doing. Yes from this slide, I understand right. The meaning of sync, the third column relative to the fourth column, is that what you're interpreting what the layer is doing, based on what the image is actually yeah? So we're going to look at this in more detail, so these these gray ones basically say this is kind of what the filter looks like. So, on the first layer you can see exactly what the filter looks like, because the input to it of pixels right. So you can absolutely say, and remember we looked at what a convolutional kernel was like. Was that three by three thing? So this look like there's seven by seven kernels, you can say this is actually what it looks like, but later on it's combined. You know the the the input to it are themselves activations which are combinations of activations relation to activations.
So you can't draw it, but there's clever technique that I learned focus created which allowed them to say this is kind of what the filters tended to look like on average alright. So this is kind of what the photos look like and then here is specific examples of patches of image which activated that filter highly. So yet the pictures are the ones that I kind of find more useful because it tells you this kernel is kind of a mini cycle. We all find right. How do we know? That's it well, we'll come back well, we may come back to that. If not in this part in the next part, that probably a part two actually because this paper this paper uses to create these things, this paper uses something called a deconvolution which I'm pretty sure we won't do in this part, but we will do it in part. Two, so if you're interested check out the paper, it's it's in the notebook has a link to it: xyler in Fergus, it's a very clever technique and not terribly intuitive um right. So so you mentioned that it was good that the dog took up the full picture and it would have been a problem if it was kind of like off in one of the corners in really tiny. Well, what would you, what would you technique have been to try to make that work, something that we'll learn about in part two, but basically there's a technique that allows you to to kind of figure out, roughly which parts of an image and most likely to have The interesting things in them and then you can like crop out those bits if you're interested in learning about it.
We did cover it briefly in lesson, seven of part one, but I'm going to actually do it properly in part. Two of this course, because I didn't really cover it thoroughly enough - maybe we'll find time to have a quick look at it, but we'll see I know your Nets.
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Written some of the code that we need already. Ah, so once I have something like this notebook, that's basically working, I can immediately make it better by doing two things. Assuming that the size image I was using is smaller than the average size of the image that we've been given. I can increase the size and, as I showed before, with the dog breeds, you can actually increase it during training. The other thing I can do is to create is to use a better architecture. Now an architect we're going to talk a lot in this course about architectures, but basically there are different ways of putting together like what size convolutional filters and how are they connected to each other and so forth, and different architectures have different like numbers of layers and Sizes of kernels and number of filters and so forth, and so there are some the one that we've been using. Resnet 34 is a great starting point and often a good finishing point, because it's like it's pretty. It doesn't have too many parameters. Often it works pretty. Well, with small amounts of data as we've seen and so forth, but there's actually an architecture that I really like called not res net but res next, which was actually the second-place winner in last year's image: net competition and like ResNet, you can put a number after The res next to say like how big it is and like my next step after resume 34 is always res next 50 now you'll find res next 50 takes like can take like twice as long as ResNet 34 that can take like 2 to 4 times as Much memory as retina 34, so what I wanted to do was I wanted to rerun that previous notebook with res next and increasing the image size to turn on a node.
So here I just said: architecture equals res next, 50 size equals 299, and then I found that I had to take the batch size all the way back to 28 to get it to fit. My GPU is 11 gig. If you're using AWS or cresol. I think they're, like 12 gigs, they might be a bit higher, but this is what I found I had to do so then I this is literally a copy of the previous notebook, so you can actually go file, make a copy right and then rerun it with With these different parameters, and so I deleted some of the pros and some of the expiratory stuff to see you know basically, I said everything else is the same, all the same steps as before. There's my in fact, you can kind of see what this minimum service desk looks like. I didn't need to worry about learning rate finder, so I just left it as is so transforms. Data equals loan equals bit pre computed false feet with cycle integrals. One and freeze differential learning rates bits and more - and you can see here - I didn't do the cycle mop thing, because I found like now that I'm using a bigger architecture, it's got more parameters. It was overfitting pretty quickly so, rather than like cycle length equals one. Never finding the right spot, it actually did find the right spot, and if I used longer cycle legs, I found that my validation error was higher than my training error. It was over there so check us out, though, by using these, you know three steps. I got plus TTA 99.75. So what does that mean? That means I have one incorrect dog for incorrect cats and when we look at the pictures of them, my incorrect dog has a cat.
Now this one is not a either. This one is not either so. I've actually got one mistake and then my incorrect dog is teeth right. So like we're at a point where we're now able to train a classifier, that's so good that it has like basically one's dead right, and so when people say like we have superhuman image performance. Now this is kind of what they're talking about right. So did you actually, when I looked at the dog breed one I did this morning I was like it was. It was getting the dog breeds much better than I ever could so like hits this. This is what we can get to if you use a really modern architect like redneck, and this suddenly took out a tall way and remember, don't like 20 minutes to Train, so that's kind of where we're up to so. If you want to do
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Satellite imagery instead right, then it's the same thing and in fact the the planet. Satellite data sets already oh and Chris or if you're, using Chris, or you can jump straight there right and I just went into this data stash planet and I can do exactly the same thing right. I can image classifier from CSV right and you can see these three lines are actually exactly the same as my dog breed lines. You know how big, how many lines are in the file grab. My validation indexes this get data, as you can see, it's identical, except I've changed side on to top down the satellite images about top down, so I can fit them vertically and they still make sense right, and so you can see here I'm doing this trick round. Back to size, equals 64 and train a little bit first learning rate find on right and, interestingly, in this case you can see it. I want really high learning rates. I don't know what it is about this particular data set. This is true, but it's clearly I can use super high learning rate, so I use a lot here at a point too, and so I've trained for a while differential learning rates right and so remember - I said like if the data sets very different to image net. I probably want to train those middle layers a lot more, so I'm using divided by three rather than divided by ten, all right, the other than that is the same thing cycle, Nauticals all right and then I just kind of keep an eye on it.
So you can actually plot the loss if you go and learned up shared a plot loss. You can see here that here's the first cycle is. The second cycle is the third cycle right, so you can see, it gets better. Pops out gets better pops out if better pops out and each time it finds something better than the last time then set the size up to 128 and just repeat exactly the last few steps and then set up to 256. Repeat the last two steps and then do TTA and if you submit this - and this gets about 30th place in this competition, so these basic steps work super well this this thing where I went all the way back to a size of 64. I wouldn't do that. If I was doing like dogs and cats or dog breeds because like this is so small that, if if the thing I was working on is very similar to imagenet, I would kind of destroy those imagenet weights like 64 by 64, is so small. But in this case the satellite imagery data, it's so different to imagenet um, you know I really found that it worked pretty well start right back to these tiny images. It really helped me to avoid overfitting and interestingly, using this kind of approach. I actually found that even with using only 128 by 128, I was getting like much better cackled results than really everybody on the leader board and when I say 30th place, this is a very recent competition right, and so I find like in the last year, like A lot of people have got a lot better at computer vision, and so the people in the top 50 in this competition were generally ensemble in dozens of models. Lots of people on a team, lots of pre-processing, specific satellite data and so forth so like to be able to get xxx using this totally standard technique is pretty cool.
Alright. So now that we've got to this point right, we've got through two lessons. If you're still here, then
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Hopefully, you're thinking, okay, this is actually pretty useful. I want to do more, in which case Crestle might not be where you want to stay the issues with Crestle. I mean it's, it's it's pretty handy, it's pretty cheap and something we haven't talked about. Much is paper. Space is another great choice. By the way paper space are short, they're going to be releasing kress or like instant Drupal notebooks, unfortunately, they're not ready quite yet, but they do have an ability to. Basically, they have the best price performance relationship right now and they you can SSH into them and use them so they're also a great choice, and, probably by the time this, the MOOC will probably have a separate lesson showing you how to set up set up paper Space because there they're likely to be a great option, but at some point you're probably going to want to look at AWS a couple of reasons why the first is, as you all know, by now, amazon have been kind enough to donate about $ 200,000 worth of Compute time to this course, so I want to say thank you very much to Amazon. We've all been given credit, so everybody this year. So thanks very much hey, don't worry, we're so sorry, you're sure in the MOOC. We didn't get it for you, but everybody here is like AWS credits for everybody, so um, but you can get even if you're, not here in person, you can get AWS credits from lots of places. Github has a student pack, Google for github student pack.
That's like 150 bucks worth of credits. Aws educate can get credits these our office students, so there's lots of places you can get started on AWS, pretty much everybody everybody. A lot of the people that you might work with will be using AWS, because it's like super flexible right now AWS has the fastest available GPUs. You can get in the cloud. They're p3s they're kind of expensive at three bucks an hour. But if you've got like a model where you've done all the steps before you're thinking, this is looking pretty good. You know for 6 bucks you could get a p3 for 2 hours and run turbo speed right um. We didn't start with AWS because well hey it's like twice as expensive as Chris Hall for the cheapest GPU and being a Texan setup right. But I wanted to kind of go through and show you how to get your AWS setup and so we're going to be going slightly over time to do that. But I want to show you a very quick place. I feel prettier if you have to, but I want to show you very quickly how you can get your AWS setup right from scratch. So, basically, you have to go to consult on AWS, but amazon.com and it'll take you to the console right, and so you can follow along on the video with this quickly from here. You have to go to AC. This is where you set up your instances and so from ec2. You need to do what's called launching an instance, so launching an instance means you're, basically creating a computer right now creating a computer on Amazon. So I say launch instance, and what we've done is we've created a fastai.
It's got an amo and ami is like a template for how your computer's going to begin. So if you've got a community, a Mis and type in fastai you'll see that there's one there called fastai part: 1 version 2 for the p2. Ok, so I'm going to select that and then we need to say what kind of computer do you want, and so I can say I want a GPU compute computer and then I can say I want a p2 x large. This is the cheapest reasonably effective for deep learning instance type they have and then I can say launch and then I can say launch, and so at this point they asked you to choose a key pair right now. If you don't have a key pair, you have to create one right so to create a key pair. You need to open your terminal. If you don't have a terminal, if you've got a Mac or Linux box, you've definitely got one if you've got Windows. Hopefully, you've got Ubuntu. If you don't already have Ubuntu setup, you can go to the Windows, Store and click on Ubuntu right, we'll get it from the Windows Store. So from there. You basically go SSH caged in and that will create like a special password for your computer, to be able to log in to Amazon, and then you just hit enter three times. Okay and that's going to create for you, your key. You can use to get into Amazon alright. So then, what I do is I copy that key somewhere that I know where it is so it'll be in the dot SSH folder, it's called IDRs a dub, and so I'm going to copy it to my hard drive.
So if you're in a macro and Linux it'll already be in an easy to find place, it'll be in your SSH folder that in documents so from there back in AWS, you have to tell it that you've created this key. So you can go to key pairs and you say import key pair and you just browse to that file that you just created there. It is, I say, import okay, so if you've ever used SSH before you've already got the key pair, you don't have to do those depths if you've used AWS before you've already imported it. You don't have to do that step. Maybe haven't done any of those things. You have to do both steps so now I can go ahead and launch my instance community. I am eyes search last day I select launch, and so now it asks me what's where's your key pair and I can choose that one that I just grabbed okay. So this is going to go ahead and create a new computer for me to log into, and you can see here it says the following have been initiated, and so, if I click on that, it'll show me this new computer that I've created okay, so it'll be Able to log into it I need to know its IP address, so here it is the IP address there. Okay, so I can copy that and that's the IP address of my computer so to get to this computer, I need to SSH to it so SSH into a computer means connecting to that computer so that it's like you're typing in that computer.
So I type SSH and they username, for this instance - is always Ubuntu right and then I can paste in that IP address and then there's one more thing I have to do, which is. I have to connect up the jupiter notebook on that instance to the jupiter notebook on my machine and so to do that, there's just a particular flag that i said. Okay, we can talk about it on the forums as to exactly what it does, but you just type l.a today date: localhost, 8, 8. 8. 8. Ok, so like once, you've done it once you can like save that as an alias and type. In the same thing, every time, so we can check here we can see it says that it's running so we should be able to now hit enter first time ever which sit reconnect to it. It does checks. This is okay, I'll, say yes, and then that goes ahead and SSH is in so this ami is all set up for you. Alright, so you'll find that the very first time you log in it takes a few extra seconds because it just kind of is getting everything set up. But once it's logged in you'll see there that there's a directory called fastai and the fastai directory contains our fastai repo that contains all the notebooks or the code, etc.  All right. First thing you do when you get in is to make sure it's updated, so you just go git pull right and that updates to make sure that your repo is the same as the most recent video. And so, as you can see there, we go. Let's make sure it's got all the most recent code.
The second thing you should do is type Condor and update. You can just do this, maybe once a month or so, and that makes sure that the libraries there are all the most recent libraries I'm not going to run that. So it takes a couple of minutes. Okay and then the last step is to type particular notebook. Okay, so this is going to go ahead and launch the triplet notebook server on this machine again, the first time I do it the first time you do everything on AWS. It just takes like a minute or two and then once you've done it in the future. We just as fast as running it locally. Basically, okay, so you can see it's going ahead and firing out the notebook, and so what's going to happen. Is that because, when we SSH into it, we said to both connect our notebook port to the remote notebook port. We're just going to be able to use this locally, so I see he says here copy paste this URL, so I'm going to grab that URL and I'm going to paste it into my browser and that's it okay. So this notebook is now actually not running on my machine. It's actually running on AWS, okay, using the AWS GPU. It's got a lot of memory, it's not the fastest around, but it's not terrible. You can always fire up a p3. If you want something. That's super fast. This is costing me ninety cents a minute okay. So when you're finished, please don't forget to shut it down right so to shut it down. You can right-click on it and say, instance. Date. Stop. Okay, we've got five hundred bucks of credit.
Assuming that you put your code down in the spreadsheet one thing I forgot to do the first time. I showed you this by the way I said, make sure you choose a p2. The second time I went through, I didn't choose p2 by mistake. So, just don't forget: choose gpq compute P. Do you have a question my Bernice it's an hour. Thank you, 90 cents an hour. It also costs, like I don't know three or four bucks a month for the storage as well. Thanks for checking that all right see you next week, sorry we're a bit over