fast.ai: Intro to Machine Learning 1 (v1) (2018)
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Well, welcome back to machine learning, one of the most exciting things this week, almost certainly the most exciting. The thing this week is that fastai is now on Pitt, so you can pip install fastai, and so thank you to Prince and perk to creme for making that happen to USF students who had never published a hit package before - and this is one of The harder ones to publish because it's got a lot of dependencies, so it's you know, probably still easiest just to do the Condor end of update thing, but a couple of places that it would be handy instead to pip install fastai would be well. Obviously, if you're working outside of the the repo in the notebooks, then this gives you access to fastai everywhere also, I believe they submitted a pull request to CAG or to try and get it added to the Capitol kernels, so hopefully you'll be able to use It on Kapil kernels soon and yeah. You can use it at your work or whatever else. So that's that's exciting. I mean I I'm not gon na say it's like officially released. Yet you know it's still very early, obviously, and we're still you're helping, add documentation and all that kind of stuff. But it's great that that's now there a couple of cool kernels from USF students this week thought of highlight two that were both from the text: normalization competition, which was about trying to take text which was written out. You know, wrote a standard English text.
They also had one per Russian and you're trying to kind of identify things that could be like first. Second, third and say like that's a cardinal number, or this is a phone number or whatever, and I did a quick little bit of searching and I saw that there had been some attempts in academia to use deep learning for this. But they hadn't managed to make much progress and actually noticed so up here is Colonel here, which gets 0.99 two on the leader board, which i think is like top is yeah it's kind of entirely heuristic and it's a great example of kind of feature engineering. That's in this case, the whole thing is basically entirely feature engineering, so it's basically looking through and using most of regular expressions to figure out for each token. What is it you know, and I think she's done a great job here, kind of laying it all out. Clearly, as to what all the different pieces are and how they all fit together - and she mentioned that she's - maybe hoping to turn this into a library which I think would be great right - you know you could use this to grab a piece of text and pull Out what are all the pieces in it? It's the kind of thing that the the natural language can, like natural language processing community hopes to be able to do without, like lots of hand, written code like this, but for now this is I'll, be interesting to see. Like what the winners turn out to have done, but I haven't seen machine learning being used really to do this, particularly well, perhaps the best approaches or ones which combine this kind of feature engineering along with some machine learning.
But I think this is a great example of effective feature engineering, and this is a another USF student who has done much. The same thing got a similar kind of score, but used used her own different set of rules. Again this is gets you. It would get you a good leader board position with these as well. So I thought that was interesting to see examples of some of our students entering a competition and getting kind of top 20 ish results by you know basically just handwritten heuristics, and this is where, for example, computer vision was six years ago. Still, basically, all the best approaches was a whole lot of like carefully handwritten heuristics, often combined with some simple machine learning, and so I think, over time you know the field is kind of definitely trying to move towards automating much more of this, and actually, interestingly, very Interestingly, in the safe driver, diction competition was just finished. One of the Netflix Prize winners won this competition and he invented a new algorithm for dealing with structured data which basically doesn't require any feature engineering at all. So he came first place using nothing but five. Deep learning models and one gradient, boosting machine and his his basic approach was very similar to what we've been learning in this class so far.
And what we'll be learning also tomorrow, which is using fully connected neural network somewhere and one hot encoding and specifically embedding which we'll learn about? But he had a very clever technique, which was. There was a lot of data in this competition which was unlabeled. So, in other words, where they didn't know whether that driver would go under claim or not or or whatever so unlabeled data. So when you've got some labeled in some unlabeled data, we call that semi-supervised learning and in real life, most learning is semi-supervised. Learning like in real life, normally you have some things that are labeled and some things that are unlabeled, so this is kind of the most practically useful kind of learning and then structured data is it's the most common kind of data that companies deal with day to Day so the fact that this competition was a semi-supervised structured data, competition made it incredibly practically useful, and so what his technique for winning this was was to through data augmentation, which those of you doing. The deep learning course have learned about, which is basically the idea. Like if you had pictures, you would like flip them horizontally or rotate them a bit later. Orientation means creating new data examples which are kind of slightly different versions of ones. You already have and the way he did it was for each row from the data.
He would like at random replaced 15 % of the variables with a different row, so each row now would represent like a mix of like 80 %, 85 % of the original row, the 15 % randomly selected from a different, and so this was a way of Like randomly changing the data a little bit and then he used something called an autoencoder which we will probably won't study into a part too, with a deep learning course. But the basic idea of an autoencoder is your dependent variable is the same as your independent variable. So, in other words, you try to predict your input, which obviously is trivial. If you're allowed to like, like you know the identity transporting, for example, Rivoli predicts the input, but the trick with an autoencoder is to have less activations in at least one of your layers than your input right. So if your input was like a hundred dimensional vector - and you put it through a 100 by 10 matrix, okay, create ten activations and then have to recreate the original 100 long vector from that, then you've basically come. You had to have compressed it effectively, and so it turns out that that kind of neural network you know, is forced to find correlations and features and interesting relationships in the data, even when it's not labeled, so he used that rather than doing any, he didn't do Any hand engineering he just used an autoencoder.
So you know these are some interesting kind of directions that if you keep going with your machine learning studies, you know, particularly if you do a part two with a deep learning course next year, your your lone about - and you can kind of see how feature engineering Is going away - and this was just yeah an hour ago - so this is very recent using to eat, but it's one of this is one of the most important breakthroughs I've seen in a long time. Okay, so we were working
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Through a simple logistic regression trained with SGD for MS and here's, the summary of where we got to, we have nearly built a module, a model module and a training loop from scratch. And we were going to kind of try and finish that and after we finished that and then going to go through this entire notebook backwards right so having gone like top to bottom, but I'm going to go back through bottom to top okay. So you know this was that little handwritten and end module class. We created, we defined our loss, we defined our learning rate and we defined our optimizer, and this is the thing that we're going to try and write by hand in a moment so that stuff that and that we're stealing with from pytorch, but that we've written Ourselves and this week written us all, so the basic idea was we're going to go through some number of epochs. So let's go through one epoch: okay and we're going to keep track of how much for each mini batch. What was the loss so that we can report it at the end, we're going to turn our training data loader into an iterator so that we can loop through it live through every mini batch, and so now we can go and go ahead and say for tensor. In the length of the data loader and then we can call next to grab the next independent variables and the dependent variables from our data loader from that iterator. Okay, so then remember.
We can then pass the X tensor into our model by calling the model as if it was a function, but first of all we have to turn into a variable. Last week we were typing variable blog CUDA, to turn it into a variable. A shorthand for that is just the capital V. Now it's a capital T for a tensor capital B for a fever variable, that's just a shortcut in fastai, okay, so that returns our predictions. And so the next thing we needed was to calculate our loss because we can't calculate the derivatives of the loss of. U and calculate the loss, so the loss takes the predictions and the actuals okay, so the actuals again are the the Y tensor and again we have to turn that into a variable now. Can anybody remind me what a variable is and why we would want to use a variable here? I think once you turn it to variable, then it tracks it. So then you can do a backward on that. So you can yeah what sorry, when you turned a variable it it contract like this process of, like you know, as you add the function as the function starting earlier than each other, they can track it and I may need to backward on it back propagates and Those days yeah right so
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Right so a variable keeps track of all of the steps to get computed and so there's actually a fantastic tutorial on the pytorch website. So on the pipe torch website. There's a tutorial section and there's a tutorial there about Auto grad. Auto grad is the name of the automatic differentiation package that comes with a torch and it's a it's. An implementation of automatic differentiation, and so the variable class is really the key. The key class here, because that's the thing that makes turns a tensor into something where we can keep track of its gradients. So basically here they show how to create their variable, do an operation to a variable, and then you can go back and actually look at the grand function, which is the the function that it's keeping track of. Basically to calculate the gradient right. So as we do more and more operations to this very variable and the variable calculated from that variable, it keeps keeping track of it so later on. We can go dot, backward and then print grad and find out the gradient right, and so you notice we never defined the gradient. We just defined it as being X, plus 2 squared times 3 whatever, and it can calculate the gradient okay. So that's why we need to turn that into a variable, so L is now a variable containing the loss, so it contains a single number for this mini batch, which is the loss for this mini batch. But it's not just a number.
It's a it's a number as a variable, so it's a number that knows how it was calculated all right, so we're going to append that loss to our array just so we can get the average of it later. Basically, and now we're going to calculate the gradient. So L drop backward is a thing that says calculate the gradient. So remember when you recall the the network, it's actually calling our forward function. So that's like cap go through it forward and then backward is like using the chain rule to calculate the gradients backwards. Okay and then this is the thing we're about to write, which is update, the weights based on the gradients and the learning rate. Okay, zero grad will explain when we write this out by hand okay and so then, at the end, we can turn our validation data loader into an iterator, and we can then go through its length grabbing each X and way out of that and asking for the Score which we defined up here to be equal, to which thing did you predict, which thing was actual and so check, whether they're equal right and then the main of that is going to be our accuracy? Okay, could you pass that over to Chen XI? What's the advantage that you found converted into iterator resident like used normal and on a Python loop or we're using a normal Python loop? So it's still them. This is a normal Python loop. So the question really is like compared to what right so, like the alternative.
Perhaps nothing here could be like we could use like a something like a list with an index. Oh okay, so you know the problem. There is the we want. As a few things, I mean one key one: is we want each time we grab a new mini batch, we want to be random, we want
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
A different different shuffled thing, so this you can actually kind of iterate from forever. You know you can look through it as many times as you like. So this is kind of idea. It's called different things in different languages, but a lot of languages. A lot like stream processing and it's this basic idea that, rather than saying I want the third thing or the ninth thing is just like. I want the next thing right, it's great for, like network programming, it's like grab the next thing from the network. It's great for UI programming. It's like we have the next event. Where there's somebody clicked a button. It also turns out to be great for this kind of numeric programming's. It's like. I just want the next batch of data. It means that the data like can be kind of arbitrarily long as we're just grabbing one piece at a time yeah. So you know I mean, and also I guess, the short answer is because it's our PI approach works. If I torch that light torches, data loaders are designed to be poured in this way and then so Python has this concept of a generator which is like an and and just a different type of generator. I want to physical gon na, be a snake generator or a computer of generator. Okay, a generator is a way that you can create a function that, as it says, behaves like an iterator so, like python, has recognized that this stream processing approach to programming is like super handy and helpful and supports it everywhere.
So, basically, anywhere that you user for in loop any way you use a list comprehension, those things can always be generators or iterators. So by programming this way we just get a lot of flexibility. I guess: does that sound about right, Terrence you're, the programming language expert. Did you do you want to grab that box, so you can hear so. Terrence actually does programming languages for a living, so we should ask him yeah I mean the short answer is what you said. You might say something about space, but in this case that all that data has to be in memory because we've got no doesn't have to be a memory so that most of the time, if we could pull a mini batch from something that most of the time With pipe torch, the mini batch will be read from like separate images spread over your disk on demand. So most of the time it's not in memory, but in in general. You want to keep his little in memory as possible at a time, and so the idea of stream processing also is great, because you can do compositions, you can pipe the data to a different machine. You can yeah yeah, the composition is great. You can grab scrap the next thing from here and then send it off to the next stream, which you can then grab it and do something else, which you guys all recognize, of course, in the command-line pipes and redirection. Yes, okay, thanks Terrence, it's a benefit of working with people that actually know what they're talking about all right.
So, let's know take that and get rid of the optimizer okay. So the only thing that we're going to be left with is the negative log likelihood loss function which we could also replace. Actually, we have a implementation of that from scratch that you know wrote in the in the notebooks. So it's only one line of code, as we learned earlier, you can do it with a single if statement. Okay, so I don't know why I was so lazy is to include this. So what we're going to do is we're gon na again grab this module that we've written ourselves the logistic regression module we're going to have one epoch again, we're going to loop through each thing in an iterator again they're going to grab our independent and dependent variable For the mini batch again pass it into our network again calculate the loss, so this is all the same as before, but now we're going to get rid of this optimizer dot step and we're going to do it by hand. So the basic trick is, as I mentioned, we're not going to do the calculus by hand, so we call L drop backward to calculate the gradients automatically and that's going to fairly in our weight matrix. So do you remember when we created our let's go back and look at the code for here's that module we built so the weight matrix for the for the linear layer weights, recall l1 w and for the BIOS record l1 B? All right! So though they were the attributes we created, so I've just put them into things called W and B just to save some typing.
Basically so W is our weights B is our biases, and so the weights remember the weights are a variable and to get the tensor out of the variable. We have to use data all right, so we want to update the actual tensor. That's in this variable, so we say weights data minus equals, so we want to go in the opposite direction to the gradient. The gradient tells us which wears up. We want to go down whatever is currently in the gradients times the learning rate. So that is the formula for gradient descent all right. So, as you can see it's it's like as as easier thing as you can possibly imagine. It's like literally update the weights to be equal to be equal to whatever they are now minus the greater the gradients times our learning rate and do the same thing for the bias. Anybody have any questions about that step in terms of like why we do it or how it did. You have a question about that that step, but when we do the next job deal the next here. Yes, yes, so when is the end of the loop? How do you grab the next element, so this is going through each h-index in range of length. So is this going 0, 1. 2. 3. At the end of this loop, it's going to print out the mean of the validation set, go back to the start of the epoch, at which point it's going to recreate a new, a new iterator, okay.
So, basically, behind the scenes in Python, when you call it a on on this, it basically tells it like reset its state to create a new iterator and, if you're interested in how that works, the the code is all you know available for you to look at So we could look at like MD trained. Your is a fastai dot data setup model data loader. So we could like take a look at the code of that. So we could take a look at the code of that and see exactly how it's being built right, and so you can see here that here's the next function right, which basically is keeping track of how many times it's been through in this self dot. I and here's the Edith function, which is the thing that gets pretty cold when you, when you create a new iterator, and you can see it's basically passing it off to something else, which is a type data loader, and then you can check out data loader if You're interested to see how that's implemented as well um, so the data loader that we wrote basically uses multi-threading to allow it to have multiple of these going on. At the same time, it's actually a great. It's really simple. It's like it's only about a screen full of code, so if you're interested in simple Modi threaded programming, it's a good thing to look at okay. Now, oh yes, why? How do you wrap this in four epoch in range? One since that'll only run once because in real life, we would normally be running multiple so like in this case, because it's a linear model, it actually basically trains to as good as it's going to get in one he park.
So if I type three here it actually, it actually won't really improve after the first epoch. Much at all, as you can see right, but when we go back up to the top we're going to look at some slightly deeper and more interesting versions which will take more a box. So you know if I was turning this into a into a function. You know I'd be going like you know: death, train model and one of the things you would pass you and is like number of epochs kind of thing. Okay, great so one thing to remember is that when you're, you know creating these neural network layers and remember like this is just as pipe Rogers concerned. This is just a it's an end up module it could be a week could be using it as a layer who could be using to a function. We could be using it as a neural net. Pipe torch doesn't think of those as different things right. So this could be a layer inside some other network. Okay. So how do gradients work so if you've got a layer which remember is just a bunch of we can think of it, basically as its activations right or some activations, that get computed through some other nonlinear activation function or through some linear function and from that layer. We it's very likely that we're then, like, let's say putting it through a matrix product right to create some new layer, and so each one of these.
So if we were to grab like one of these activations right is actually going to be used to calculate every one of these outputs right. And so, if you want to calculate the the derivative, you have to know how this weight matrix, impacts, that output and that output and that output and that output right and then you have to add all of those together to find. Like the total impact of this, you know across all of its outputs, and so that's why in pytorch, you have to tell it when to set the gradients to zero right, because the idea is that you know you could be like having lots of different loss Functions or lots of different outputs in your next activate set of activations or whatever, all adding up, increasing or decreasing your gradients right, so you basically have to say: okay, this is a new calculation reset. Okay, so here is where we do that so before we do LDAP backward, we say reset, okay, so let's take our weights. Let's take the gradients, let's take the ten so that they point to and then zero underscore. Does anybody remember from last week what underscore does as a suffix in pytorched yeah? I have to read the language, but basically it changes. I put into place that language is in place yeah exactly so. It sounds like a minor technicality, but it's super useful to remember.
Every function, pretty much has an underscore version suffix, which does it in place yeah, so normally zero returns, a tensor of zeros of a particular size, so zero underscore means replace the contents of this.  So that's like SGD from scratch and if I get rid of my menu bar, we can officially say it fits within a screen yeah. So of course we haven't got our definition of logistic regression. Here, that's another half the screen, but basically there's there's not much to it. Yes, sir, so later on, we have to do this more the gradient. Is it because you might find like a wrong minima local minima? That way, we have to kick it out and that's we have to do multiple times when the surface is getting more common. Why do you need multiple epochs? Is that your question? Well, I mean a simple way to answer. That would be, let's say our learning rate was tiny right. Then it's just not gon na get very far right. There's nothing that says going through one epoch is enough to get you all the way there so then it'd be like okay. Well, let's increase our learning rate and it's like yeah sure will increase our learning rate but who's to say that the highest learning rate that learned stabili is is enough to learn this as well as it can be learned and for most data sets for most architectures. One epoch is very rarely enough to get you to the best result. You can get too.
You know, linear models are just they're very nicely behaved, you know, so you can often use higher learning rates and learn them well quickly. Also, they they don't. You can't like generally get as good at accuracy, so there's not as far to take them either so yeah doing one epoch is going to be the rarity all right. So let's go backwards so going backwards. We're basically going to say all right. Let's not write those two lines again and again again: let's not write those three lines again and again and again: let's have somebody do that for us right. So that's like that's. The only difference between that version in this version is rather than saying, dot zero ourselves. Rather than saying gradient times, Amara ourselves, these are wrapped up for us. Okay, there is another wrinkle here, which is this approach to updating the the weights is actually pretty inefficient. It doesn't take advantage of momentum and curvature, and so in the do course we learn about how to do momentum from scratch as well. Okay, so if we actually just use plain old SGD, then you'll see that this Alone's much slower so now that I've typed just plain old SGD here. This is now literally doing exactly the same thing as our slow version, so I have to increase the learning rate. Okay, there we go so this. This is now the same as the one we wrote by hand. So then all right, , let's do a little bit more stuff automatically.
Let's not you know, given that every time we train something we have to loop through epoch, flip through batch do forward get the loss zero. The gradient do backward, do a step of the optimizer. Let's put all that in a function, okay and that function is called fit. Alright, there it is okay, so let's take a look at fit fit go through each epoch go through each batch. Do one step keep track of the loss and, at the end, calculate the validation all right, and so then step so if you're interested in looking at this, this stuff's all inside fastai dot model, and so here is dead right, zero. The gradients calculate the loss. Remember pipe torch tends to call it criterion rather than loss, all right. Do it backward and then there's something else we haven't learned here, but we do learn the deep learning course which is gradient reading. So you can ignore that alright, so you can see now, like all the stuff that we've learnt when you look inside the actual frameworks. That's the code, you see, okay, so that's what fit does and so then the next step would be like okay. Well, this idea of like having some weights and a bias and doing a matrix product. In addition, let's put that in a function this thing of doing the log softmax, let's put that in a
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Function and then the very idea of like first doing this and then doing that this idea of like chaining functions together, let's put that into a function and that finally gets us to that. Okay, so sequential simply means through this function. Take the result. Send it to this function, etc, right and linear means create the weight matrix, create the biases okay. So that's that's yeah right, so we can. Then you know, as we started to talk about like turn this into any deep neural network. By saying you know, rather than sending this straight off into ten activations, let's, let's put it into say: 100 activations. We could pick whatever one number. We, like put it through a RAL you to make it nonlinear, put it through another linear layer, another rally ER and then our final output with our final activation function right, and so this is now a deep network. So we could fit that and this time now, because it's like deeper I'm actually going to run a few more a pox right and you can see the accuracy increasing okay. So if you try and increase the learning rate here, it's like 0.1 further. It actually starts to become unstable. Now I'll show you a trick. This is called learning rate annealing and the trick is this: when you're trying to fit to a function right, you've been taking a few steps step step step as you get close to the middle like get close to the bottom, your steps probably want to become smaller Right otherwise, what tends to happen? Is you start finding you're doing this right, and so you can actually see it here right.
It got 93 94 and a bit 94 694 eight, like it's kind of starting to flatten. Now, right now, that could be because it's kind of done as well as it can, or it could be, that it's going to growing backwards and forwards. So what does a good idea, as is later on in training, is to decrease your learning rate and to take smaller steps? Okay, that's called learning rate annealing, so there's a function in fastai called set learning rates. You can pass in your optimizer and your new learning rate, and you don't see if that helps right and very often it does about about an order of magnitude in the deep learning course. We learn a much much better technique than this to do this all automatically and at a more granular level, but if you're doing it by hand, you know like an order of magnitude at a time, is what people generally do so you'll see people in papers talk About learning rate schedules, this is like a learning rate schedule, so this schedule just a moment Erica will come to us. First has got us 297. Okay and I tried kind of going further and we don't seem to be able to get much better than that. So yeah, so here we've got something where we can get 97 % accuracy. Yes Erica. So it seems like you, change the learning rate to something very small ten times smaller than we started with. So we had point one now its point: everyone yep, but that makes the whole model train really slow.
So I was wondering if you can make it so that it changes dynamically as it approaches closer to the minimum yeah pretty much yes. So so that's some of the stuff we learned in the deep planning course, these more advanced approaches, yep, so how it is different from using an immobilizer or something that that's the kind of stuff we can do. I mean you still need annealing. As I say, we do this kind of stuff in the deep learning course so for now we're just going to stick to standard SGD. I had a question about the data loading yeah. I know it's a fast day i Function, but could you go into a little bit detail of how it's creating batches, how it's done and how it's making those decisions? Sure I'd be good to ask that on Monday night, so we can talk about it in detail. In the dig learning class, but let's, let's do the quick version here so basically there's a really nice design in pytorch, where they basically say: let's, let's create a thing called a data set right and a data set is basically something that looks like a list. It has a length right, and so that's like how many images are in the data set and it has the ability to index into it like a list right. So if you had like D equals data set, you can do length D and you can do D or some index right. That's. Basically, all the data set is as far as pipe torch is concerned, and so you start with the data set. So it's like.
Okay. D3 gives you the third image you know or whatever, and so then the idea is that you can take a data set and you can pass that into a constructor for a data loader, and that gives you something which is now iterable right. So you can now say it a DL and that's something that you can call next on and what that now is going to do is if, when you do this, you can choose to have shuffle on or shuffle off shuffle on means. Give me random, mini-batch, shuffle off means drove through it sequentially and so what the data loader does now when you say next, is it basically, assuming you said, shuffle equals true it's going to grab. You know if you've got a batch size of 64 64 random integers between 0 and length and call this 64 times to get 64 different items and jam them together. So fastai uses the exact same terminology and the exact same API. We just do some of the details differently so specifically, particularly with computer vision. You often want to do a lot of pre pre processing data augmentation like flipping changing the colors a little bit rotating those turn out to be really computationally expensive. Even just reading the JPEGs turns out to be computationally expensive, so plate or chooses an approach where it fires off multiple processors to do that in parallel, where else the faster I librarian says, does something called multi-threading, which is a much faster way of doing it. Yes, you know so I mean pork.
Is there really pork in the sense that all of the elements? So it's a shuffle at the beginning of the pork, something like that? Yeah yeah, I mean not all libraries work, the same way. Some do sampling with replacement some. Don't we actually the first a a library hands off the shuffling off to the set to the actual pipe torch version and I believe the pipes version, yeah, actually shuffles and an epoch covers everything once I believe. Okay, now the thing is when you start to get these bigger networks, potentially you're, getting quite a few parameters right. So I want to ask you to calculate how many parameters there are, but let's, let's remember here: we've got 28 by 28 input into a hundred output and then a hundred into a hundred and then 100 into 10, all right and then for each of those we've Got weights and biases, so we can actually do this net dot parameters returns a list where each element of the list is a matrix or actually a tensor of the parameters for that not just for that layer, but if it's a layer with both weights and biases, That would be two parameters right so basically returns us a list of all of the tensors containing the the parameters. Some elements in pytorch tells you how how big that is right. So, if I run this here is the number of parameters in each layer. So I've got 784 inputs and the first layer has a hundred outputs.
So therefore, the first weight matrix is of size, 78,000, 400 and the first bias vector is of size, 100, okay and then the next one is a hundred by hundred okay and there's 100 and then the next one is 100 by 10 and then there's my bias. Okay, so there's the number of elements in each layer - and I add them all up - it's nearly a hundred thousand okay and so I'm possibly at risk of overfitting yeah all right. So we might want to think about using regularization. So a really simple, common approach to regularization in all of machine learning is something called l2 regularization and it's super important super handy. You can use it with just about anything right and the basic idea anyway. So LT rotor ization. The basic idea is this: normally we'd say our loss is equal to. Let's just do our MSE to keep things kind of simple, it's equal to our predictions, our actuals, you know squared and then we sum them up. Take the average take the square root. Okay. So what if we then want to say you know what like, if I've got, lots and lots of parameters, don't use them unless they're really helping enough right like if you've got a million parameters, and you only really needed ten parameters to be useful. Just use ten. All right, so how could we like tell the loss function to do that, and so basically, what we want to say is hey. If a parameter is zero, that's no problem! It's like it doesn't exist at all. So let's penalize a parameter for not being zero right.
So what would be a way we could measure that? How can we like calculate how under Oh our parameters are I can you pass that to Qin Shi? Please Ernest. You calculates the average of positive parameters, can't quite be the average plus. Yes, Taylor yeah. Yes, you figured it out. Okay, so I think if we like, assuming all of our data, has been normalized standardized. However, you want to call it. We want to check that they're like significantly different from zero right, not the data that the parameter is rather would be significantly and the parameters don't have to be normalized or anything that is calculated right. Yes, a significantly different from zero right. I suppose I just assuming that the data has been normalized so that we can compare that money. Oh yeah, I thought of yeah right and then those that are not significantly different from zero. We can provoke. I just draw - and I think Chen she's going to tell us how to do that. You just figured it out right. The could do that. That would be called l1, which is great, so l1 would be the absolute value of the weights. Average l2 is actually the sum yeah yeah exactly so we just take this. We can just we don't even have to square root, so we just take the squares of the weights themselves and then like. We want to be able to say like okay. How much do we want to penalize not being zero right? Because if we actually don't have that many parameters, we don't want to regularize much at all.
If we've got heaps, we do want to regularize a lot right. So then we put a parameter yeah right, except I have a role in my classes, which is never to use Greek letters. So normally people use alpha, I'm going to use hey okay, so so this is some number which you often see something around kind of 1e. Neg 6 to 1 enoch, 4 ish right now. We actually don't care about the loss. When you think about it, we don't actually care about the loss other -- than like, maybe to print it out or we actually care about, is the gradient of the loss. Okay, so the gradient of that right is that right. So there are two ways to do this: we can actually modify our loss function to add in this square penalty or we could modify that thing where we said weights equals weights minus gradient times, learning rate to subtract that as well right actually to add that as Well - and these are roughly, these are kind of basically equivalent, but they have different names. This is called l2 regularization right. This is called weight decay. So in the neural network, literature you know that version kind of was the how it was first posed in the neural network. Literature where else this other version is kind of how it was posed in the statistics, literature and yeah. You know they're they're equivalent, as we talked about in the deep learning class.
It turns out they're not exactly equivalent, because when you have things like momentum and atom, it can behave differently and two weeks ago a research figured out a way to actually do proper weight. Decay in modern optimizers and one of our first day, students just implemented that in the first day I library so first a is now the first library to actually support this properly so anyways. So for now, let's do the the version which applied torch calls weight decay, but actually it turns out, based on this paper two weeks ago, is actually l2 regularization. It's not quite correct, but it's close enough so here we can they weight decay as one in x3. So it's going to set our constant, our penalty, multiplier a21 in X 3, and it's going to add that to the loss function, okay and so let's make a copy of these cells just so we can compare hope. This actually works. Okay and we'll set this running. Okay, because there's now optimizing well, except if you actually so, I've made a mistake here, which is, I didn't rerun this cell. This is an important thing to kind of remember, since I didn't run this rerun this cell here when it created the optimizer and said net dot parameters. It started with the parameters that I had already trained right, so I actually hadn't recreated my network okay. So, actually you go back and rerun this cell first to recreate the network then go through and run this okay.
There we go. So, let's see what happens so you might notice them notice something kind of kind of counterintuitive here, which is that that's our training error right now. You would expect our training error with regularization to be worse. That makes sense right because we're like we're penalizing parameters that specifically, can make it better, and yet actually it started out better not worse, so why could that be? So the reason that can happen is that if you have a function that looks like that right, it takes potentially a really long time to train. Where else, if you have a function, that kind of looks more like that. It's going to train a lot more quickly and there are certain things that you can do, which sometimes just like can take a function, that's kind of horrible and make it less horrible and it's sometimes weight decay. You can actually make your functions. A little more nicely behaved, and that's actually happened here so, like I just mentioned that to say like don't let that confuse you right, like white decay, really does penalize the training set and look so strictly speaking. The final number we get to for the training set shouldn't end up, be beat being better, but it can train sometimes more quickly. All right! Yes, can you pass it a chance? You don't get it okay. Why making it faster like Zee Time Matters like the training time, memories? No, it's just after one epoch right. So after one epoch.
Now, congratulations for saying I don't get it! That's like the best thing anybody can say you know so hopeful. This here was our training. Without wait, okay, okay and this here is our training with wait: okay, okay, so this is not really just a time. This is related to just an epoch right after one Apoc. My claim was that you would expect the training set all other things being equal to have a worse loss with weight decay because we're penalizing it. You know this has no penalty. This has a penalty, so the thing with the penalty should be worse and I'm saying oh, it's not that's weird right, and so the reason it's not is because in a single epoch it matters a lot as to whether you're trying to optimize something that's very bumpy Or whether you're trying to optimize something that's kind of nice and smooth, if you're, trying to optimize something, that's really bumpy, like imagine in some high dimensional space right, you end up kind of rolling around through all these different tubes and tunnels and stuff. You know where else, if it's just smooth, you just go boom item, it's like imagine, of marble rolling down a hill where one of them you've got like it's a called Lombard Street in San Francisco. It's like backwards forwards backwards forwards. It takes a long time to drive down the road right.
Where else you know, if you kind of, took a motorbike and just went straight over the top, you just make sure boom right, so so weather so kind of the shape of the loss function. Surface. You know, impacts or kind of defines how easy it is optimized and therefore, how far can it get in a single epoch and based on these results, it would appear that weight decay here has made it this function easier to optimize, so just to make sure it's The penalizing is making the optimizer more than likely to reach the global minimum. No, I wouldn't say that my claim actually is that at the end, it's probably going to be less good on the training set, and indeed this does look to be the case. At the end, after five epochs, our training set is now worse with weight decay. Now, that's what I would expect right. I would expect like, if you actually find like, I never used a term global optimum, because it's just not something we have any guarantees about. We don't really care about, or just care like, where do we get to after a certain number of epochs. We hope that we found somewhere, that's like a good solution, and so by the time we get to like a good solution, the training set with weight decay. The loss is worse because it's very right, but on the validation set, the loss is better right because we penalized the training set in order to kind of try and create something it generalizes better. So we've got more parameter.
You know that the parameters that are kind of pointless in l0 and it generalizes better right so so all we're seeing is that it just got to a good point after one epoch is really orbiting. Obviously, no, no I, but if you buy, if you mean just weight decay, you always make the function surface smoother. No, it's not always true, but it's like it's worth, remembering that if you're having trouble training of function, adding a little bit of weight, decay may may help what so by analyzing the parameters, what it does as its moons out the loss function. I mean it's not it's not why we do it. You know the reason why we do. It is because we want to penalize things that aren't zero. To say, like don't, make this per a high number, unless it's really helping the Lasser lodge right, set it to 0, if you can, because setting as many parameters to zero as possible means that it's going to generalize better right, it's like the same as having a Smaller network, okay, so that's that's! We do that's why we do it, but it can change how it learns as well. So let's, okay, this one moment, okay, so I just wanted to check how we actually went here so after the second epoch yeah. So you can see here, it's really has helped right after the second epoch, before we got to 97 % accuracy, now we're nearly up to about 98 % accuracy right and you can see that the loss was 0.08 versus 0.13 right.
So adding regularization has allowed us to find a you know: three percent versus two percent, so like a fifty percent, better solution, yes Erica. So there are two pieces to this right. What is l2 regularization and the way to key know they're the same, so my claim was they're the same thing right. So white decay is the version. If you just take the derivative of LT regularization, you get weight decay, so you can implement it either by changing the loss function with an with a squared loss penalty or you can implement it by adding the weights themselves as part of the the gradient okay yeah. It's just gon na finish the questions. Yes, okay, pass it to division. Can we use the regularization convolution, absolutely so a compilation layer just is: is White's aim? Jeremy? Can you explain why you thought you needed weight decay in this particular problem, not easily? I mean other than to say it's something that I would always try speaking well yeah I mean okay. So even if I yeah okay, that's a good point unit. So if if my training loss was higher than my validation loss, then I'm under fitting right. So, there's definitely no point recognizing right if, like that, would always be a bad thing. That would always mean you need like more parameters in your model. In this case, I'm I'm overfitting that doesn't necessarily mean regularization will help, but it's certainly worth trying. Thank you. You know that's a great point.
There's one more question: yep Tyler doing a pass over there. So how do you choose the optimal number of epic? You do my take learning cause it's a it's! That's a long story and a lot to lots of users by here on here, ollie, it's a bit of both. We just don't. As I say, we don't have time to cover best practices in this class, we're going to learn the kind of fundamentals yeah okay. So, let's take a six minute break and
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Come back at 11:10, alright, so something that we cover in great detail in the deep learning course. But it's like really important to mention here is that is it the secret? In my opinion, to kind of modern machine learning techniques is to massively over parameterize the solution to your problem. Right like as we've done here, you know, we've got like a hundred thousand weights when we only had a small number of 28 by 28 images and then use regularization. Okay, it's like the direct opposite of how nearly all statistics and learning was done for decades before and still most kind of like senior lecturers at most universities in most areas of have this background, where they've learned the correct way to build a model is to like Have as few parameters as possible right - and so hopefully, we've learnt two things so far. You know one is: we can build very accurate models even when they have lots and lots of parameters like a random forest has a lot of parameters - and you know this here. Deep network has a lot of parameters and they can be accurate right and we can do that by either using bagging or by using regularization. Okay and regularization in neural nets means either weight decay, also known as kind of filter, regularization or drop out, which we won't worry too much about yeah.
So like it's a it's a very different way of thinking about building useful models and like I just wanted to kind of warn you that once you leave this classroom like even possibly when you go to the next faculty members, talk like there'll, be people at USF As well, who entirely trained in the world of like models with small numbers of parameters, you know your next boss is very likely to have been trained in the world of models with small numbers of parameters, the idea that they are somehow more pure or easier or Better or more interpretable or whatever I I am convinced that that is not true, probably not ever true, certainly very rarely true, and that actually models with lots of parameters can be extremely interpretive. All as we learn from our whole lesson of random forest interpretation, you can use most of the same techniques with neural nets, but with neural nets are even easier right. Remember how we did feature importance by randomizing a column to see how it changes in that column. Would impact the output? Well, that's just like a kind of dumb way of calculating its gradient. How much does varying this import change the output with a neural net? We can actually calculate its gradient right so with pytorch, you can actually say what's the gradient, that the output with respect to this column, okay, you can do the same kind of thing to do partial dependence plot with an Anette - and you know I mentioned - for Those of you interested in making a real impact nobody's written basically any of these things for neural nets right, so that that that whole area needs like libraries to be written blog posts to be written.
You know some papers have been written, but only in very narrow domains like computer vision, as far as I know, nobody's written the paper saying: here's how to do structured data neural networks. You know, interpretation methods, so it's a really exciting big area. So what we're going to do, though, is we're going to start with applying this with a simple
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Linear model - this is mildly terrifying for me, because we're going to do NLP and NLP faculty expert is in the room, so David just yell at me. If I screw this up too badly and so NLP refers to, you know any any kind of modeling where we're working with with natural language tests right and it. Interestingly enough, we're going to look at a situation where a linear model is pretty close to the state of the art for solving a particular problem. It's actually something where I actually surpassed this baited state of the art in this using a recurrent neural network. A few weeks ago, but this is actually going to show you pretty close to the state of art with with a linear model, we're going to be working with the IMDB. I am DVD data set. So this is a data set of movie reviews. You can download it by
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Following these steps, and once you download it, you'll see that you've got a train and a test directory and in your train directory you'll, see, there's a negative and a positive directory and in your positive directory, you'll see, there's a bunch of text files and here's an Example of a text file, so somehow we've managed to pick out a story of a man who has a natural feelings for a pig as our first choice that wasn't intentional, but it'll be fun. So we're going to look at these movie reviews and for each one we're going to look to see whether they were positive or negative. So they've been put into one of these folders. They were downloaded from from IMDB the movie database and review site, the ones that were strongly positive, went positive, strongly negative, went negative and the rest they didn't label at all, so there's only highly polarized reviews. So, in this case you know we have an insane violent mob, which, unfortunately it is too absurd, too off-putting those in the area be turned off, so the label for this was a zero which is negative okay. So this is a negative review, so in the fastai library, there's lots of little functions and classes to help with most kinds of domains that you do machine learning on for NLP. One of the simple things we have is texts from folders there's just going to go ahead and go through and find all of the folders in here with these names and create a labeled data set, and you know don't let these things ever. Stop you from understanding.
What's going on behind the scenes, okay, we can grab its source code and, as you can see, it's tiny. You know it's like five lines. Okay, so I don't like to write these things out in full. You know, but hide them behind at all functions. So you can reuse them, but basically it's going to go through each directory and then within that, so I go through yeah, go through each directory and then go through each file in that directory and then stick that into this array of texts and figure out what Folder it's in and stick that into the array of labels. Okay, so that's how we basically end up with something where we have an array of the reviews and an array of the labels. Okay, so that's our data, so our job will be to take that and to predict that okay and the way we're going to do it is we're gon na throw away like all of the interesting stuff about language, which is the order in which the words are In right now, this is very often not a good idea, but in this particular case it's going to turn out to work like not too badly. So let me show you what I mean by like throwing away the order of the words like normally the order of the words matters a lot if you've got a not before something, then that not refers to that thing right so, but the thing is when, in This case we're trying to predict whether something is positive or negative.
If you see the word absurd appear a lot right, then maybe that's a sign that this isn't very good. So you know cryptic, maybe there's a sign that it's not very good. So the idea is that we're going to turn it into something called a term document matrix where for each document, are you each review, we're just going to create a list of what words are in it, rather than what order they're in so. Let me give an example: can you see this okay? Okay, so here are four movie reviews that I made up. This movie is good. The movie is good they're. Both positive. This movie is bad. The movie is bad they're, both negative right. So I'm going to turn this into a term document matrix. So the first thing I need to do is create some in-court of vocabulary. A vocabulary is a list of all the unique words that appear. Okay, so here's my vocabulary. This movie is good, the bad. That's all the words okay, and so now I'm going to take each one of my movie reviews and turn it into a vector of which words appear, and how often do they appear right, and in this case none of my words appear twice. So this movie is good. Has those four words in it? Where else this movie is bad, has those four words in it? Okay, so this is called a term document matrix alright and this representation. We call a bag of words, representation right, so this here is a bag of words, representation of the view of the review. It doesn't contain the order of the text anymore. It's just a bag of the words.
What words are in it. It contains bad is movie this okay. So that's the first thing we're going to do is we're going to turn it into a bag of words, representation, and the reason that this is convenient for linear models is that this is a nice rectangular matrix that we can like do. Math on okay, and specifically, we can do a logistic regression and that's what we're going to do is we're going to get to a point. We do a logistic regression before we get there, though we're going to do something else, which is called naive, Bayes, okay, so scikit-learn has something
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Which will create a term document matrix for us? It's called count: vectorizer, okay. So what does use it now in NLP? You have to turn your text into a list of words and that's called tokenization. Okay and that's actually non-trivial because like if this was actually, this movie is good dot right or if it was. This movie is good like how do you deal with like that punctuation? Well, perhaps more! Interestingly, what if it was this movie, isn't good, alright, so how you turn a piece of text into a list of tokens is called tokenization right and so a good tokenizer would turn this movie isn't good into this. This base quote movie. Space is space and good space right, so you can see in this version here. If I now split this on spaces, every token is either a single piece of punctuation or like this suffix and is considered like a word right, that's kind of like how we would probably want to tokenize that piece of text, because you wouldn't want good to be Like an object right because there's no concept of good right or double-quote movie is not like an object right, so tokenization is something we hand off to a tokenizer. First, AI has a tokenizer in it that we can use. So this is how we create our term document matrix with a tokenizer s. Kaitlyn has a pretty standard API, which is nice, I'm sure you've seen it a few times now before so once we've built some kind of model think we can kind of think of this.
As a model just ish, this is just defining what it's going to do, but you can call fit transform to do that right. So, in this case, fit transform is going to create the vocabulary. Okay and create the term document matrix based on the training set. Transform is a little bit different that says, use the previously fitted model, which in this case means use the previously created vocabulary. We wouldn't want the validation set in the training set to have. You know the words in different orders in the matrices right, because then they'd like to have different meanings, so this is here saying: use the same vocabulary to create a bag of words for the validation set. Could you pass that back in please what if the violation set has different set of words other than training yeah? That's a great question! So, generally most of these kind of vocab, creating approaches will have a special token for unknown. Sometimes you can you'll also say like hey if a word appears less than three times call it unknown, but otherwise it's like, if you see something you haven't seen before, call it unknown. So that would just become a column in the bag of words is good question. All right so when we create this term document matrix the training set, we have 25,000 rows because there are 25,000 movie reviews and there are 70 5132 columns.
What does that represent? What does that mean? There are seven hundred and seventy-five thousand hundred thirty-two. What can you pass that to device
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Just a moment: okay power sector diversion locality, yeah come on. What do you mean so like the number of words Union or the number of words that the number of unique words yeah exactly good? Okay, now most documents don't have most of these 75,000 words right. So we don't want to actually store that as a normal array in memory, because it's going to be very wasteful, so instead we store it as a sparse matrix, okay and what a sparse matrix does. Is it just stores it as something that says whereabouts of the non-zeros right? So it says, like okay term number, so document number one word number four appears and it has four of them. You know document one term number 123 has that that appears and it's a one right and so forth. That's basically how it's done, there's actually a number of different ways of storing and if you do, Rachel's, computational, linear algebra course. You'll learn about the different types and why you choose them and how to convert and so forth, but they're all kind of something like this right and you don't really on the whole have to worry about the details. The important thing to know is it's: it's efficient. Okay, and so we could grab the first review all right, and that gives us 75,000 long, sparse one long one row long matrix, okay, with 93 stored elements, so in other words, 93 of those words, are actually used in the first document.
Okay, we can have a look at the vocabulary by saying vectorizer, docket fetcher feature names that gives us the vocab and so here's an example of a few of the elements of feature names. I didn't intentionally pick the one that had Ozzie, but you know that's the important words. Obviously I haven't you the tokenizer here, I'm just bidding on space, so this isn't quite the same as what the vectorizer did, but to simplify things. Let's grab a set of all the lowercased words by making it a set, we make them unique. So this is roughly the list of words that would appear right and that length is 91, which is pretty similar to 93, and just the difference will be that I didn't use a real tokenizer yeah right. So that's, basically all that's been done there. It's probably created this unique list of words and mapped them. We could check by calling vectorizer cavalry underscore to find the ID of a particular word. So this is like the reverse map of this one right. This is like integer. Two word here is word two integer, and so we saw absurd appeared twice in the first document. So, let's check train term dark zero, comma one, two nine seven there it is - is two right or else unfortunately, Ozzie didn't appear in the unnatural relationship with a pig movie, so zero comma five thousand is zero. Okay. So that's that's our term document matrix.
Yes, so does it care about the relative relationship between the words as in the ordering of the words no we've thrown away the orderings? That's why it's a bag of words and I'm not claiming that this is like necessarily a good idea. What I will say is that, like the vast majority of NLP work, that's been done over the last few decades generally uses this representation, because we didn't really know much better nowadays, increasingly we're using recurrent neural networks instead, which we'll learn about in our last deep learning. Lesson of part one, but sometimes this
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Representation works pretty well, and it's actually going to work pretty well in this case. Okay, so in fact you know most like back when I was at first me or my email company, a lot of the spam filtering. We did used this next technique, naivebayes, which is as a bag of words approach just kind of like you know, if you're getting a lot of email containing the word viagra and it's always been a spam and you never get email from your friends talking about viagra, Then it's very likely something that says by Agra, regardless of the detail of the language, is probably from a spammer all right. So that's the basic theory about like classification using a term document matrix okay. So let's talk about naive, Bayes and here's the basic idea we're going to start with our term document matrix right, and these first two is our corpus of positive reviews. These next two is our corpus of negative reviews, and so here's our whole corpus of all reviews. So what I could do is now to create a probability. I've got to call the as we tend to call these more generically features rather than words right. This is a feature movie as a feature is as a feature right. So it's kind of more now like machine learning, language, a column is a feature. Look all those we call those earth in naive Bayes. So we can basically say the probability that you would see the word this, given that the class is 1, given that it's a positive review is just the average of how often do you see this in the positive reviews right now, we've got to be a bit Careful, though, because if you never ever see a particular word in a particular class right, so if I've never received an email from a friend that said viagra right, that doesn't actually mean the probability of a friend.
Send me sending me an email about viagra is zero. It's not really zero right. I I hope I don't get an email. You know from Terrence tomorrow saying like Jeremy, you probably could use this. You know effortless meant for viagra, but you know it could happen. You know I'm sure it'd be in my best interest yeah. So so what we do is we say actually what we've seen so far is not the full sample of everything that could happen. It's like a sample of what's happened so far, so let's assume that the next email you get actually does mention viagra and every other possible word right. So basically we're going to add a row of ones. Okay, so that's like the email that contains every possible word. So that way, nothing's ever infinitely and unlikely yeah. So I take the average of all of the times that this appears in my positive corpus, plus the ones okay. So that's like the the probability that feature equals. This appears in a document given that class equals one, and so not surprisingly, here's the same thing for probability that this feature this appears given class equals zero all right same calculation, except for the zero rows, and obviously these are the same because this appears twice in The positives, sorry, once in the positives and once in the negatives, okay, let's just put this back to what it was all right, so we can do that for every feature for every class right. So our trick now is to basically use Bayes rule to kind of fill this in.
So what we want is the probability that, given that I've got this particular document, so somebody sent me this particular email or I have this particular IMDB review. What's the probability that its class is equal to, I don't know positive right so for this particular movie review. What's the probability that it's plus is positive right, and so we can say: well that's equal to the probability that we got this particular movie review, given that its class is positive, multiplied by the probability that any movie reviews plus is positive, divided by the probability of Getting this particular movie review all right, that's just basis rule okay, and so we can calculate all of those things. But actually what we really want to know is: is it more likely that this is plus 0 or plus 1 right? So what if we actually took probability, that's plus 1 and divided by probability, that's plus 0? What if we did that right and so then we could say like okay, if this number is bigger than 1, that it's more likely to be class 1. If it's smaller than 1, it's more likely to be class 0 right. So in that case, we could just divide this whole thing right by the same version for class 0 right, which is the same as multiplying it by the reciprocal, and so the nice thing is now that's going to put a probability D on top here, which we Can get rid of right and a probability of getting the data given zero down here and the probability of getting class zero here right, and so, if we, basically what that means is we want to calculate the probability that we would get this particular document, given that The class is 1 times the probability that the class is 1 divided by the probability of getting this particular document.
Given the classes to 0 times the probability that the class is 0. So the probability that the class is 1 is just equal to the average of the labels write probability. The class is 0, is just 1 minus that right. So so there are those two numbers right. I've got an equal amount of both, so it's both 0.5. What is the probability of getting this document, given that the class is 1? Can anybody tell me how I would calculate that? Can somebody pass that so remember, so it's going to be for a particular document. So, for example, would be saying like what's the probability that this review is positive, all right, so what so you're on the right track. But what we have to gon na have to do is going to have to say: let's just look at the words it has and then multiply. The probabilities together for class equals 1 right. So the probability that a class 1 review has this is 2/3. The probability it has movie is one is, is 1 and good is 1, so the probability it has all of them is all of those multiplied together, kinda and the kinder phyla. Why is it not really, can you press it in Taylor?
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
So glad you look horrified and skeptical where choices, not independence. Thank you. So nobody can call Tyler naive, because the reason this is naive, Bayes is because this is what happens if you take bayes's theorem, Xander naive. Why and Tyler is not naive. Anything better right!    Bayes ain't actually very good, but I'm kind of teaching it to you, because it's going to turn out to be a convenient piece for something we're about to learn later. It's okay right! I mean it's, it's it's! I would never. I would never choose it like. I don't think it's better than any other technique, that's equally fast and equally easy, but you know it's the thing you can do and it's certainly going to be a useful foundation. So so here is now calculation right of the probability that this document is that we get this particular document, assuming it's a positive review, here's the probability given us a negative and here's the ratio and this ratio is above one. So we're going to say. I think that this is probably a positive review. Okay, so that's the Excel version, and so you can tell that I let your net touch this, because it's got latex in it. We've got actual math, so so here is.
The here is the same thing. The log count ratio of H, feature FH would if - and so here it is written out as Python okay, so our independent variable is our term document matrix. A dependent variable is just the labels of the way so using numpy. This is going to grab the rows where the dependent variable is one okay, and so then we can sum them over the rows to get the total word count for that feature across all the documents right plus one all right, because that's the email Terrance is totally Going to send me something about Viagra today, I can tell that's that's that yeah okay, so do the same thing for the negative reviews right and then, of course, it's nicer to take the log right, because if we take the log, then we can add things together. Rather than multiply them together - and once you like multiply enough of these things together, it's going to get kind of so close to zero. That you'll probably run out of floating-point right. So we take the log of the ratios and then we come. As I say, we then multiply that or a log we subtract that from the so you add that to the ratio of the class, the whole plus probabilities all right. So in order to say for each document multiply the phase probabilities by the accounts. We can just use matrix model, plane, okay and then to add on the the log of the class ratios. We can just use plus B, and so we end up with something that looks a lot like our logistic regression right, but we're not learning anything right, not in kind of assess, GED point of view we're just we're calculating it using this theoretical model, okay, and so, As I said, we can then compare that as to whether it's bigger or smaller than zero, not one anymore, because we're now in log space right and then we can compare that to the mean - and we say: okay - that's 80 %. Accurate 81 was inaccurate right.
So naive, Bayes, you know is not it's not nothing. It gave us something. Okay, it turns out that this version, where we're actually looking at how often a word appears like absurd, appeared twice. It turns out, at least for this problem, and quite often it doesn't matter whether absurd appeared twice or once all that matters is that it appeared. So what what people tend to try doing is to say take the two other term document matrix and go dot sine dot sign replaces anything positive as one and anything negative with negative one. We don't have any negative counts. Obviously so this buying arises it so it says it's, I don't care that you saw absurd twice and it's care that you saw it right. So if we do exactly the same thing with the binarized version, then you get a better result. Okay, okay. Now
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
This is the difference between theory and practice. Right in theory, naive Bayes sounds okay, but it's it's naive, unlike Tyla, it's naive right, so what Tyler would probably do would instead say rather than assuming that I should use these coefficients. Ah, why don't we learn them? So it sound reasonable, Tyler, yeah, okay, so let's learn them so we can. You know we can totally learn them. So let's create a logistic regression right and let's fit some coefficients and that's going to literally give us something with exactly the same functional form that we had before. But now, rather than using a theoretical R and a theoretical B. We're going to calculate the two things based on logistic regression and that's better okay. So so it's kind of like yeah. Why? Why do something based on some theoretical model, because theoretical models are never gon na, be as accurate, pretty much as a data-driven model right because theoretical models unless you're dealing with some - I don't know like physics, thing or something where you're like okay? This is actually how the world works. There really is, no, I don't know we're working in a vacuum, and this is the exact gravity and blah blah blah right, but most of the real world. This is how things are like it's better to lie on your coefficients and calculate them.
Yes, you know generally, what's this do liquid through, I was hoping you'd ignore, not notice, but or basically in this case, our term document matrix is much wider than it is tall. There is a reformulation of mathematically, basically almost a mathematically, equivalent reformulations of logistic regression. That happens to be a lot faster when it's wider than it is tall. So the short answer is, if you don't put that here anytime, it's wider than it is tall. What really comes true it'll run. This runs in like two seconds. If you don't have it here, it'll take a few minutes so like in math. This is kind of concept of dual versions of problems which are kind of like equivalent versions that sometimes work better for certain situations. Okay here is so here is the binarized version. Okay and it's it's about the same right, so you can see I've fitted it with the the sine of the dock of the dr. Murdock matrix and predicted it with this right now. The thing is that this is going to be a coefficient for every term, where I was about 75,000 terms in their vocabulary, and that seems like a lot of coefficients given that we've only got twenty five thousand reviews. So maybe we should try regularizing this, so we can use regularization built into SK. Learns logistic regression class, which is C, is the parameter that they use. A small process is slightly weird, as smaller parameter is more regularization right.
So that's why I used wanting eight to basically turn off regularization yeah. So if I turn on regularization set it to 0.1, then now it's 88 %, okay, which makes sense. You know you, wouldn't you would think like 75,000 parameters for 25,000 documents. You know it's likely to overfit; indeed it did overfit, so this is adding l2 regularization to avoid overfitting. I mentioned earlier that as well as l2, which is looking at the weight squared, there's also l1, which is looking at just the absolute value of the way right. I I was kind of pretty sloppy in my wording before I said that l/2 who tries to make things zero, that's kind of true, but if you've got two things that are highly correlated, then l2 regularization will make move them both down together. It won't make one of them zero and one of them nonzero right so l1 regularization actually has the property that it'll try to make as many things zero as possible. Where else l2 regularization has a property that it tends to try to make kind of everything smaller. We actually don't care about that difference in really any modern machine learning, because we very rarely try to directly interpret the coefficients. We try to understand our models through interrogation using the kind of techniques that we've learned. The reason that we would care about l1 versus l2 is simply like which one ends up with a better error on the validation set. Okay - and you can try both with scikit-learn logistic regression l2 actually turns out to be a lot faster because you can't use your equals true unless you have L 2, so you know and l2 is the default.
So I didn't really worry too much about that difference. Yet so you can see here if we use regularization and binarized, we actually do pretty well. Okay! So yes, can you pass that back to W please before we learned about elastic net right like combining l1 and l2 yeah yeah yeah? You can do that, but I mean it's like you know, with with deeper models and yeah I've never seen anybody find that useful. Okay, so the last thing
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
I'll mention is that you can, when you do your count, vectorizer order. That was when you do your account vectorizer. You can also ask for n grams right by default, we get uni grams. That is single words, but if we, if we say Engram range, equals 1, comma 3, that's also going to give us by grams and trigrams, by which I mean. If I now say: okay, let's go ahead and do the count. Vectorizer cat feature names now. My vocabulary includes a bigram right, bypassed by vengeance and a trigram by vengeance, 5 euro miles right. So this is now doing the same thing. But after tokenizing it's not describing each word and saying that's part of our vocabulary: two words next to each other and each three words next to each other and this 10. This turns out to be like super helpful in like taking advantage of bag of word approaches, because we now can see like the difference between, like you know, not good versus, not bad. This is not terrible right or even like double quote. Good double quote, which is probably going to be sarcastic right, so using trigram features, actually is going to turn out to make both naivebayes and logistic regression quite a lot better. It really takes us quite a lot further and makes them quite useful. I have a question about the token icers, so you are saying some marks features. So how are these diagrams and trigrams selected right? So since I'm using a linear model, I didn't want to create too many features.
I mean it actually worked. Fine, even without max features. I think I had something like I can't remember: 70 million coefficients it still worked right, but just there's no need to have 70 million coefficients. So if you say max features equals 800,000, the count vectorizer will sort the vocabulary by how often everything appears whether it be uni ground by graham diagram, and it will cut it off after the first 800,000. Most common engrams Engram is just a generic word for uni gram by Graham and trigram. So that's why the train term doctype is now 25 thousand by 800,000 and like if you're, not sure what number this should be. I just picked something that was really big and you know didn't didn't worry about it too much and it seemed to be fine, like it's not terribly sensitive, all right, okay! Well, that's we're out of time. So what we're going to see next week and by the way you know we could have replaced this logistic regression with our pytorch version and next week, we'll actually see something in the fast, a library that does exactly that. But also what we'll see next week. Starting next week, tomorrow is how to combine logistic regression and naive Bayes. Together, you get something that's better than either and then we'll learn how to move from there to create a deeper neural network to get a pretty much state-of-the-art result for structured learning all right.
So we'll see you then


A ouanga bag confiscated by Marines in 1921 near Gonaives is described as follows:
It was a hide bag, and in it were these objects : luck stones, snake bones, lizard jaws, squirrel teeth, bat bones, frog bones, black hen feathers and bones, black lamb wool, dove hearts, mole skins, images of wax and clay, candy made of brown sugar mixed with liver, mud, sulphur, salt, and alum, and vegetable poisons.
“From The Author's Notebook”, appendix to The Magic Island (1929) by W B Seabrook (1884–1945), an “American Lost Generation occultist, explorer, traveller, cannibal, and journalist”, says the Wikipedia. Oh, also an alcoholic, a sadist, and a suicide.


The app lets you insert your birthday date i.e. day,month and year and also your gender. It then calculates the day of the week you were born i.e. whether you were born on Monday, Tuesday etc. It then gives an Akan name (derived from Ghanian culture whereby children in Ghana are at times given their first name as a 'day name' which corresponds to the day in the week they were born and their gender)
Date of Birth
Gender


Both taskbar are draggable between edges, but only within their
orientation. Drag taskbars around to see the effect.
Drag horizontal taskbar to the left or right edge of the main window
in X axis.
Drag vertical taskbar to the top or bottom edge of the main window
in Y axis.


Loading…
Heightfield
·
Isosurface
Isosurface Terrain
(Press ALT to hide the overlay)
NTNU, Norway
Brandenburg University of Applied Sciences
Summer Semester 2016