fast.ai: Intro to Machine Learning 1 (v1) (2018)
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Okay, so welcome back so we're going to start by doing some review and we're going to talk about test sets. Training sets, validation, sets and oob. Something we haven't covered yet, but we will cover in more detail. Later is also cross validation. But I'm going to talk about that as well right, so we have a data set with a bunch of rows in it and we've got some dependent variable, and so what's the difference between my machine learning and kind of pretty much any other kind of work that The the difference is that in machine learning, the thing we care about is the generalization accuracy or the generalization error. Where else in pretty much everything else. All we care about is is how well we could have mapped to the observations, all stop, and so this this thing about generalization is the key, unique piece of with machine learning, and so, if we want to know whether we could doing a good job of machine Learning we need to do know whether we're doing a good job of generalizing. If we don't know that we know nothing right by generalizing, do you mean like scaling being able to scale larger? No III, don't mean scaling at all. So scaling is an important thing in many many areas. It's like okay, we've got something that works. I'm on my computer with ten thousand items. I do need to work, make it work on ten thousand items per second or something so scaling is important, but not just a machine learning for just about everything we put in production.
Generalization is where I say: okay, here is a model that can predict cats from dogs. I've looked at five pictures of cats, five pictures of dogs and I've built a model that is perfect and then I look at a different set of five cats and dogs, and it gets them all wrong. So, in that case, when it learned, was not a good Street, a cat and a dog that it learned what those five exact cats looked like in those five exact dogs, the play - or I built a model of predicting grocery sales for a particular product. So for toilet rolls in New Jersey last month and then I go and put it into production and it scales great. In other words, it can have the great latency. I don't have a high CPU load, but it fails to predict anything well other than toilet rolls in New Jersey. It also it turns out. It only did it well for last month, not the next month, so these are all generalization failures, so the most common way that people check for the ability to generalize is to create a random sample. So they'll grab a few rows at random and pull it out into a test set and then they'll build all of their models on the rest of the rows and then, when they're finished they'll check that the accuracy they got on there. So the rest of the rows are called the training set everything else, everything else we could call the training set, and so, at the end of their modeling process, on the training set, they got an accuracy of 99 percent of predicting cats from dogs. At the very end, they check it against a test set to make sure that the model really does generalize.
Now the problem is what, if it doesn't right so, okay? Well, I could go back and change. Some hyper parameters. Do some data augmentation and whatever else, try to create a more generalizable model and then I'll go back again after doing all that and check, and it's still no good and I'll keep doing this again and again until eventually, after fifty attempts, it does generalize. But does it really generalize, because maybe all I've done is accidentally found this one which happens to work just for that test set because I've tried 50 different things right, and so, if I've got something which is like right? Coincidentally, 0.05, 5 percent of the time. They're. Not very likely to accidentally get a good result, so what we generally do is we put aside a second data, set they'll, get a couple more of these and put these aside into a validation set. It's an audacious set right and then everything - that's not in the validation or tests is now training. And so what we do is we train a model check it against the validation to see if it generalizes do that a few times, and then, when we finally got something we were like okay, we think this generalizes successfully based on the validation set and then at The end of the project, we check it against the test set yeah. So, basically, if I making this two layer test, that validation said if he gets one right, the other one wrong, you're kind of double-checking, your errors, it's checking that we have an overfit to the validation set.
So if we're using the validation set again and again, then we could end up not coming up with a generalizable sort of hyper parameters and a set of private creditors that just so happened to work on the training set and the validation set. So so, if we try 50 different models against the validation set and then at the end of all that, we then check that against the test set and it's still generalized as well, then we're kind of going to say. Okay, that's good. We've actually come up with generalizable model. If it doesn't, then that's going to say: okay, we've actually now overfit to the validation set at which point you're kind of in trouble right because you don't you know, you don't have anything left behind right. So the idea is to use effective techniques during the modeling so that so that doesn't happen right. But but if it's going to happen, you want to find out about it like you need that test set to be there, because otherwise, when you put it in production and then it turns out that it doesn't generalize, that would be a really bad outcome right. You end up with less people clicking on your ads or selling less with your products or providing car insurance to very risky vehicles or whatever so just make sure to need to ever check if the validation set and the test antics is coherent or you just keep Tested so if you've done what I've just done here, which is to randomly sample, there's no particular reason to check as long as there as long as they're big enough right, but we're going to come back to your question in a different context.
In just a moment. Now another trick: we've learned for renin forests is a way of not needing a validation set and the way what we learnt was to use instead use the oob era for the OO be scored, and so this idea was to say well, every time we train a Tree in a random forest, there's a bunch of observations that are held out anyway, because that's how we get some of the randomness. And so let's calculate our score for each tree, based on those held out samples and therefore the forest. By averaging the trees that that each row was not part of training, okay and so the oob score gives us something which is pretty similar to the validation score, but on average it's a little less good. Can anybody either remember or figure out why, on average, it's a little less good, quite a subtle one, don't give it to Kenji, I'm not sure, but is it because you are treating like you're doing every kind of probe pre-processing on your tests and so the OB Score is reflecting the performance on testing set no for the other piece, because not using the test set at all the other Peace Corps is using the held out rows in the training set at page tree. So I mean Z. You are basically testing each tree of some data from Z. Training set. Yes, so you are. You have the potential of over feeding with agents. It shouldn't cause overfitting, because each one is looking at a held out sample. So it's not an overfitting issue.
It's quite a subtle issue: Enes through never trained aren't. This sample is from OB bootstrap samples. They also then you're, never gon na grab, 63 % of writes chance to OB is one minus 63 percent exactly yeah both you sure. So then, if you know, why would the score be lower than the validation school and then that you're, leaving sort of like a black hole in the data that there's like there, two points you're never going to sample and I'm not gon na be represented by the Model, ah, no, that's not true, though, because each tree is looking at a different set right so that I won't be so like we've got like, I don't know dozens of models right and a niche one there's a different set of rows, which, which happened to be Held out right, and so when we calculate the oeob score for like let's say row three, we say: okay, row three is in this tree this tree and that's it, and so we calculate the prediction on that tree and for that tree and we'd average. Those two predictions, and so with enough trees, you know each one has a 30 or so percent chance, sorry, forty or so percent chance that the row is in that tree. So if you have fifty trees, it's almost certain that every row is going to be mentioned somewhere. Did you have an idea term, which relevation said we can use the whole forest to make the predictions, but here we cannot use the whole forest, so we cannot exactly see exactly so.
Every road is going to be using a subset of the trees to make its prediction and with less trees. We know we get a less accurate prediction. So that's that's a subtle one right and if you didn't get it have a think during the week. Until you understand why this is because it's a really interesting test of your understanding of random forests, it like why is our B score on average, less good and your validation is for they're both using random subnet subsets anyway, it's really close enough right. So why have a validation set at all when you're using random forests? If it's a randomly chosen, validation set it's not strictly speaking necessary, but you know you've got like four levels of things to test right, so you could like test on the oeob when that's working. Well, you can test on the validation set, you know, and hopefully, by the time you check against
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
The test set, there's going to be surprises, so that'll be one good reason. Then what cattle do the way they do? This is kind of clever while CAG will do. Is they split the test set into two pieces, a public and a private, and they don't tell you which is rich, so you submit your predictions to cattle and then a random 30 % of those are used to tell you the leaderboard score. But then, at the end of the competition that gets thrown away and they use the other 70 % to calculate your real score. So what that's doing is that you're making sure that you're not like continually using that feedback from the leaderboard to figure out some set of hyper parameters that happens to do well on the public that actually doesn't generalize okay. So it's a great test like this is one of the reasons why it's good practice to use cattle, because at the end of a competition at some point, this will happen to you and you'll drop a hundred places on the leaderboard the last day of the competition. When they use the private test set and say: oh okay, that's what it feels like to overfit and it's much better to practice and get that sense there than it is to do it in a company where there's hundreds of millions of dollars on the line. Okay! So this is like the easiest possible situation where you're able to use a random sample for your validation set.
Why might I not be able to use a random sample from my validation set or possibly fail in the case of something where we're forecasting we can't randomly sample, because we need to maintain the temporal ordering hello? What is that? Because it doesn't, it doesn't make sense. So, in the case of like an ARMA model, I can't use like I can't pull out random rows, because there's I'm thinking that there's like a certain dependency or I'm trying to model a certain dependency that relies on like a specific lag turn. And if I randomly sample those things, then that lag term isn't there for me to okay. So it could be like a technical, modeling issue that, like I'm using a model that relies on like yesterday, the day before and the day before that and if I randomly removed some things I don't have yesterday and my model might just fail. Okay, that's true, but there's a more fundamental issue. You want to pass it to Tyler and it's a really good point, although you know in general we're going to try to build models that are not little more resilient than that, particularly with yet temporal order. We expect things that are close by in time to be related to things close to them so weeks, so we destroy the water like. If, if we destroy the order, we really aren't going to be able to use that this time is close to this other time.
I don't think that's true, because I can pull out a random sample for a validation set and still keep everything nicely ordered well lame reject things in the future, which we would require as much data close to the hand alert. Okay, that's true. I mean we could be like limiting the amount of data that we have by taking some of it out, but my claim is stronger. My claim is that by using a random validation set, we could get totally the wrong idea about our model. Caribou wan na have a try. So if our data is imbalanced, for example, we can, if you're randomly sampling it, we can only one class in our validation set. So our fitted model - maybe that's true as well, so maybe you're trying to predict in a medical situation. Who's going to die of lung cancer, and that's only one out of a hundred people and we pick out a validation set that we accidentally have nobody that died of lung cancer. That's also true. These are all good niche examples, but none of them quite say like. Why could the validation set just be plain wrong, like give you a totally inaccurate idea of whether this is going to generalize, and so let's talk about, and the closest is, is what Tyler was saying about time closeness in time. The important thing to remember is when you build a model you're, always you always have a systematic error, which is that you're going to use the model at a later time than the time that you built it right, like you're, going to put it into production, by Which time the world is different to the world that you're in now, and even when you're building the model you're using data which is older than today anyway.
Right so there's some lag between the data that you're building it on and the data that it's going to. Actually be used on your life and a lot of the time, if not most, of the time that matters right. So, if we're doing stuff in like predicting who's going to buy toilet paper in New Jersey - and it takes us two weeks to put it in production and we did it using data from the last couple of years and by that time you know things may look Very different right, and particularly our validation, said if we randomly sampled it right and it was like from a four year period, then the vast majority of that data is going to be over a year old right and it may be that the toilet buying habits of Folks in New Jersey may have dramatically shifted. Maybe they've got a terrible recession there now and they can't afford a high-quality toilet paper anymore or maybe they know their paper. Making industry has gone through the roof and suddenly you know they they're buying what's more toilet paper, because it's so cheap or whatever right, so the world changes and therefore, if you use a random sample for your validation set, then you're actually checking. How good are you at predicting things that are totally obsolete now, but how good are you at predicting things that happened four years ago? That's not interesting, okay, so what we want to do in practice, anytime, there's some temper or peace is to instead say assuming that we've ordered it by time all right.
So this is old, and this is new. That's our validation set okay or if we, you know, I suppose actually do it properly. That's how a validation set. That's our test set, make sense right. So here's that training set - and we use that and we try and be able to model that still works on stuff. That's later in time than anything, the model was built on, and so we're not just testing generalization in some kind of abstract sense, but in a very specific time sense, which is it generalizes to the future? Could you pass it to Suraj please? So when we are, as you said, as you said, there is some temporal ordering in the data. So in that case, is it wise to take the entire full data for training, or only a few recent data set for validation, test or training training? Yeah. That's a whole other question all right. So how do you? How do you get the validation set to be good? So I build a random forest on all the training data. It looks good on the training data. It looks good on the oob right and this is actually a really good reason to have OB if it looks good on the OB that it means you're, not overfitting in a statistical sense right like it's, it's working well on a random sample, but then it looks Bad on the validation set, so what happened? Well, what happened was that you, you somehow failed to predict the future. You're only predicted the past and so Suraj had an idea about how we could fix. That would be okay.
Well, maybe we should just train so like maybe we shouldn't use the whole training set. We should try a recent period only and now it on the downside, we're now using less data, so we can create less rich models on the upside. It's it's more up-to-date data, and this is something you have to play around with most machine learning functions have the ability to provide a weight that is given to each road solve, for example, with a random forest rather than bootstrapping at random. You could have a weight on every row and randomly pick that row with some probability right and we could like say here's our like probability. We could like pick a curve that looks like that, so that the most recent rows have a higher probability of being selected. That can work really well yeah. It's it's something that you have to try and and if you don't have a validation set that represents the future compared to what you're training on you have no way to know which of your techniques are working. How do you make the compromise between amount of data versus recency of data, so what I tend to do is is when I have this kind of temporal issue, which is probably most of the time once I have something that's working well on the validation set. I wouldn't then go and just use that model on the test set, because the thing that I've trained on is now like, but you know, the test set is much more in the future compared to the training set.
So I would then replicate building that model again, but this time I would combine the training and validation sets together. Okay and retrain the model and at that point you've got no way to test against a validation set. So you have to make sure you have a reproducible, script or notebook that does exactly the same steps in exactly the same ways, because if you get something wrong, then you're going to find on the test set that you've you've got a problem. So so what
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
What I do in practice is, I need to know, is my validation set, a truly representative of the test set. So what I do is I build five models on the training set. I build five models on the training set and I try to have them kind of vary in how good I think they are right and then and then I score them. My five models on the validation set all right and then I also score them on the test set that so I'm not cheating. So I'm not using any feedback from the test set to change my hyper parameters, I'm only using it for this one thing, which is to check my validation set. So I get my five scores from the test set and then I check that they fall in a line. Okay and if they don't, then you're not going to get good enough feedback from the validation set. So keep doing that process until you're getting a line, and that can be quite tricky right. Sometimes the the test set. You know trying to create something: that's as similar to the real-world outcome as possible. It's difficult right and when you're it kind of in the real world, the same is true of creating the test set like the test set, has to be a close to production as possible, so like. What's the actual mix of customers that are going to be using this? How much time is there actually going to be between when you build the model and when you put it in production, how often you're going to be able to refresh the model? These are all the things to think about when you build that test set okay, so you want to say that first make five models on the training data, yeah and then dilute get a straight-line relationship change your validation and death set. You can't really change the test set generally, so this is assuming that the test sets given it changed to change the validation set.
So if you start with a random sample validation set and then it's all over the place - and you realize oh, I should have picked the last two months and then you pick the last two months.  Oh, I should have picked it, so that's also from the first of the month to the fifteenth of the month and they'll keep going until changing your validation set until you found a validation set, which is indicative of your test, set results, no sort of five models, Like he was started, maybe like just random data and average, and they just make it their own yeah, yeah, yeah yeah, maybe a exactly maybe yeah, I kind of five like not terrible ones, but you want some variety and you also particularly want some variety and like How well they might generalize through time so one that was trained on the whole training set one that was trained on the last two weeks, one that was trained on the last six weeks. One which used, as you know, lots and lots of columns and might have a fit a bit more yeah. So you kind of want to get a sense of like oh, if my validation set fails to generalize temporarily. I'd want to see that if it fell to generalize statistically I'd want to see that sorry, can you explain a bit more detail what you mean by change your validation set, so it indicates the test set like what does that look like so posit? So, let's take the groceries competition where we're trying to predict the next two weeks of grocery sales, so possible validation sets that Terrence and I played with was a random sample.
The last month of data, the last two weeks of data and the other one we tried was same day range one month earlier, so that the test set in this competition was the first to the 15th of August. Sorry, if this 15, that maybe the 15 to the 30th of August, so we tried like a random sample as four years. We tried the 15th of July to the 15th of August. We tried the 1st of August to the 15th of August and we tried the 15th of July to the 30th of July, and so there were four different validation sets we tried and so with random. You know our kind of results were all over the place with last month. You know they were like not bad, but not great. The last two weeks there was a couple that didn't look good, but on the whole they were good and same day range of months. Early they've got a basically perfect line. That's the part, I'm talking right there. What exactly are you comparing it to from the test site? I just confused what you're creating that graph so for each of those. So for each of my so I build five models right. So there might be like just predict the average. Do some kind of simple group mean of the whole data set? Do some group mean, over the last month of the data set, build a read on forests of the whole thing, build a random forest from the last three weeks on each of those I calculate the validation score and then I retrain the model on the whole training Set and calculate the same thing on the test set, and so each of these points now she tells me how about it ago, in the validation set.
How well did it go in the test set, and so if the validation set is useful, we would say every time the validation set improves. The test set should also score should also improve. Yes, so you just said rate ring dreaming rich rings in modeling on training and validations yeah. That was a step I was talking about here. So once I've got the validation score based on just the training set and then retrain it on the train and validation and check against history. Somebody else so just to clarify my test.
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Set you mean submitting it to kaibaland and checking the school if it's cattle, then your test set is Carol's leaderboard in the real world. The test set. Is this third data set that you put aside and it's that third data set that having it reflect real world production differences is the most important step in a machine learning project? Why is it the most important step?  If you screwed up creating the test set. That would be awful right because then you don't know. If you've made a mistake right, you try to build a model. You test it on the test set. It looks good, but the test set was not indicative of real-world environment, so you don't actually know if you better destroy the company right now. Hopefully, you've got ways to put things into production gradually, so you won't actually destroy the company, but you'll at least destroy your reputation at work right. It's like Oh Jeremy, tried to put this thing into production and in the first week the cohort we tried it on their sales halved and we're never better give Jeremy machine-learning job again right. But if Jeremy had used a proper test set then like he would have known.
Oh, this is like half as good. As my validation set said. It would be I'll, keep trying right and now I'm not going to get in any trouble. I was actually like. Oh Jeremy is awesome. He identifies ahead of time when there's going to be a generalization problem. Okay, so this is like this is something that kind of everybody talks about a little bit in machine learning classes, but often it kind of stops at the point where you learned that there's a thing in scikit-learn called make test trains flipped and it returns. These things and off you go right, but the fact that like or here's the cross-validation function right, so the fact that these things always give you random samples tells you that, like much, if not most of the time, you shouldn't be using them. The fact that random forest gives you an oo B for free, it's useful, but it only tells you that this generalizes in a statistical sense, not in a practice since right. So then, finally, there's cross-validation right, which outside of class you guys have been talking about a lot which makes me feel
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Somebody's been over emphasizing the value of this technique, so I'll explain what cross-validation is and then I explain why you probably shouldn't be using it most of the time so cross validation says: let's not just pull out one validation set, but let's pull out five say. So, let's assume that we're going to randomly shuffle the data first, all right. This is critical right. We first randomly shuffle the data and then we're going to split it into five groups and then for model number. One we'll call this the validation set and we'll call this. The training set: okay and we'll train and we'll check against the validation and we'll get some rmse R squared whatever and then we'll throw that away and we'll call this. The validation set and we'll call this. The training set and we'll get another score we'll do that five times and then we'll take the average okay. So that's a cross-validation average accuracy. So who can tell me, like a benefit of using cross-validation over a the kind of standard validation set? I talked about before: how could you pass the phone if you have a smokiness, an chosen course validation will make you solve with a data you have yeah, you can use all of the data. You don't have to put anything aside and you kind of get a little benefit as well in that, like you've, now got five models that you could ensemble together each one of used, which used 80 % of the data. So you know sometimes that ensemble lane can be helpful, I'm fun.
Could you tell me like what what could be some reasons that you wouldn't use? Cross-Validation, we have enough data, so we don't not want the validations and to be included in the model. Trainings process like okay yeah, I'm not sure the cross-validation is necessarily polluting the model. What would be a key like downside of cross-validation but like for deepening? If you have learned them P, be chosen, as annual Network will know the pictures it's more likely to predicative, as is right so sure, but if we, if we put aside some data each time in the cross-validation, can you pass it to Suraj, I'm not so worried About, like I don't think, there's like one of these validation sets is more statistically accurate, yes, Suraj Steven, will you be all fitting together late? I think that's what fun was worried about. I don't see why that would happen like each time we're fitting a model just 100. Each time we're fitting a model. We are absolutely holding in 20 percent of the sample right. So, yes, the five models between them have seen all of the data, but but it's kind of like a random forest independence, is a lot like a random first, each model has only been trained on a subset of the data. Yes, you should see. Please David largely received like it is deep load of time. Oh yes, exactly right, so we have to fit five models rather than one.
So here's a key downside number one it's time, and so, if we're doing deep learning - and it takes a day to run suddenly it takes five days or we need five GPUs. Okay. What about my earlier issues about validation sets Jona pass it over there. What's remaining was a so if you had like temporal data, wouldn't you be like shuffling when you e breaking that relation? Well, we could unravel it afterwards. We could reorder it like. We could shuffle get the training set out and then sort it by time like and like this, presumably there's a date column there. So I don't think I don't think it's going to stop us from building a model. Did you have with cross-validation your building? Five, even validation sets, and if there is some sort of structure that you're trying to capture in your validation sets of Mary, your test set you're, essentially just throwing that a chance to construct that yourself right. I think you're gon na say that I think you said the same thing as I'm gon na say, which is, which is that our earlier concerns about why random validation sets are a problem are entirely relevant here. Well, these validation sets a random. So if a random validation set is not appropriate for your problem most likely because, for example, of temporal issues, then none of these four validation set of five validation sets are any good they're all random right.
And so, if you have temporal data like we did here, there's no way to do cross, validation really or like probably no good way to do cross validation. I mean you're wan na, have your validation set be as close to the test set as possible, and so you can't do that by randomly sampling, different things. So so, as fone said, you may well not need to do cross validation because, most of the time in the real world, we don't really have that little data right unless your data is based on some very, very expensive labeling process or some experiments that take a Look cost a lot to run or whatever, but nowadays that's data. Scientists are not very often doing that kind of work. Some um, in which case this is an issue it must have assigned. So we probably don't need to, as nishan said, if we do do it, it's going to take a whole lot of time all right and then, as earnest said, even if we did do it and we took up all that time. It's like it was totally the wrong answer, because random validation sets are inappropriate for a problem. Okay, so I'm not going to be spending much time on cross validation, because I just I think it's an interesting tool to have it's easy to use. Sosuke learn has a cross validation thing. You can go ahead and use, but it's it's. It's not that often that it's going to be an important part of your toolbox. In my opinion, you'll come up some points. Okay, so that is validation tips.
So then the other thing we started talking about
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Last week and got a little bit stuck on because I screwed it up was tree interpretation, so I'm actually going to cover that again without the error and dig into it in a bit more detail. So can anybody tell me what tree interpreter does and how it does it? What do you remember it's a difficult one to explain. I don't think I did a good job of explaining it. So don't worry if you don't do a great job, but does anybody wan na have a go explaining it? Well? Okay, that's fine! So let's start with the output of tree interpreter. So if we look at a single model, a single tree, in other words here, is a single tree. Okay, so to remind us, the top of a tree is before there's been any split at all. So ten point one: eight nine is the average log price of all of the options in our training set. So I'm going to go ahead and draw right here, ten point: one: a nine eight nine is the average of all okay and then, if I go a couple of system less than or equal to 0.5, then I get ten point three, four, five. Okay. So, for this subset of 16,800 coupler is less than or equal to point five. The average is ten point three four five and then off the people with a couple of system less than or equal to point 5. We then take the subset, we're enclosure at less than or equal to two and the average there of rob.
Sale price is nine point: nine, five, five against nine point: nine five and then final step in our tree as model ID just for this group with no capitalist system with enclosed lesson or then let's just take model ID less than or equal to forty five. Seventy three - and that gives us ten point, two, two six. Okay, so then we can say all right, starting with ten point one: oh nine one, eight nine average for everybody in our training set for this particular tree, subsample of twenty thousand, adding in the capital decision or couple or less than a two point. Five increased. Our prediction by point one: five: six, so if we predict it with a naive model of just the mean that would have been ten point when I known adding in just the coupler decision, would have changed it to ten point three, four five. So this variable is responsible for a point: one: five: six increase in a prediction from that: the enclosure no decision was responsible for a point. Three nine five decrease the model. Id was responsible for eight point: two, seven, six increase until eventually that was our final decision. That is our prediction for this option of this particular sale price. So we can draw that as what's called a waterfall block right and what our four plots are. One of the most useful plots.
I know about and weirdly enough, there's nothing in Python to do them, and this is one of these things where there's this disconnect between, like the world of management, consulting and business, where everybody uses water for plots all the time and like academia who have no idea. What these things are, but like every time like you're looking at say here is last year's sales for Apple, and then there was a change in that iPhones increased by this amount max decreased by that amount and iPads increased by that amount. Every time you have a starting point in a number of changes and a finishing point, waterfall charts are pretty much always the best way to show it. So here our prediction for price based on everything, ten point: one: eight nine there was an increased blue means increase of 0.156, the coupler decrease of 0.395 or enclosure increase model ID of point two, seven six so decrease, but I increase decrease increase to get to our Final ten point: two: six: six, so you see how what a porch light works so with Excel 2016 its built-in you just click, insert waterfall chart and there it is. If you want to be a hero, create a waterfall chart package format, plot lab, put it on pip and everybody will love you for it. There are some like really crappy, gist's and manual notebooks and stuff around.
These are actually super easy to build, like you, basically do a stacked column plot, where the the bottom of this is like all white right like you can kind of do it, but if you can wrap that up or all and put the data the points in The right spots and color them nicely that would be totally awesome. I think you've all got the skills to do it and would make you know, be a terrific thing for your portfolio. So there's an idea could make an interesting cattle Colonel even like here's. How to build a waterfall plot can scratch and by the way I've been putting this up on yep, you can all use it. So in general, therefore, obviously going from the all and then going through each change, then the some both all of those is going to be equal to the final prediction. So that's how we could say if we were just doing a decision tree, then you know you're. Coming along and saying like how come this particular option, was this particular price? And it's like: well your prediction for it and like oh, it's because of these three things had these three impacts right so for a random forest. We could do that across all of the trees right. So every time we see coupler, we add up that change. Every time we see enclosure, we add up that change. Every time we see model, we add up that change, okay, and so then we combine them all together.
We get what tree interpreter does right, so you could go into the source code for tree interpreter right and it's not at all complex logic or you could build it yourself and you can see how it does exactly this. So when you go tree and predict with a random forest model for some specific option, so I've got a specific row here. This my zero index row. It tells you okay, this is the prediction the same as the random forest prediction bias. This is going to be always the same. It's the average sale price for for everybody for each of the random samples in the tree, and then contributions is the average of all so the total of all our contributions for each time. We see that specific column appear in a tree right. So last time I made the mistake of not sorting this correctly. So this time MP, dot arc sort, is a super handy function at sorts. It doesn't actually sort contribution zero. It just tells you where each item would move to if it were sorted so now by passing ID access to each one of the column, the level contribution. I can then print out all those in the right order. So I can see here, here's my column. Here's the level and the contribution, so the fact that it's a small version of this piece of industrial equipment meant that it was less expensive right, but the fact that was made pretty recently meant that was more expensive.
The fact that it's pretty old, however, made that it was less expensive right, so this is not going to really help you much at all with like a cattle styled situation where you just need predictions, it's going to help you a lot in a production environment or Even pre production right so like something which any good manager should you should do. If you say here's a machine learning model, I think we should use as they should go away and grab a few examples of actual customers or actual options or whatever and check whether your model looks intuitive, alright and if it says, like my prediction, is that you Know lots of if people are going to really enjoy this crappy movie.   Actually people like you did like that movie: okay, yeah. What's the second element of each temple, this is saying for this particular row. It was a mini and it was 11 years old and it was a hydraulic excavator track. 3 to 4 metric tons, so it's just feeding back and telling you it's, because this is actually what it was. It was these numbers, so I just went back to the original data to actually pull out the descriptive versions of each one.
Okay. So if we sum up all the contributions together and then add them to the bias, then that would be the same as adding up those three things: adding it to this and, as we know from our waterfall chart, that gives us our final prediction. This is a almost totally unknown technique and this particular library is almost totally unknown as well. So like it's. A great opportunity to you know show something that a lot of people like it's totally critical in my opinion, but but rarely none. So that's
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
That's kind of the end of the random forest interpretation piece and hopefully, you've now seen enough that when somebody says we can't use modern machine learning techniques because they're black boxes that are interpretive all you have enough information to say: you're full of right, like they're, extremely Interpretable and the stuff that we've just done, you know trying to do that with a linear model. Good luck to you, you know, even where you can do something similar with a linear model trying to do it. So that's not giving you totally the wrong answer and you had no idea it's a wrong answer. It's going to be a real challenge, so the last step we're going to do before we try and build up our own random forest is deal with this tricky issue of extrapolation. So in this case, if we look at our tree, let's look at the accuracy of our most recent trees. We still have you know a big difference between our validation score and our training score the actually, in this case, it's not too bad - that the difference between the oob and the validation is actually pretty close. So if there was a big difference between validation and olb, like I'd, be very worried about that, we've dealt with the temporal side of things correctly. Let's just have a look at, I think the most recent model here it was yeah, so there's a tiny difference right and so on Tagle at least you kind of need that last decimal place in the real world.
I probably stopped here, but quite often you'll see. There's a big difference between your validation score and your OB score. Now I will show you how you would deal with that, particularly because actually we know that the the oeob should be a little worse because it's using this less trees. So it gives me a sense that we should get to do a little bit better and so the reason we the way we should be able to a little bit better, is by handling the time component a little bit better. So here's the problem with random forests when it comes to extrapolation when you, when you've got a data set. That's, like you know, four got four years of sales data in it and you create your tree right and it says like oh if these, if it's in some particular store and at some particular item - and it is on special, you know: here's the average price and It actually tells us the average price you know over the whole training set, which could be pretty old right, and so when you then want to step forwards like what's going to be the price next month, it's never seen next month and and where else, with a Kind of a linear model: it can find a relationship between time and price where, even though we only had this much day when you then go and predict something in the future, it can extrapolate that, but a random forest can't do that.
There's no wave, if you think about it, for a tree to be able to say well next month, it would be higher still so there's a few ways to deal with this and we'll talk about it over the next couple of lessons. But one simple way is just to try to avoid using time variables as predictors. If there's something else, we could use that's going to give us a better. You know something that kind of a stronger relationship, that's actually going to work in the future. So in this case, what I wanted to do was to first of all figure out: what's the difference between our validation set and our training set like if I understand that difference between our validation set and our training set, then that tells me what are the predictors, Which which have a strong temporal component and therefore they may be irrelevant by the time I get to the future time period. So I do something really interesting, which is I create a random forest where my dependent variable is? Is it in the validation set right? So I've gone back and I've got my whole data frame with the training and validation altogether and I've created a new column Court is valid which I've set to one and then for all of the stuff in the training set. I set it to zero. That's about a new column which is just is this in the validation set or not, and then I'm going to use that as my dependent variable and build a random first. So this is a random forest not to predict price.
That protect is this in the validation set or not, and so, if your variables were not time dependent, then it shouldn't be possible to figure out if something's, in the validation set or not. This is a great trick in cattle records in cattle. They often won't tell you whether the test set is a random sample or not. So you could put the test set and the training set together, create a new column called, is test and see. If you can predict it, if you can, you don't have a random sample, which means you have to come and figure out how to create a validation set from it right, and so, in this case I can see I don't have a random sample, because my validation Set can be predicted with a 0.9999 r-squared, and so then, if I look at future importance, the top thing is sales ID, and so this is really interesting. It tells us very clearly. Sales ID is not a random identifier, but probably it's something. That's just set consecutively as time goes on. We just increase the sales ID so elapsed. That was the number of days since the first date in our data set. So not surprisingly, that also is a good predictor. Interestingly machine ID.  Don't just look at the order. Look at the value so 0.7. 0.1. 0.0. 7.00. Okay, stop right! These top three are hundreds of times more important than the rest right.
So let's next grab those top three right and we can then have a look at their values both from the training set and in the validation set, and so we can see. For example, sales ID on average is divided by thousand on averages: 1.8 million in the training set and 5.8 million in the validation set right. So you'd like you, can see just confirm like okay they're very different, so let's drop them okay. So after I drop them. Let's now see, if I can predict whether something's in the validation set, I still can with 0.98 pass quit. So once you remove some things, then other things can like come to the front and it now turns out. Okay, that's not surprisingly age. You know things that are old. Are you know more likely, I guess to be in the validation set, because there's you know earlier on in the training set, yet they can't be old. Yeah yeah made same reason. So then we can try removing those as well and so once we let's see where do we go up here, yeah, so what we can try doing is. We can then say: alright, let's take the saleslady. So, let's machine ID from the first one, the age year, made sale day of year from the second one and say: okay: these are all time dependent features, so I still want them in my random forest if they're important right, but if they're not important, then taking Them out, if there are some other long-term dependent variables that that work just as well, that would be better right, because now I'm going to have a model that generalizes over time better. So here I'm just going to go ahead and go through each one of those features and drop each one, one at a time: okay, retrain, a new random forest and print out the score okay.
So before we do any of that, our score was point. Eight eight for our validation versus point eight 900 B, and you can see here when I remove sales ID. My score goes up and this this is like what we're hoping for we've removed a time dependent variable. There were other variables that could find similar relationships without the time dependency, so removing it cost our validation to go up now. Oob didn't go up right because this is genuinely statistically you're useful, predictor right, but it's a time dependent one when we have a time dependent, validation set. So this is like really subtle, but it can be really important right. It's trying to find the things that gives you a generalizable time across time prediction and here's how you can see it. So it's like okay, we should remove sales ID for sure right, but sale elapsed didn't get better okay, so we don't want that machine. Id did get better right from eight eight, eight to eight nine three right. So it's actually quite a bit better age got a bit better. You made got worse. Sale day of year, got a bit better okay. So now we can say alright, let's get rid of the three where we know that getting rid of it actually made it better. Okay and, as a result, look at this we're now up to nine one five. Okay, so we've got rid of three time dependent things and now, as expected, validation is better than our Obi okay.
So that was a super successful
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Approach there right and so now we can check the feature importance and let's go ahead and say all right. That was pretty damn good. Let's now leave it for a while, so give it a hundred and sixty trees, don't show it and see how that goes. Okay, and so, as you can see like, we did all of our interpretation, all of our fine-tuning, basically with smaller models subsets and at the end we run the whole thing. You actually still only took 16 seconds, and so we've now got an RMS see of 0.21. Okay, so now we can check that against cattle again we can't we. Unfortunately, this older competition we're not allowed to enter anymore to see how he would have gone. So the best we can do is check whether it looks like we could have done we're all based on their validation set, so it should be in the right area and yeah. Based on that, we would have come first. Okay, so you know. I think this is an interesting series of steps right, so you can go through the same series of steps in your cattle projects and, more importantly, your real-world projects. So one of the challenges is once you leave this learning environment, suddenly you're, surrounded by people who they they've had not have enough time. They always want you to be in a hurry.  You need to find the time to step away right and go back, because this is a genuine real-world, modeling process you can use and it gives, when I said, kids world-class results.
I mean it right like this guy who won this lista costs. Sadly, he's passed away, but he is the top kaggle competitor of all time like he. He won, I believe, like dozens of competitions, so we can get a score even within cuy of him, then we are doing really really well okay. So, let's take a five-minute break and we're going to come back and build our own random. First, I just wanted to clarify something quickly. A very good point during the break was going back to the change in R squared between here, and here it's not just due to the fact that we removed these three predictors. We also went reset our F samples right, so they actually see the impact of just removing. We need to compare it to the final step earlier, so it's actually compared to 907. So removing those three things took us from 907 nine one, five. Okay, so I mean - and you know in the end of course, what matters is our final model that yep just to clarify? Okay, so um? Some of you have asked me about writing your own random forests from scratch. I don't know if any of you have given it a try. Yet my original plan here was to do it in real time and then, as I started to do it, I realized that that would have kind of been boring because to you, because I screw things up all the time. So, instead we might do more of like a walk through the code together. Just as an aside, this reminds me talking about the exam hammock.
She somebody
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Asked on the forum about like what what can you expect on the exam? The basic plan is to make it a exam, be very similar to these notebooks, so it'll probably be a notebook that you have to you know, get a data set, create a model. Trainer feature importance whatever right and the plan is that it'll be open, block open Internet. You can use whatever resources you like. So basically, if you're entering competitions, the exam should be very straightforward. I also expect that there will be some pieces about like here's. A partially completed random forest or something you know finish finish, writing this step here or here's a random forest implement feature importance or in you know, implement one of the things we've talked about, so it open it. You know the exam will be much like what we do in class and what you're expected to be doing during the week. There won't be any define this or tell me the difference between this word and that word or whatever, there's not going to be any rote learning it'll be entirely like. Are you an effective machine learning practitioner ie can use the algorithms do you know? Can you create an effective validation set and can you can you create parts of the algorithm implement them from scratch? So it'll be all about writing code. Basically, so if you're not comfortable writing code to practice machine learning, then you should be practicing that all the time, if you are comfortable, you should be practicing that all the time, also whatever you're doing write code to implement random to do machine learning.
Okay, so I I kind of have a particular way of writing code and I'm not going to claim it's the only way of writing code, but it might be a little bit different to what you're used to and hopefully you'll find it at least interesting. Creating implementing random forest
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Algorithms is actually quite tricky, not because the codes tricky like generally speaking, most random first algorithms are pretty conceptually easy at all that, generally speaking, academic papers and books have a knack of making them look difficult, but they're not difficult. Conceptually, what's difficult is getting all the details right and knowing and knowing when you're right, and so in other words, we need a good way of doing testing. So if we're going to reimplemented a we want to create a random forest in some different framework, different language, different operating system, you know I would always start with something that does exist right. So in this case, we're just going to do is learning its exercise. Writing a random forest in Python, so for testing, I'm going to compare it to an existing random forest implementation. Okay, so that's like critical anytime you're doing anything involving like non-trivial amounts of code and machine learning, knowing whether you've got it right or wrong is kind of the hardest fit.  How do I figure out that I screwed it up right and then much to my surprise from time to time I actually get something right and then I can move on okay, but most of the time I get it wrong.
So, unfortunately, with machine learning there's a lot of ways you can get things wrong that don't give you an error, they just make your result like slightly less good, and so that's that's what you want to pick up so, given that I want to kind of compare It to an existing implementation, I'm going to use our existing data, set our existing validation set and then to simplify things. I'm just going to use two columns to start with, so let's go ahead and start writing a random forest. So my way of writing nearly all code is top-down, just let my teaching, and so, if I top-down I start by assuming that everything I want already exists, so in other words the first thing I want to do, I'm going to call this a tree ensemble right. So, to create a random forest. The first question I have is: what do I need to pass in right? Why I need to initialize my random first, so I'm going to need some independent variables, some dependent variable pick how many trees I want I'm going to use the sample size parameter from the start here. So how big you want each sample to be, and then maybe some optional parameter of. What's the smallest leaf size, okay, for testing, it's nice to use a constant random seed, so we'll get the same result each time. So this is just how you set a random seed. Okay, maybe it's worth mentioning is for those of you, unfamiliar with it.
Random number generators on computers aren't random at all, now, actually constitute a random number generators and what they do is given some initial starting point. In this case 42, a pseudo-random number generator is a mathematical function that generates a deterministic, always the same sequence of numbers such that those numbers are designed to be as uncorrelated with the previous number as possible: okay and as unpredictable as possible and as uncorrelated as possible. With something with a different random seed, so the second number in in the sequence, starting with 42, should be very different to the second number, starting with 41 and generally they involve kind of like taking. You know you know using big prime numbers and taking mods and stuff like that, it's kind of an interesting area of math. If you want real random numbers, the only way to do that is again. You can actually buy hardware called a hardware. Random number generator that'll have inside them like a little bit of some radioactive substance and and like something that detects how many things it's spitting out or you know there will be some hardware thing any current system time is. Is it a valid random, like random number generation, no sense, so that would be for maybe for a random seed right. So this thing of like what do we start the function with so one of the really interesting areas is like in your computer.
If you don't set the random seed, what is it set to and yeah? Quite often, people use the current time for security, like obviously, we use a lot of random number stuff for security stuff like if you're generating an SSH key, you need some. It needs to be random, it turns out, like you know, people can figure out roughly when you created a key like they could look at like oid RSA has a timestamp and they could try. You know all the different nanoseconds starting points for a random number generator around that time, step and figure out your key. So in practice, a lot of like really random high randomness requiring applications actually have a step. That say, please move your mouse and type random stuff at the keyboard for a while, and so it like gets you to be a sort. That's called entropy to be a source of entropy. Other approaches is they'll. Look at, like you know the hash of some of your log files, or you know stuff like that. It's a really really fun area. So, in our case our purpose actually is to remove randomness. So we're saying: okay generate a series of pseudo-random numbers starting with 42. So it always should be the same. So if you haven't done much stuff in Python oo, this is a basically standard idiom. At least I mean I write it this way most people don't, but if you pass in like 1, 2 3 4 5 things that you're going to want to keep inside this object, then you basically have to say self dot x equals x, self dot y equals Y self, that sample equals sample right, and so we can assign to a tuple from at a port.
So you know okay. This is like my way of coding. Most people think this is horrible, but I prefer to be able to see everything at once, and so I know in my code anytime, I see something that looks like this. It's always all of the stuff in the method being set. If I did it a different way than half the codes now come off the bottom of the page, and you can't see it all right so um, so that was the first thing I thought about is like okay to create a random forest. What information do you need, then? I'm going to need to store that information inside my object, and so then I need to create some treats. I had a random forest is something that creates and is something that has some trees.  Comprehension to create a list of trees, how many trees do we have or you put n trees trees? That's what we asked for so range entries gives me the numbers from 0 up to n trees. Minus 1. Ok! So if I create a list comprehension that lips through that range calling create tree each time, I now have entries trees and also, I add, to write that I didn't have to think at all, like that's all like obvious and so I've kind of delayed. The thinking to the point where it's like well wait: we don't have something to create a tree. Okay, no worries, but let's pretend we did if we did. We've now created a random forest. Okay we'd still need to like do a few things. On top of that, for example, once we have it, we need a predict function.
So, okay! Well, let's write a prediction function. How do you predict in a random forest? Can somebody tell me either based on their own understanding or based on this line of code? What would be like your one or two-sentence answer? How do you make a prediction in a random forest, positive, Spencer uh? You would want to over every tree for your like the row that you're trying to predict on average the values that your that each tree would produce for that book. Good, and so you know that's a summary of what this says right so for a particular row. Right or maybe this is a number of rows - go through each tree calculate its prediction. So here is a list comprehension that is calculating the prediction for every tree for X. I don't know if X is one row or multiple rows, it doesn't matter right as long as as long as trade, I predict works on it and then once you've got a list of things, a cool trick to know is you can pass numpy dot mean a Regular non mum pie list, okay and it'll - take the mean you just need to tell it: access equals zero means everage it across the lists. Okay, so this is going to return the average of that predict for each tree, and so I find list comprehensions. Allow me to write the code in the way that the brain was like. You could take the word Spencer said and like translate them into this code, or you could take this code and translate them into words.
Like the one Spencer said right, and so when I write code, I want it to be as much like that as possible. All right, I want it to be readable and so hopefully you'll find like. When you look at the past AI code, you can understand how to journey through X. I try to write things in a way that you can read it and like it kind of turn it into English in your head. So if I say correctly, that predict method is recursive, it's no, it's calling trade predict and we haven't written a tree. Yet so self trees is going to contain a tree object. So this is tree. Ensemble dot predict and inside the trees is a tree, not a tree ensemble. So this is called an trade product, not tree ensemble, dotted it the question. Okay, so we nearly finished riding around and for our seventh week. All we need to do now is write, create tree right, so um, based on this code here or on your own understanding, of how we create trees in a random forest. Can somebody tell me, let's take a few seconds? Have a raid have to think and then I'm going to try and come up with a way of saying how do you create a tree in a random forest? Okay, who wants to tell me yes, okay, let's tireless work cluster, you take your you're, essentially taking a random sample or of the original data and then you're just it just constructing a tree. However, that happens so construct a decision tree like a non random tree from a random sample of the data. Ok, so again, like we've delayed any actual thought process.
Here, we've basically said: ok, we could pick some random IDs. This is a good trick to know. If you call NP random permutation passing in an inch it'll give you back a randomly shuffled sequence from zero to that inch right, and so then, if you grab the first n items of that, that's now a random subsample. So this is not doing bootstrapping we're not doing sampling with replacement here, which i think is fine. You know for my random forest, I'm deciding that it's going to be something where we do subsampling, not bootstrapping, ok, so here's a good line of code to know how to write because it comes up all the time like I find in machine learning most algorithms. I use are somewhat random and so often I need some kind of random sample. Can you pass that tartaric entry? Won't they give you 1 1 extra, because the easier it will go from 0 to length? No. So this will give you if lens self dot y is of size n. This will give you n a sequence of length, n, so 0 to n minus 1 and then from that I'm picking out self dot sample size, so the first sample size. Ladies, I have a comment on bootstrapping. I think this method is better because we have transfer giving more weights to each observation, or am I thinking wrong? I think you proposed wrapping. We could also give weights, I mean we single observations more than they are like without one thing that weights, because I'm bootstrapping with with replacement, we can have a single observation and dr. pitz of it yeah the same tree yeah.
It just feel weird, but I think the actual theory or empirical results backs up higher intuition that it's worse it'd be interesting. To look look back at that. Actually, personally, I prefer this because I feel like most of the time we have more data than we want to put a tree at once. I feel like back when Bremen created random forests. It was 1999, it was kind of a very different world. You know where we pretty much always wanted to use all the data we had, but nowadays I would say: that's generally not what we want. We normally have too much data, and so what people tend to do is they'll like fire up a spark cluster and they'll run it on hundreds of machines when it makes no sense, because if they had just used a subsample each time they could have done it. On one machine and like the the overhead of like spark, is a huge amount of i/o overhead, like I know you guys are doing distributed computing now, if you, if you've, looked at some of the benchmarks, yeah yeah exactly so, if you do something on a single Machine, it can often be hundreds of times faster, because you don't have all this this i/o overhead. It also tends to be easier to write. The algorithms like you can use like SK, learn easier to visualize and cheaper, so forth. So, like I almost always avoid distributed computing and I have my whole life like even 25 years ago, when I was studying in machine learning, I you know still didn't use clusters because I so I always feel like whatever I could do with a cluster now I Could do with a single machine in five years time, so one of us focus on always being as good as possible with the single machine. You know and that's going to be more interactive and more iterative and work for me.
So, ah, okay, so so again, we've like delayed thinking to the point where we have to write decision tree, and so hopefully you get an idea that this top-down approach, that goal is going to be that we're going to keep delaying thinking so long that that we Delay it forever like, like eventually we've somehow written the whole thing without actually having to think right, and that's that's kind of what I need is I'm kind of slow right. So this is why I write code this way and notice, like you, never have to design anything in. You just say hey what, if somebody already gave me the exact API I needed, how would I use it? Okay and then and then okay to implement that next stage,
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
What would be the exact API? I would need to implement that right. You keep going down until eventually you're like oh, that already exists. Okay, so this assumes we've got a class port decision tree, so we're going to have to create that so a decision tree is something so we already know what we're going to have to pass it because we just passed it right. So we're passing in a random sample of X's a random sample of Y's um uhh indexers is actually so. We know that down the track, so I've got a plan a tiny bit. We know that a decision tree is going to contain decision trees which themselves contain decision trees, and so, as we go down the decision tree there's going to be some subset of the original data that we've kind of got, and so I'm going to pass in the Indexes of the data that we're actually going to use here - okay, so initially it's the entire random sample all right, so I've got the whole team. I've got the whole range and I turn that into an array. So that's 0, the indexes from 0 to the size of the sample and then what is passed down the mean left side. So everything that we got for constructing the random forest where to pass down the decision tree except, of course, num trees, which is irrelevant for the decision tree so again, now that we know that's the information we need, we can go ahead and store it inside this Object, so I'm pretty likely to need to know how many rows we have in this tree, which I generally call n.
How many columns do I have, which I generally call C. So the number of rows is just equal to the number of indexes well given, and the number of columns is just like. However, many columns there are in our independent variables, so then we're going to need this value here. We need to know for this tree. What's its prediction right, so the prediction for this tree is the mean of our dependent variable or those indexes which are inside this part of the tree, alright, so at the very top of the tree, it contains all the indexes right, I'm assuming that by the time We've got to this point. Remember: we've already done the random sampling right. So when we talk about indexes we're not talking about the random sampling to create the tree, we're assuming this tree now has some random sample inside decision tree. This is this: is the one of the nice things right inside decision tree hole, random sampling, things gone right that was done by the random forest right. So at this point, we're building something. That's just a plain old decision tree. It's not in any way a random sampling, anything, it's just a plain old position tree right, so the indexes is literally like which subset of the data that we got to so far in this tree and so at the top of the decision tree. It's all the data right, so it's all of the indexes, okay, so all of the indexes.
So this is therefore all of the dependent variable that are in this part of the tree, and so this is the value mean of that. That makes sense anybody could be any questions about about that.  I mean all P experiments, okay, yeah sure, so so, quick so quick over P prenup would be helpful, great yeah, okay, who is done object-oriented programming in some programming, language, okay, so you've all used actually lots of object-oriented programming in terms of using existing classes right.  So fit is a method you can tell, because it's got parentheses after it right where else yeah, oh I'll, be score, is a property or an attribute doesn't have parentheses after it? Okay, so inside an object there are kind of two kinds of things there: the functions that you can call. So you have object, dot, function, parentheses, arguments or there are the properties or attributes you can grab, which is object, dot and then just the attribute name, no parentheses. So when and then the other thing that we do with objects, is we create them? Okay, we pass in the name of a class and it returns us the object and you have to tell it all of the parameters necessary to get constructed.
So let's just copy this code and see how we're going to go ahead and build this. So the first step is we're not going to go. N equals random forest regressor. We're going to go M equals tree ensemble we're creating a classical tree ensemble and we're going to pass in various bits of information. Okay, so maybe we'll have ten trees sample size of a thousand or maybe a min leaf of three okay, and you can always like choose to name your admits or not. So when you've got quite a few, it's kind of nice to name them so that just so we can see what each one means. It's always optional. So we're going to try and create a class that we can use like this and then the notional we're going to bother with dot fit because we've passed in the X and the y right like in scikit-learn. They use an approach where, first of all, you construct something without telling it what they did here is, and then you pass in the day we're doing these two steps at once. We're actually passing in the data right and so then, after that we're going to be going and dot so we're going to go. Creds equals m predict passing in maybe some validations there. Okay, so we that's that's the API, we're kind of creating here. So this thing here is called a constructor, something that creates an object is called a constructor and Python. There's a lot of ugly hideous things about Python, one of which is they it uses.
These special magic method, names underscore underscore init underscore underscore - is a special magic method. That's caught, it's called when you try to construct a class. So when I call tree ensemble parentheses, it actually calls tree ensemble dot, they see people say dunder init, I kind of hate it, but anyway timed it. You know double underscore in it double underscore dunder init. So that's why we've got this method called dunder, init.   So so that's why you always see this, and in fact I can immediately see here. I have a bug, anybody see the bug in my predict function. I should have so right. I like it always do it right. So anytime.
Oh okay to a programming language, but the older languages like Python often did this because they kind of needed to they started out not being oo and then they kind of added oo in a way that was hideously ugly, so Perl, which predates plaything by a little Bit kind of, I think, really came up with this approach and unfortunately other languages of that era stuck with it. So you have to add in this magic self, so the magic self. Now, when you're inside this class, you can now pretend, as if any property name you like exists, so I can now pretend there's something called self dot X. I can read from it. I can write to it right, but if I read from it - and I haven't yet written to it, I'll get an error, so the stuff that's passed to the constructor gets thrown away by default. Like there's nothing that like says you need to rip, this class needs to remember what these things are, but anything that we stick inside self is remembered for all time. You know, as long as this object exists, you can access it. It's remembered so now that I've gone. In fact: let's do this right so that let's create the tree ensemble class and let's now instantiate it. Okay, of course, we haven't got X. We need to call X, train y trade. Ok decision tree is not defined. So, let's we had a really minimal decision tree there we go okay, so here is enough to actually instantiate our tree ensemble. Okay, so we have to find the inert for it. We have to find the inert for decision tree.
We need decision trees in it to be defined because inside our ensemble in it they're called self directory and then self create tree called the decision tree constructor and then decision tree constructor basically does nothing at all other than save some information right. So at this point we can now go m dot. Okay, and if I press tab at this point, can anybody tell me what I would expect to see press it to Taylor tension? Could you possibly say like we would see a drop-down of all available methods for that class? Okay, it would be in this case. So if M is a tree ensemble, we would have create tree and predict okay, anything else. What oh yeah, as well as earnest, whispered to variables as well yeah, so that the variable could made a lot of things well attributes, so the things that we put inside self. So if I hit tab right there, they are right, as Taylor said, there's create tree. There's predict and then there's everything else we put inside so all right. So if I look at m dot min leaf, if I hit shift enter, what will I see yep? The number that I just put there I put in leaf is three so that went up the air dam in leaf. This here is a default argument such as, if I don't pass anything it'll be five, but I did pass something right, so three self dot min leaf here. Is it going to be equal to min leaf yeah, so something which like because of this rather annoying way of doing oh, oh, it does mean that it's very easy to accidentally forget so do that right? So if I don't assign it to self dot min leaf right, then I get an error, and so here tree ensemble doesn't happen in leaf right.
So how do I create that attribute? I just put something in it: okay, so if you want to like, if you don't know what a value of it should be yet, but you kind of need to be able to refer to it, you can always feel like self dot min leaf equals. None! That's at least there's something you can read check for numbness and not have an error great now, interestingly, I was able to instantiate tree ensemble, even if I predict refers to a method of decision tree that doesn't exist, and this is actually something very nice about the Dynamic nature of Python is that because it's not like compiling it, it's not checking anything unless you're using it right. So we can go ahead and create decision to predict later and then our our instantiated object will magically start working right. It doesn't actually look up that functions that methods details until you use it, and so it really helps with top-down programming. Okay, so when you're inside a class definition, in other words you're at that indentation level, you know indented one in so these are all class definitions. Any function that you create, unless you do some special things that we're not going to talk about yet is automatically a method of that class, and so every method of that class magically gets a self passed to it.
Ensemble, we could call em create tree and we don't put anything inside those parentheses, because the magic self will be passed and the magic self will be whatever M is okay, so m dot create tree returns a decision tree just like we asked it to right. So m dot create tree dot. E excess will give us the self ID access inside the decision tree okay, which is set to NP dot. A range range self dot sample size Y is data scientists. Do we care about object-oriented programming, because a lot of the stuff you use is going to require you to implement stuff with oo P? For example, every single pytorch model of any kind is created with olp. It's the only way to create by torch models. Um good news is what you see here is the entirety of what you need to know. So you, this is all you need to know. You need to know to create some in code in it to assign the things to the pasta in it to something call it self and then just stick the word self after it give your methods, okay, and so the nice thing is like now to think. As an AOP programmer is to realize you don't now have to pass around X Y sample size and min leaf to every function that uses them by assigning them to attributes itself they're now available like magic all right. So this is why our peas super handy. If you're, particularly, I started trying to create a decision tree initially without using oot and try to like keep track of like what that decision tree was meant to know about.
It was very difficult. You know where else with our P, you can just say even side. The decision tree - you know self indexes equals this and everything displace okay, okay, that's great! So we're out of time. I think that's that's great timing, because there's an introduction, 200 P, but this week you know next class, I'm going to assume that you can use it right, so you should create some classes. Instantiate some classes, look at their methods and properties. Have them call each other and so forth until you feel comfortable with them, and maybe for those of you doesn't haven't done our P before you can find some other useful resources. You could flop them onto the wiki thread so that other people know what you find useful right. Thanks. Everybody


By the end of this blog post, you should be able to:
To fully understand what a display property is, we must first understand how we style a webpage. When creating a webpage, we use styling documents known as CSS (Cascading Style Sheets), which enable us to arrange and add colour to a website and its content. Within these styling sheets, we utilise properties that specify exactly how this content will be arranged and coloured. One such property we use regularly while creating our webpages styling is the ‘display’ property. As the name states, the display property helps dictate exactly how this content will be displayed. Taking this concept further, all properties within our styling document are followed by a value. These property/ value pairs sit within a block of information, which links into our content sheet/ HTML document by the way of ‘selectors'. Using a combination of selectors, properties and values, we can stylise our webpage in any way, shape or form imaginable! See the code and resulting output below for a visual representation of the concepts just discussed.

This is a blue h2 element

Shown above is a small portion of code we would likely see within our CSS document. In this case, we have set the h2 element within our webpage to be stylised. The property has been set to colour and will affect the colour of all h2 elements on our website. The value has been set to a colour of blue.
This is a blue h2 element
Shown above is the resulting input from the code we just looked at. As desired, all h2 elements will now be displayed blue.
We will now focus on one-such property/ value paring known as ‘display inline’. When display inline is declared within our styling sheet, the desired HTML element will be rendered explicitly as an inline-level element. When we refer to an element as inline, we state that the element will only take up space around its content. This is different to a block-level element which always takes up the full width of the webpage, regardless of if content. Another main difference between the two, is that we can change the width and height of block-level elements, but not for inline. With this in mind, some HTML elements are either set as inline or block by default.

Using CSS stylings we are also able to change these default settings. See the code and resulting output below for a visual representation of the concepts just discussed.

The following span is a inline element

Shown above is a brief sentence which demonstrates how a span element will be displayed. In this case, we have wrapped a span element around the words “inline element.” If the desired result is accomplished, a yellow highlight will be shown around the words “inline element.”
The following span is a inline element
Shown above is the resulting output of our code. As expected the words “inline element” have been displayed with a yellow coloured highlight. As the span is an inline element, only the content that fits between its opening and closing tags will be highlighted.

The following paragraph is a
block-level element

Shown above is a brief sentence which demonstrates how a paragraph (p) element will be displayed. In this case, we have wrapped a paragraph element around the words “block-level element.” If the desired result is accomplished, the words “block-level element” will be pushed away from the rest of the text and displayed on its own line. The resulting output should also be highlighted a bright yellow colour and take up the whole width of the example.
The following paragraph is a block-level element
Shown above is the resulting output of our code. As expected the words “block-level element” have been pushed away from the rest of the sentence and are now seated on their own line. Our yellow highlight is also displayed across the whole width of the example.
Taking the concepts we have now learnt, let’s now look at the inline-block property/ value paring within CSS. When the inline-block property/ value paring is declared, we will be able to specify the height and width of the element. We will also be able to maintain the inline properties ability to sit next to other content and not create a new line as a block-level element would. See the code and resulting output below for a visual representation of the concepts just discussed.

The last two words in this sentence are styled with display inline-block

Shown above is a brief sentence which demonstrates how the display inline-block declaration will be displayed. In this case, we have wrapped the inline-block property around the words “display inline-block.” If the desired output is achieved, we will have a small yellow box around the last three words in this text.
The last two words in this sentence are styled with display inline-block
Shown above is the resulting output of our code. As expected the words “display inline-block” are now sitting inside a small yellow box.


Your address goes here
CONTACT BUSINESS CASUAL
Phone:123.456.7890
<EMAIL_ADDRESS>Address:3481 Melrose PlaceBeverly Hills, CA 90210
CONTACT FORM


The politics of de-icing and whether to use that Frozen CD | Jabbering Journo column
by Nicola AdamSo we tend to think of ourselves as a bit more resilient up north (hard), what with us all travelling on terrible Pacer trains, eating gravy with everything and surviving, largely, without Pret a Manger every three metres.
But when winter really kicks in we all become hugely hopeless in the face of plummeting temperatures.
No matter how frequently winter comes around (every year - maybe we should launch a crowd-funder to make it one in three?) we are all equally useless at defrosting our windscreen and at finding more than one glove when required.
Or maybe that’s just me.
READ MORE JABBERING JOURNO: Heal me with all the woo-woo
When it comes to windscreen defrosting, the wide variation in techniques is a minefield.
A quick office poll saw CDs as the number one easily-available scraping items, with credit cards a close second. (Survey based on asking three tired-looking colleagues)
This is definitely an argument for old technology - have you ever tried to scrape your car with a digital playlist?
And if you use online payment systems you’ll struggle even if you attempt to use your phone instead.
The swizziest of cars cannot avoid the frozen screen conundrum and it is a basic human fact that no matter how organised you think you are, when you are stranded late at night in the freezing cold and dark after work or when you are on an early shift, you will NOT have convenient de-icer with you and instead you are negotiating the minefield. ..
Can I sacrifice the soundtrack of Frozen 2 to the cause? (Answer -Let it go);
Will I lose my credit rating if I use my Barclaycard to clear my wing mirrors? (no);
Will warm water crack my screen? (Maybe) .
I’ve personally tried every hack going.
My top tip is rubbing a potato across your windscreen the night before - apparently they contain glycerol like anti-freeze and can help in prevention.
The only problem is that, should the sun break out, your screen becomes mashed potato.
But at least, with added gravy, it doubles as dinner.
READ MORE COLUMNISTS: Who's the daddy - Daughter is 20


Warning Better check yourself, you're not looking too good.
Welcome to Your Dashboard
This is Your Dashboard , where you can see all the most recent activities on your collections and exhibitions. Do you know that you can start exploring the media by typing letters on your keyboard? try it now!


<DASHBOARD>
will be replaced with the title of dashboard.<PANEL>
will be replaced with the title of this panel.<TITLE>
will be replaced with either the title of this panel or if it doesn't have a title it will be replaced with the title of dashboard.


Welcome to MasterMind’s documentation!¶
A project to explore nm MasterMind solution strategies, where m is the number of slots and n is the number of colors. The 64 MasterMind is the classic game. In the current state, the project only explores interactive strategies, but the code is prepared to handle static solutions as well and should support solution strategies for generic n and m.
Note on the dependencies: The game module uses the standard library only. However, some of the solvers use numpy, although at this stage, there is no measurable performance benefit over python lists - numpy is not essential and I may decide to throw it out again. Other than that, just the standard library. Oh, and it’s Python 3.5.2 but it should really work for any Python 3 environment.
This project is an exploration into solving strategies for MasterMind.
It’s core is the game.Game
class, which defines a nm Mastermind game
with m slots and n colors and provides certain methods to assist writing
solution strategies.
Simple use¶
For the impatient: here’s a quick overview of how to use this project. Normally, all you have to do is import and run, like this:
from MasterMind.game import Game
g=Game()
h=Game(slots=5,colors='Ruffy Nami Sanji Zorro Robin Frankie Lysop Trafalgar')
This will generate a 64 game g and a 85 game h with a challenge code each and with the corresponding methods and properties.
For more information about how to use this library, see the API Documentation.
How it works¶
MasterMind is essentially a finite code breaking game and each strategy needs to know the game’s properties and an evaluator plus some convenience methods and properties to generate solutions or translate codes to colors. The game can be played on the console:
in: g.evaluator('abcd')
out: (0,2)
in: h.evaluator('abcde')
out: (2,2)
Solution strategies¶
The folder solvers contains a number of strategies to solve
MasterMind. It’s actually an invitation to improve upon them and to suggest or
try new strategies. My goal was to be general both in the game.Game
class and in the solvers. Each solver takes an instantiated
game as argument and finds the solution for it. Solvers can also be written for
what’s known as static MasterMind. In static MasterMind, the codebreaker tries
to provide (in one go) a set of guesses that, together with the evaluations,
will allow her to break the code in the next guess.
Literature¶
There is a lot of literature on the game and I only want to present some pointers: Wikipedia has a great entry for MasterMind: Mastermind (board game). This paper provides a good overview over the scholarly literature: Playing Mastermind with Many Colors.


List of Authors
Author name
Books
Update option
Delete option
Books
Update author
Delete author
Create


Emergency Unit
The Emergency Unit at B C Roy Technology Hospital is located on the ground floor of the hospital, close to the main entrance and the portico of the hospital, where the ambulance can bring a patient. The Unit provides emergency services 24x7. The Medical Officer on-duty attends the patient brought to the emergency, and is supported by Nursing and Paramedical Staff.
The unit is equipped with emergency life saving drugs, and several modern monitoring equipments including:
It also is supported by 24 Hours On-Call Pathology and X-Ray Lab Facility. 2 additional beds are available for patients under observation. Specialist Services are available to a limited extent.
Facilities
Out Patient Department (OPD):
The OPD provides Outdoor Treatment to the beneficiaries in the campus community on all working days, excluding Sundays and the Institute / National Holidays. .
The regular hours are:
Monday to Saturday- 9 AM to 12 Noon and 4 PM to 7 PM
Indoor Wards
Bed Strength- 32 ( Including 2 Bedded ICU & the Isolation Wards).
Considering the need of the hospital, very recently an Operation Theatre (OT) has been added, by renovating a part of the Hospital building. The OT is now fully equipped and will be soon be functional
Lab Facilities
Pathology Lab
B C Roy Technology Hospital Pathology Laboratory is a reasonably well-equipped lab. It provides the routine Blood, Urine and Stool Examination services. In addition, the tests that the lab is equipped to conduct are:
Malaria, Dengue, Hepatitis B, RTT/ WIDAL, HIV1 and 2, Hepatitis B (HBsAg), Anti- HCV, VDRL, RA Factor, ASO TITRE, CRP Tests etc. also can be done at the lab.
Culture and Sensitivity Tests for Urine is available for indoor patients. Urine for Pregnancy test is also done at the lab.
The equipments in the Pathology Lab include Cell Counter Machine, Semi-automatic Analyzer, Electrolyte Analyzer, Full Automatic Biochemistry Analyzer, Autoclave, Laminar Flow, Incubator.
The attainment of quality service in a pathology lab depends on internal and external quality control. At B C Roy Technology Hospital, for quality assurance of the lab reports, the pathology lab has an internal control system. In addition, for external quality control, from January 2012 it is enrolled with the External Quality Assessment Scheme (EQAS), a 12 month program run by Christian Medical College, Vellore, under the banner of Association of Clinical Biochemists of India (ACBI). So far, the Pathology Lab at B C Roy Technology Hospital has been awarded two citations already for consistent quality maintenance in 2012 and 2013.
Pathology sample Collection:
The pathology sample collection is done at B C Roy Technology Hospital regularly from 8.00 AM to 10.00AM on all working days from Monday to Friday, excluding Institute Holidays. The collection is conducted and assisted by trained pathology lab staff.
In addition, the lab provides on-call emergency services during the weekend.
Pathology Report Collection:
The Pathology reports may be collected Monday to Friday from 9.00 AM to 10.00 AM and 5.30 PM to 6.30 PM . The reports are certified by competent pathologists engaged by the hospital.
Microbiology Lab
Culture & Sensitivity Test of Urine & throat swabs are available for indoor cases to limited extent.
Radiology
The Radiology Unit at B C Roy Technology Hospital provides Digital X-Ray Facility to the campus community.
The regular hours are:
Monday to Saturday- 9 AM to 12 Noon and 4 PM to 7 PM
also, provides on-call service in an emergency
Ultrasonography (USG):
Diagnostic Sonography or Ultrasonography (USG) is an ultrasound-based diagnostic imaging technique used for visualizing internal body structures for better diagnosis. At B C Roy Technology Hospital, a visiting consultant in Radiology comes two days in a week to provide this service.
The days and Timings are notified in the on-line notice board of B C Roy Technology Hospital, available through the Internal Notice Board of IIT Kharagpur.
ECG Clinic
Electrocardiogram (ECG) facility is available in the Emergency unit of the B C Roy Technology Hospital.
Timings:
Daily
Physiotherapy:
This unit is currently being upgraded.
Operation Theatre (OT):
Considering the need of the hospital, very recently an Operation Theatre (OT) has been added, by renovating a part of the Hospital building. The OT is now fully equipped and will be soon be functional
A well equipped operation theater has been developed which will enable the hospital to take up various surgical procedures, initially the minor cases and subsequently major cases as well.
Inbuilt Pharmacy
B C Roy Technology Hospital has a pharmacy within the Hospital Premises which is open 24x7 to provide the medicines at all hours to the campus community. It is outsourced to Frank Ross Pharmacy.
Ambulance:
Ambulance service is provided to transfer critically ill patients from IIT Kharagpur campus to the Higher Centers in Kolkata
Mortuary:
A 2 Bedded Air Conditioned Mortuary is available at B C Roy Technology Hospital for the preservation of dead bodies