fast.ai: Intro to Machine Learning 1 (v1) (2018)
Note: this lesson has a VERY practical discussion with USF students about the use of Machine Learning in business/corporation, Jeremy shares his experience as a business consultant (McKinsey) and entrepreneur in AI/ML. Deffo not PhD’s stuff, too real-life.
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
So we've looked at a lot of different random and random forest interpretation techniques and a question: that's come up a little bit on the forums is like what are these for really like, like how do these help me get a better score on cattle and my answers Kind of been like they don't necessarily, and so I want to talk more about like why do we do machine learning like what's the point and to answer this question, I'm gon na put this PowerPoint in the github repo, so you can have a look. I want to show you something really important, which is examples of how people have used machine learning, mainly in business, because that's where most of you are probably going to end up after this is working for some company. I'm going to show you a plication zuv machine learning which are either based on things that I've been personally involved in myself or know of people who are doing them directly. So these are none of these are going to be like hypotheticals. These are all actual things that people are doing and I'm got direct or secondhand knowledge of I'm going to split them into two groups: horizontal and vertical, so in business. Horizontal means something that you do like across different kinds of business, whereas vertical means that something that you do within you know, within a business or within a supply chain or within a process. So in other words an example of horizontal applications. Is everything involving marketing.
So like every company, pretty much has to try to sell more products to its customers, and so therefore does marketing, and so each of these boxes are examples of some of the things that people are using machine learning for in marketing. So, let's take an example: all right, let's take Chen okay, so Chen refers to a model which attempts to predict who's going to leave. That's why I've done some churn modeling fairly recently in in telecommunications, and so we're trying to figure out this big cellphone company, which customers are going to leave. That is not of itself that interesting, like building a highly predictive predictive model that says Jeremy. How it is almost certainly going to leave next month is probably not that helpful because, like if I'm almost certainly going to leave next month, there's probably nothing you can do about it, but it's it's too late. What could have cost you too much to keep me right so in order to understand like or why would we do churn modeling I've got a little framework that you might find helpful. So if you, google, for Jeremy Howard data products, I think I've mentioned this thing before there's a paper. You can find designing great data products that I wrote with a couple of colleagues a few years ago and in it I describe my experience of actually turning machine learning models into like stuff. That makes money right and the basic trick.
Is this thing I call the drive train approach, which is, which is these four steps, the starting point to actually turn a machine learning project into something? That's actually useful is to know what am I trying to achieve, and that doesn't mean like I'm trying to achieve a high area under the ROC curve around trying to achieve a large difference between classes. No, it would be I'm trying to sell more books or I'm trying to reduce the number of customers that leave next month or I'm trying to detect lung cancer earlier right. These are things that these are objectives, so the objective is something that absolutely directly is the thing that the the company or the organization actually wants. No company or organization lives in order to create a you know, a more accurate predictive model, but there's some reason right. So that's your objective. Now, that's obviously the most important thing. If you don't know the purpose of what you're modeling for then you can't possibly do a good job of it, and hopefully people are starting to. You know pick that up in out there in the world of data science, but, interestingly, what very few people are talking about, but it's just as important as the next thing, which is levers lever, is a thing that the organization can do to actually drive the objective.
So, let's take the example of churn modeling right: what is a lever that an organization could use to reduce the number of customers that are leaving? They could look, take a closer look at the model and do some of this random forest interpretation and see some of the causes that are causing people to leave and potentially change those issues in the company. Okay. So that's it. That's a data scientist answer, but I want you to go to the next level. What are the things that levers are the things they can do you want to put it past behind you, what are the things that they can do just reach like calling, or else they could call someone and say like? Are you happy anything we could do? Okay yeah, so they could like give them a free pen or something if they, you know, buy 20 bucks worth of product next month. Yep. You are going to do that as well. Okay, so you guys, you guys are the giving out carrots rather than the handing out sticks. You know over in a couple of yeah special all right, so these are levers right, and so, whenever you're working as a data scientist, you know keep coming back and thinking. What are we trying to achieve, we being the organization and how we try to achieve it? Being like what are the actual things, we can do to make that objective happen. So building a model is never ever a lever, okay, but it could help you with the lever.
So then, the next step is what data does the organization have? That could possibly help them to set that lever to achieve that objective right, and so this is not what data did they give you when you started the project right but like think about it from a first principles, point of view: okay, I'm working for a telecommunications Company they gave me some certain set of data, but I'm sure they must know where their customers live. How many phone calls they made last month?  That might help us to identify who's going to react well or badly. To that, perhaps more, interestingly, would be what, if we were doing like a fraud, algorithm right and so we're trying to figure out like who's going to like not pay for the phone that they take out of the store. You know that they once on 12-month payment plan, we never see them again now, in that case, the data we have available. It doesn't matter what's in the database, what matters is what's the data that we can get when the custom is in the shop right, so there's often constraints around the data that we can actually use.
So we need to know what am I trying to achieve? What can I actually? What can this organization actually do specifically to change that outcome and at the point that that decision is being made, what data do they have or could they collect right, and so then, the way I put that all together is with a model, and this is not A model in the sense of a predictive model, but it's a model in the sense of a simulation model. So one of the main examples I gave in this paper is when I spent many years building, which is if an insurance company changes their prices. How does that impact their profitability right and so generally, your simulation model contains a number of predictive models, so I had, for example, a predictive model called an elasticity model. That said for a specific customer if we charge them a specific price for a specific product. What's the probability that they would say yes, both when it's new business and then a year later, what's the probability that they're good new and then there's another predictive model, which is what's the probability that they're going to make a claim and how much is that plan Going to be right, and so like you can combine these models together, then to say all right if we changed our pricing by reducing it by 10 % for everybody through body between 18 and 25, and we can run it through these models that combined together into A simulation than the overall impact on our market share in ten years time is X and our cost is y and our profit is Z and so forth.
Right so in practice, most of the
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Time, you really are going to care more about kind of the results of that simulation than you do about the predictive model directly, but most people are not doing this effectively at the moment. So, for example, when I go to Amazon right, I read all of Douglas Adams as books, right and so having read all with Douglas Evans's books. The next time I went to Amazon they said, would you like to buy the collected works of Douglas Adams? This is after I had bought every one of his books so like from a machine learning point of view. Some some data scientist had said. Oh people that buy one of Douglas Adams as books often go on to buy the collected works right, but recommending to me that I buy the collected works of Douglas Adams. Isn't smart that and it's actually not smart at a number of levels like not only is unlikely to buy a box set of something of which I have everyone individually, but furthermore, it's not going to change my buying behavior, like I already know about Douglas Adams. I already know I like him so taking up your valuable web space. To tell me hey, maybe you should buy more of the author who you're already familiar with in the port.  So what? If, instead of creating a predictive model, Amazon had built an optimization model that said like that, could simulate and said if we show Jeremy this ad, how likely is he then to go on to buy this book and if I don't show him this ad, how likely Is here to go on to buy this book, and so that's the counterfactual right, the counterfactual is what would have happened otherwise and then you can take the difference and say: okay, what should we recommend him that is going to maximally change his behavior? So maximally result in more books and so you'd probably say like.
Oh he's, never bought me terry pratchet box. He probably doesn't know about terry pratchet, but like lots of people that, like to douglas adams's, did turn out to like terry pratchett. So, let's like introduce him to a new author right, so it's the difference between a predictive model, on the one hand versus an optimization model. On the other hand, so the two tend to go hand in hand right. The optimization model basically is saying well, first of all, we have a simulation model right. The simulation model is saying in a world where we put terry pratchett's book on the front page of amazon for Jeremy Howard. This is what would have happened. He would have bought it with a money, four percent probability right, and so that then tells us with this lever of like what do I put on there on my homepage for Jeremy today, we say: okay, well, the different settings of that lever that put Terry Patchett On the homepage has the highest simulated outcome right and then that's the thing which maximizes our profit from Jeremy's visit to amazon.com today. Okay, so, generally speaking, your predictive models kind of feed in to this simulation model, but you kind of got to think about like how do they all work together. So, for example, let's go back to churn right, so I'd turn out that Jeremy Howard is very likely to leave his cell phone company next month.
What are we gon na do about it? Oh, let's call him right and I can tell you if my cell phone company calls me right now and says just calling to say we love you I'd be like I'm cancelling right now like like. That would be a terrible idea. So again, you would want a simulation model that says like. What's the probability that Jeremy is going to change his behavior as a result of calling him right now right so elite, one of the levers I have is call him. On the other hand, if I like got a piece of mail tomorrow that said like for each month, you stay with us we're going to give you a hundred thousand dollars. Okay, then that's going to definitely change my behavior right so, but then feeding that into the simulation model. It turns out that overall, that would be an unprofitable choice to make. So do you see how this fits in together all right? So so, when we look at something like churn, we want to be thinking like what are the levers. We can pull right, and so what are the kind of models that we could build with what kinds of data to help us pull those levers better to achieve our objectives? And so, when you think about it, that way, you realize that the vast majority of these applications are not largely about a predictive model at all, they're about interpretation they're about understanding what happens if right.
So if we kind of take the cross-product or not the cross-product or either inter intersection between on the one hand, here are all the levers that we could pull like here or all the things we can do. And then here are all of the features from our random forest feature importance that turn out to be strong drivers of the outcome, and so then the intersection of those is here are the levers. We could pull that actually matter right, because if you can't change the thing that is not very interesting and if it's not actually a significant driver, it's not very interesting right, so we can actually use our random forest feature importance to tell us what can we actually Do to make a difference, and then we can use the partial dependence to actually build this kind of simulation model to say like okay. Well, if we did change that, what would happen? Okay, so you know there are examples, lots and lots of these vertical examples, and so what I want you to kind of think about, as you think, about the machine learning problems you're working on is like. Why does somebody care about this right and like what would a good answer to them look like and how could you you know, how could you actually positively impact this business so, if you're creating like a Keagle, you know try to think about.
From the point of view of the competition organizer like what would they want to know, and how can you give them that information so something like fraud detection? On the other hand, you probably just basically want to know whose fraudulent right, so you probably do just care about the predictive model, but then you do have to think carefully about the data availability here. So it's like, okay, that we need to know who is fraudulent at the point that we're about to deliver them a product right. So it's no point like looking at data. That's available like a month later, for instance, so you've kind of got this. This key issue of thinking about you know the actual operational constraints that you're working under you know lots of interesting applications in human resources, but, like employee churn, it's another kind of churn model, we're finding out that Jeremy Howard sick of lecturing he's going to leave tomorrow. What are you going to do about it? Well, knowing that wouldn't actually be helpful, it'd be too late right, you would actually want a model. That said what kinds of people you know are leaving USF, and it turns out that, like Oh everybody that goes to the downstairs, cafe leaves USF. You know, I guess their food is awful or you know whatever right or everybody, that we're paying less than half a million dollars a year is leaving USF. You know because they can't afford basic housing in San Francisco. So, like you could use your employee churn model, not so much to say like which employees hate us, but why do employees leave right? And so again, it's really the interpretation there that that matters now lead prioritization is a really interesting one right like this is one where a lot of companies.
Yes Dana. Can you pass that over there? You know so I was just wondering, select the Shawn thing or you suggest it. So one being is like being an employee like a 1 million a year or something, but then it sounds like there are two predictors that you need to predict. Put I mean one, each Shawn and one you need to optimize for it like you're profiting, so how does it work yeah exactly so? This is what this like simulation model is all about. So it's a great question so, like you kind of figure out this objective, we're trying to maximize which is like company profitability, you can kind of create like a pretty simple like Excel model or something that says like here's, the revenues and here's the costs and the Cost is equal to them. You know number of people we employ multiplied by their salaries, blah blah blah blah blah right and so inside that kind of Excel model. There are certain cells. There are certain inputs where you're like oh, that thing's kind of stochastic you know or that thing is kind of uncertain, but we could predict it with a model and so that's kind of what I. What I do, then, is, I then say: okay, we need a predictive model for how likely somebody is to stay if we change their salary, how much? How likely they are to leave. You know with the current salary how likely they are to leave next year if they, if I increase their salary now, why blah? So you could have ruled a bunch of these models and then you can bind them together with simple business logic, and then you can optimize that you can then say.
Okay, if I you know pay, Jeremy, Howard, half a million dollars, that's probably a really good idea. You know, and if I pay him less than you know, it's probably not or whatever like you, can figure out the the overall impact, and so it's it's really shocking to me. How few people do this, but most people in industry measure their models using like a UC or our MSC or whatever, which is never actually what you want.
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Yes, can you pass it over here? I wanted to stress the point that you made before um. In my experience, a lot of the problem was to define the problem right.  They don't know that you have to have x and y and so on. So you have to try to get that out of them.  You know actually what you can measure, which is often know what they want. So you have to kind of get a proxy for what they want and then so a lot of what you do is not that much of like well. Some people do actually just work on really good models for you know, but a lot of people also just work on this kind of. How do you put this? I say you know, classification, regression or some other type of modeling, that's actually kind of the most interesting. I think, and also kind of what's kind of what you have to do well understand the technical model building deeply, but also understand the kind of strategic context deeply, and so this is one way to think about it and, as I say like you know, I actually Think you know there aren't many articles. I wrote in 2012 I'm still recommending, but this one, I think, is still equally valid today, so yeah so like another great example is lead prioritization right.
So, like a lot of companies like every one of these boxes, I'm showing you can generally find a company or many companies whose sole job in life is to build models of that thing right. So there are lots of companies that sell laid prioritization systems, but again, like the question is how would we use that information right? So if it's like, oh our best lead is Jeremy, you know he's a highest probability of buying. Does that mean I should send a salesperson out to Jeremy, or I shouldn't like, if he's highly probable, to buy why I waste my time with him. You know so like again, it's like. He really wants some kind of simulation. That says like what's the chain, the likely change in Jeremy's behavior, if I send my best salesperson your net out to go and like encourage him to sign okay, so yeah. I think this. This is like. There are many many opportunities for data scientists in the world today to move beyond predictive modeling to actually bringing it all together. You know, and with the kind of stuff that Dena was talking about in the question so as well as these horizontal applications that basically apply to like every company, there's a whole bunch of applications that are specific to like every part of the world right. So if for those of you that end up in health care, some of you will become experts in one or more of these areas like readmission risk.
Okay, so what's the probability that this patient is going to come back to the hospital and readmission is depending on the details of the jurisdiction and so forth, it can be a disaster for hospitals when somebody is readmitted right. So if you find out that this patient has a high probability of readmission, what do you do about it? Well again, the predictive model is helpful of itself right. It rather suggests like we just shouldn't, send them home yet because they're going to come back, but wouldn't it be nice. If we had the tree interpreter - and it said to us the reason that they're at high risk is because we don't have a recent EKG for them and without a recent EKG, we can't have a high confidence about their. You know cardiac health, in which case it wouldn't be like well, let's keep them in the hospital for two weeks, it'll be like, let's give them an EKG okay. So this is. This is interaction between interpretation and predictive accuracy. The predictive models are a really great starting point, but in order to actually like answer these questions, we really need to focus on the interpretability of these models. Yeah, I think so and and more specifically, I'm saying like we just learnt a whole raft of random forest interpretation techniques, and so I kind of just to kind of try to justify like well. Why right and so the reason.
Why is because, actually maybe I'd say most of the time, the interpretation is the thing we care about and, like you can create a chart, you know or a table without machine learning, and indeed that's how most of the world works right. Most managers like build all kinds of tables and charts without any machine learning behind them, but they often make terrible decisions because they don't know the future importance of the objective they're interested in and so the table they create is of things that actually are the least Important things anyway, or they just do a univariate chart rather than a partial dependence plot, so they don't actually realize that the relationship they thought they're looking at is drew entirely to something else right. So you know I'm kind of arguing for data scientists getting like much more deeply involved in in strategy and in trying to use machine learning to really help. You know help a business with all of its objectives. Right now, there's like there, companies like dun hum B, is a huge company that does nothing but retail applications with machine learning and so, like, I believe, there's like a dun Humby product you can buy, which will help you, which will help you figure out like if I put my new store in this location versus lat location.
 So it's I think. It's also good to realize that, like the subset of machine learning applications, you tend to hear about you know in in the tech press or whatever. Is this massively biased, tiny subset of stuff which kind of Google and Facebook do where else the vast majority of stuff that actually makes the world go around? Is you know these kinds of applications that actually help people make things buy things sell things build things so forth, so about create repetition? The way we looked at the tree was we manually check, which feature could cause good was more important for for particular observation, but for businesses they would have a huge amount of data and they they want this interpretation for a lot of observations. So how do they automate it? I don't think the automation is at all difficult like you just you can run any of these algorithms like looping, through the rows or doing them in parallel. It's all this code. Oh, I miss understand your question. Is it like? They set a threshold that, if some feature is about like four different different people will have different behavior, oh so so yeah, okay, I guess it's good question that the important thing this is a really important issue.
Actually is the vast majority of machine learning models? Don't automate anything they're designed to provide information to humans right so, for example, if you're appointed sales customer service phone operator for an insurance company - and your customer asks you - why is my renewal $ 500 more expensive than last time? Then? Hopefully you know the insurance company has provides in your terminal those little screen that shows the result of the tree interpreter or whatever, then chills so that you can jump there and tell the customer like okay. Well, here's last year, you're in this different zip code, which you know is less, has lower amounts of car theft and this year also you've actually changed your vehicle to more expensive one or whatever right. So it's not so much about thresholds and automation, but about you know making these model outputs available to the decision-makers in an organization whether they be at the top strategic level of like you know, are we going to shut down this whole product or not all the Way to the operational level, look like it that that individual discussion with a customer so like another example, is like aircraft scheduling and gate management like there's lots of companies that do that right and basically, what happens is that the the people you know there are there Are people in at an airport whose job it is to basically tell each aircraft what gate to go to to figure out when to close the doors stuff like that? And so the idea is, you know, you're giving them software, which has the information they need to make good decisions, so the machine learning models end up, embedded in that software to kind of say, like okay, that plane that's currently coming in from Miami there's. A 48 percent chance that it's going to be over five minutes late and if it does, then this is going to be the knock-on impact through the rest of the terminal, for instance, okay.
Well, that's kind of how these things turned a bit together, so there's so many of these right there's lots and lots, and so it's not like. I don't expect you to like remember all these applications, but what I do want you to do is to like spend some time like thinking about them like sit down with one of your friends and like talk about a few examples of like okay, how would we Go about like doing failure, analysis and manufacturing, like you know, who-who would be doing that. Why would be they doing it? What kind of models might they use? What kind of data might they use like start to like practice this and get a sense? So because then this you're like interviewing and then when you're at the workplace and you you know you're talking to managers, you wouldn't be like straightaway able to kind of recognize that the person you're talking to what do they try to achieve. What are the levers that they have to pull right? What if the data they have available to pull those levers to achieve that thing, and therefore how could we build models to help them do that and what kind of predictions would they have to be making right? And so then you can have this really thoughtful. Empathetic conversation with those people and the same like hey, you know, in order to reduce the number of customers that are leaving, you know I guess you're trying to figure out like you know who, should you be providing better pricing to or whatever and so forth? So what I'm noticing like from your beautiful little chart above, is that, like a lot of this, to me at least still seems like.
The primary purpose is like at the at least base level like is predictive power, and so I guess my thing is is like for explanatory problems like a lot of the ones that are people are faced with, like in social sciences. Is that something machine learning can be used for or is used for, or is that not really the realm that it yeah? It's um, that's a great question and I've had a lot of conversations about this with people in social sciences and currently machine learning is not well applied in, like economics or psychology or whatever on the whole, but I'm Jun convinced it can be, for the exact reasons We're talking about so, if you're, trying to figure out like you're, going try to do some kind of behavioral economics and you're trying to understand like why some people behave differently to other people. You know a random forest with a feature importance. What would be a great way to start or like more interestingly, if you're trying to do some kind of sociology experiment or analysis based on a large social network data set where you have an observational study, you really want to try and pull out all of the Sources of kind of exogenous variables, you know all the stuff that's going on outside, and so, if you use a partial dependence plot with a random forest that happens automatically.
So I actually gave a talk at MIT a couple of years ago for the first conference on digital experimentation, which was really talking about like how do we experiment in you know things like social networks in kind of these digital environments and yeah economists? Economists all do things with, like you know, classic statistical tests but um the group of yeah yeah um, so but anyway, in this case the The Economist's. I talked to were absolutely fascinated by this and they actually asked me to give a a introduction to machine learning session at MIT to these various faculty and graduate folks in the Economics Department and so and some of those folks have gone on to be. You know write some pretty famous books and stuff, and so hopefully it's been useful so like it's definitely early days, but it's a it's.  The skepticism comes from unfamiliarity, basically with like this totally different approach so like if you've, if you spent 20 years studying econometrics and somebody comes along and says you know, here's a totally different approach to all the econometrics, all these stuff. That econometricians do you know. Naturally, your first reaction will be like prove it. You know. So let's do enough, but I think it's you know over time. The next generation of people who are growing up with machine learning, some of them will move into the social sciences.
They'll make huge impacts that nobody's ever managed to make before and people will start going wow. You know just like happened in computer vision right when you know computer vision spent a long time of people saying like hey. Maybe you should use deep learning for computer vision and everybody in computer vision is like proven. You know we have decades of work on amazing feature detectors for computer vision and then finally, in 2012 you know Hinton and cadets key came along and said: okay and models like twice as good as yours, and you know, we've only just started on this and everybody Was like okay, that's pretty convincing. Nowadays, every computer vision researcher basically uses the big wedding, so I think that time will come in there in this area too. Okay, I think what we might do, then, is take a break and we're going to come back and talk about these random forest interpretation techniques and do a do a bit of a review. So let's come back at two o'clock, so let's have a go at
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Talking about these different random forest interpretation methods, having talked about like why they're important, so, let's now remind ourselves like what they are, so I got ta. Let you folks have a go. So, let's, let's start with confidence based on tree variance, so can one of you tell me one or more of the following things about confidence based on tree variance to do what does it tell us? Why would we be interested in that and how is it calculated? This is going back a ways first, when we looked at even if you're, not sure you only know a little piece of it, give us your piece and we'll build on it together. I think I got a piece of it. It's it's getting. The variance of our predictions from random forests - that's true, that's the! How can you be more specific? What is it the variance of? I think it's remembering correctly. I think it's just the overall prediction: the variance of the predictions of the trees. Yes, so normally the prediction is just the average. This is the variance of the trees, and so it kind of just gives you an idea of how much your prediction is going to vary. So, if maybe even want to minimize variance. Maybe that's your goal for whatever reason that could be: that's not so much the reason. So I, like your calculation description, let's see if somebody else can tell us how you might use that it's okay, if you're not sure, now have a start.
So I remember that we talked about the independence of the trees and so maybe something about if the branch of the trees is higher or lower than no, not so much that um. That's that's that's an interesting question, but it's it's not what we're going to see here. I'm gon na pass it back behind you. So to remind you just to fill in a detail here. What we generally do here is we take us just one row like one observation, often and like find out how confident we are about that like how much variance there are in the trees for that or we can do it, as we did here for different groups For each row, we calculate the standard deviation that we get from the random forest model and then maybe group according to different variables or predictors and see for which particular predictor. The standard deviation is high, then go deep, donnas why it is happening. Maybe it is because a particular category of that variable has very less number of observation. Yeah, that's great, so that that would be one approach is kind of. What we've done here is to say, like is there any groups that have that, where we're very unconfident, something that I think is even more important, would be when you're using this like operationally right, let's say, you're doing a credit, decisioning algorithm, so we're trying to say, Like okay is jeremy, a good risk or a bad risk, should we loan him a million dollars and the random forest says, I think, he's a good risk, but I'm not at all confident, in which case we might say.
Okay, maybe I shouldn't give him a million dollars where else, if we, if, if the Rena first said, I think, he's a good risk, I am very sure of that. Then we're much more comfortable, giving him a million dollars right and I'm a very good risk, so feel free to give me a million dollars right. I checked the random forest before a different notebook, not in the repo so like. This is like it's quite hard for me to give you folks direct experience with this kind of like single observation interpretation stuff, because it's really like the kind of stuff that that you actually need to be putting out to the front line. Do you know what I mean like it's, not something which you can really use so much in a kind of casual context, but it's more like okay, if you're actually putting out some algorithm which is making like big decisions that could cost a lot of money. You probably don't so much care about the average prediction of the random forest, but maybe you actually care about, like the average minus a couple of standard deviations, you know like what's the kind of worst-case prediction and so magic I mentioned it's like. Maybe there's a whole group that we're kind of unconfident about so yeah. So that's
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Confidence based on tree variance all right, who wants to have a go with answering feature importance. What is a, why is it interesting? How do we calculate it or any subset thereof? You know, I think it's like that's, basically to find out which, which of those which features are important for your model. So you take each feature and you like randomly sample all the values in the feature, and you see how the predictions are if it's very different, it means that that feature was actually important as if it's fine to take any random values. For that feature, it means that, maybe probably it's not very okay - that was terrific, there's this. That was all exactly right. There was some details that maybe were skimmed over a little bit. I wonder if anybody else wants to jump into like a more detailed description of how it's calculated, because I know this morning, some people were not quite short. Is there anybody who's like not quite sure? Maybe he wants to like have a go or yeah well, it was put it next to there. Let's see how exactly do we calculate feature importance for a particular feature check the validation school if it gets pretty bad for after opening one of the columns. That means that column was important so and as higher importance. I'm not exactly sure how we quantify the feature importance. Okay, great Dana, do you know how we quantify the feature importance? That was a great description or score or some sort exactly yeah.
So let's say, we've got our dependent variable, which is price right and there's a bunch of independent variables, including year made right, and so we basically, we use the whole lot to build a random forest right. And then that gives us our predictions right. And so then we can, let's call this Y right, and so then we can compare that to get. I don't know whatever R squared R MSC, whatever you're interested in right from the model. Now the key thing here is, I don't want to have to retrain my whole random forest, that's kind of slow and boring right, so using the existing random forests. How can I figure out how important year made was right, and so the suggestion was, let's randomly shuffle the whole column right so now that column is totally useless. It's got the same mean same distribution. Everything about it is the same, but there's no connection at all. Between particular people actual year made and what's now in that column, I've randomly shuffled it okay, and so now I put that new version through with the same random forest. So there's no retraining done okay to get some new y hat. I call it Y hat way M right and then I can compare that to my actuals to get like an RSC, ym right and so now I can start to create a little table. So now I can create a little table where I basically got like the original here, our MSC and then I've got with year made scrambled. So this one had an ARMA see of like three.
This one had an ARMA see of like two enclosure scrambling that had a our MSC of like 2.5 right, and so then I just take these differences. So I'd say year made. The importance is one 3-2. Enclosure is 0.53 minus, 2 and 1/2 and so forth. Right, so how much worse did my model get after? I shuffled that variable. Anybody have any questions about that. Can you pass that to Danielle? Please um. I assume you just chose those numbers randomly, but I question I guess is: does it stop? Do all them be ready, I'm not a perfect model to start out with like. Are they welded all the important says seven to one, or is that not no they're? Just I don't honestly, I've never actually looked at what the units are. So I'm not I'm actually not quite sure. Sorry. We can check it out during the week if somebody's interested, how the Vulcans have a look at the this scikit-learn code and see exactly what those units of measure are cuz. I've never bothered to check, although I don't check, like the units of measure. Specifically, what I do check is the relative importance, and so like here's an example. So, rather than just saying like what are the top ten yesterday, one of the practicum students asked me about a feature importance where they said like. Oh, I think these three are important and I pointed out that the top one was a thousand times more important than the second one right.
So like look at the relative numbers here, and so in that case it's like no don't look at the top three look at the one. That's a thousand times more important and ignore all the rest, and so this is where sometimes the kind of your natural tendency to want to be like precise and careful, you need to override that and be very practical. It's like okay, this thing's a thousand times more important, don't spend any time on anything else right. So then, you can go and talk to the manager of your project and say like okay, this thing's a thousand times more important and then they might say. Oh, that was a mistake. It shouldn't have been in there. We don't actually have that information at the decision. Time or you know, but for whatever reason we can't actually use that variable, and so then you could remove it and and have a look or they might say gosh. I had no idea that, like that was more by far more important than everything else put together. So, let's forget this random virus thing and just focus on like understanding how we can better collect that one variable and better use that one variable. So that's like something which comes up quite a lot and actually another place that came up just yesterday again. Another practicum student asked me: hey I'm doing this medical diagnostics project and my r-squared is 0.95 for a disease which I was told, is very hard to diagnose. You know, is this random forrester genius or is something going wrong, and I said like remember the second thing you do after you build a random forest is to do feature importance, so do feature importance, and what you'll probably find is that the top column is something That shouldn't be there, and so that's what happened.
He came back to me half an hour later, he said yeah. I did the feature importance. You were right. The top column was basically a something that was another encoding of the dependent variable. I've removed it, and now my a squared is negative 0.1. So that's an improvement. Okay. The other thing I like to look at is this chat. Right is to basically say, like you know, where the kind of things flatten off in terms of like which ones should I be really focusing on. So that's the most important one right, and so when I did credit scoring in telecommunications, I found there were nine variables that basically predicted very accurately. Who was it going to end up paying for their phone and who wasn't and like, apart from ending up with a model that saved them three billion dollars a year in in fraud and credit costs? It also let them basically rejigger their their process. So they focused on collecting those nine variables, much better, all right who
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Wants to do partial dependence. This is an interesting one, very important, but you know somebody's kind of tricky to think about yeah. Please do not always necessarily like a relationship between the strictly the dependent variable in this independent variable. That necessarily like is showing importance, but rather than an interaction between two variables that are working together, right, yeah, oh, that's, weird, yeah, and so for this example. What we found was that it's not necessarily your maid or when the sale was elapsed, but it's actually the age of the model, and so it's that is easier to just like to tell a liked company. Well, obviously, you're younger models are gon na sell for more and it's less about when the year was made yeah. So, let's come back to how we calculate this in a moment, but the first thing to realize is that the vast majority of the time you know post your course here when somebody shows you a chart, it'll be like a univariate chat, that'll just like grab the Data from the database and they'll plot X against Y and then managers have a tendency to want to like make a decision. All right so be like. Oh there's this like drop off here. So we should like stop dealing in equipment made between 1990 and 1995 or whatever right, and this is like a big problem because, like real world data, has lots of these interactions going on so, like you know, maybe there was a recession going on around the time That those things are being sold or maybe around that time people were buying more of a different type of equipment or whatever right so generally.
What we actually want to know is all other things being equal. What's the relationship between ear made and sale price right because, like if you think about the drivetrain approach idea of like the levers, you really want a model that says if I change this lever, how will it change my objective, okay and it's by pulling them apart? Using partial dependence that you can say: okay, actually, this is the relationship between year made and sale price, all other things being equal right. So how do we calculate that? For the variable you made, for example, you're gon na train? You keep every other car variable constant and then you're gon na pass every single value of the year mate and then train the model after that. So for every model you might have their light blue or the values of it and the median is gon na. Be the yellow line? Okay, so let's try and draw that so bye leave everything else constant. What she means is leave them at whatever they are in the data set so just like when we did feature importance right, we're going to leave the rest of the data set as it is, and we're going to do. Partial dependence plot for year made all right. So we've got all of these other rows of data that we'll just leave as they are, and so, instead of randomly shuffling em8. Instead, what we're going to do is replace every single value with exactly the same thing: 1960, okay and just like before.
We now pass that through our existing random forests, which we have not retrained or changed in any way to get back out a set of predictions, why 1960, okay and so then we can plot that on a chart, yeah I'd against partial dependence, 1960 here, okay, now We can do it for 1960, one two, three, four five and so forth right, and so we can do that for four on average for all of them, or we could do it just for one of them right and so when we do it for just one Of them and we change its year may to pass that single thing through our model. That gives us one of these blue lines. That's a each one of these blue lines is a single row as we change its year made from 1960 up to 2008, and so then we can just take the median of all of those blue lines to say you know on average, what's the relationship between year Made and price all other things being equal, so why is it that? Why is it that this works? Why is it that this process tells us the relationship between year made and price, all other things being equal? Well, maybe it's good to think about like a really simplified approach, a really simplified approach would say: what's the average auction you know, what's the average sale date, what's the most common type of machine we sell, which location do we mainly mostly sell things and like we Could come up with a single row that represents the the average option and then we could say: okay, let's run that row through the random forest, but replace its year made with 1960 and then do it again with 1961 and then do it again with 1962 and We could like plot, you know those on our little chart right and that would give us a version of the relationship between year made and sale price, all other things being equal right. But what if like tractors, looked like that and backhoe loaders looked like that right then taking the average one would hide the fact that there are these totally different relationships right.
So instead we basically say: okay, our data tells us what kinds of things we tend to sell and who we tend to sell them and when we tend to sell them. So let's use that right. So then we actually find out like for every blue line like here are actual examples of these relationships right and so then, what we can do is as well as flooding the median is. We can do a cluster analysis to find out like a few different shapes right, and so we may find in this case they all look like pretty much the different versions of the same thing with different slopes. So my main takeaway from this would be that the relationship between sale, price and year is basically a straight line. Right and remember, this was log of sale price right, so this is actually showing us an exponential, and so this is where I would then like bring in the domain expertise which is like okay, things depreciate over time by a constant ratio. So therefore, I would expect older stuff year made to have this exponential shape, so this is where, like I said of mentioned, like the very start of my machine learning project, I generally try to avoid as using as much domain expertise as I can and let the Data do the talking all right so, like one of the questions I got this morning was like if there's like a sale, I'd be a model ID. I should throw those away right because they're just IDs, no, don't assume anything about your data right leave them in and if they turn out to be super important predictors, you want to find out.
You know. Why is that? Okay? But then now I'm at the other end of my project right, I've done my feature importance. I've pulled out the stuff which is like you know from that dendogram. You know the kind of redundant features, I'm looking at the partial dependence and now I'm thinking like okay. Is this shaped what I expected? Okay, so even better before you plot this. First of all, think what shape would I expect this to be because it's always easy to justify to yourself after the fact. Oh, I knew it would look like this right, so what straight you expect and then is it that shape so in this case I'd be like yeah. This is this is what I would expect. Okay, where else this is definitely not what I'd expect. So the partial dependence plot has really pulled out the underlying truth. Okay, does anybody have any questions about like why we use partial dependence or how we calculate it? It's got the. Are you better, don't say you have a few thousand say twenty features. Everything are important. Are you gon na measure the partial dependence for every single one of them? If there are twenty features that are important, then I will do the partial dependence for all of them. We're important means like it's a lever. I can actually pull it's like. The magnitude of its size is like not much smaller than the other nineteen. Like you know, based on all of these things, it's like yeah, it's a feature I ought to care about. Then I will want to know how it's related.
It's pretty unusual to have that many features that are important both operationally and from a modeling point of view. In my experience, so how do you define internationally? So important means it's it's a lever, so it's something I can change and it's like you know, kind of at the spiky end of this tail or you know it. Maybe it's not a lever directly like maybe it's like zip code, and I can't actually tell my customers where to live, but I could like focus my new marketing attention on a different, zip code. You know, would it make sense to do pairwise shuffling for every combination of two features and hold everything else constant, like in future importance to see interactions and compare scores, so you wouldn't do that so much for partial dependence. I think your question is really getting to the question of. Could we do that for feature importance all right, so I think interaction. Feature importance is a very important and interesting question, but doing doing it by randomly shuffling every pair of columns. You know if you've got a hundred columns, sounds computationally intensive, possibly infeasible. So what I'm going to do is after we talk about tree interpreter I'll, talk about interesting, but largely unexplored approach that will probably work. Okay, who wants to do
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Tree interpreter all right over here Prince. Can you pass that over here to principal? I was thinking this to be more like feature importance, but teacher importance is for complete random forest model, and this tree interpreter is for feature importance for particular observation. So if that, let's say it's about hospital readmission, so if a patient a one is, it is going to be readmitted to a hospital which featured for that particular patient is going to impact. And how can we change that, and it is calculated starting from the prediction of mean then seeing how each feature is changing the behavior of that particular patient. I'm smiling, because that was one of the best examples of technical communication. I've heard in a long time. So it's really good to think about like what. Why was that effective right? So what Prince did there was he used as specific an example as possible right so if humans are much less good at understanding, abstractions right? So if you kind of say, oh, it takes some kind of feature and then there's an observation in that feature river. You know where's like know it's it's the hospital readmission, okay, and so we take a specific example. The other thing he did was very effective was to kind of take an analogy to something we already understand. So we already understand the idea of feature importance across all of the rows in a data set. So now we're going to do it for a single row.
Okay, so, like you know, one of the things I was really hoping we would learn from from this experience is how to become effective technical communicators. So you know that was a really great role model from Prince of like using all the tricks we have at our disposal for effective technical communication. So hopefully you found that useful explanation. I don't have a hell of a lot to add to that other than to show you. You know what that looks like so with the tree interpreter. We peaked out a row, okay and so remember when we talked about the the confidence intervals. At the very start, the confidence based on tree variance - we mainly said - like you, probably mainly use that for a Rome, so this would also be for our crow. So it's like okay, why is this patient likely to be readmitted? Okay, so here is all of the information we have about that patient or in this case this option. Now, why is this auction so expensive? So then we call tree interpreter dot predict and we get back the prediction of the price right. The bias which is the root of the tree, so this is just the average price for everybody, so this is always going to be the same and then the contributions, which is how important is each of these. Each of these things right and so the way we calculated that so the way we calculated, that was to say, okay at the very start, the average price was ten right and then we split on enclosure right and for those with dissin closure.
The average was nine point five and then we split on year made - I don't know less than 1990 and for those with that year made the average price was nine point, seven right, and then we split on the number of hours on the meter and for you Know with this branch we got nine point four all right, and so we then have a particular option which we we pass it through the tree, and it just so happens that it takes this path all right. So one row can only have one path through the tree right, and so we ended up at this point. Okay, so then we can create a little table right and so as we go through, we start at the top and we start with 10 right. That's that bias - and we said enclosure resulted in a change from 10 to 9 and 1/2 minus 0.5 yeah man changed it from nine point: five to nine point, seven so plus 0.2 right and then meter changed it from nine point: seven down to nine point: four, Which is minus 0.3 and then, if we add all that together can minus a half is nine and a half plus 0.2 is nine point. Seven minus point: three is nine point four low and behold that's that number, which takes us to our Excel spreadsheet. Where's Chris, who did our waterfall there? You are all right so last week we had to use Excel for this because there isn't a good Python library for doing waterfall charts, and so we saw we got our starting point.
This is the bias and then we had each of our contributions and we ended up with our total. The road is now a better place, because Chris has created a Python waterfall chart module for us and put it on pip, so never again where we have to use Excel for this, and I wanted to point out that, like waterfall, charts have been very important in Business communications, at least as long as I've been in business. So that's about 25 years Python. Is you know what couple of decades old a little bit less yeah, maybe a couple of decades old. But you know, despite that, no one in the Python world ever got to the point where they actually thought. You know I'm gon na make a waterfall chart, so they didn't exist until two days ago, which is to say, like the world, is full of staff which ought to exist in, doesn't and doesn't necessarily take a hell, a lot of time to build Chris. How long did it take you to build the first Python waterfall chart? Well, there was a you know, a gist of it: yeah yeah yeah about eight hours. Okay, so you know a hefty TYIN amount, but not unreasonable and now forevermore people when they want the place and waterfall chart, will end up at Chris's, github repo and hopefully find lots of other USF contributors who have made it even better so for in order for You to help improve Chris's plaything waterfall. You need to know how to do that right and so you're going to need to submit a pull request.
Life becomes very easy for submitting pull requests if you use something called hub. So if you go to github, slash hub, that will send you over here and what they suggest you do is that you alias get to hub, because it turns out that hub actually is earth strict superset if get, but what it lets you do is you can Go get fork, get push, get pull request and you've now sent Chris a pull request now without hub. This is actually a pain and requires like own to the website and filling informants and stuff right. So this gives you no reason not to do pull requests, and I mention this because, like when you're interviewing for a job or whatever, I can promise you that the person you're talking to will check your github. And if they see you have a history of submitting thoughtful pull requests that are accepted to interesting libraries. That looks great right. It looks great because it shows you're somebody who actually contributes. It also shows that, if they're being accepted that you know how to create code that fits with people's coding standards has appropriate documentation, passes their tests and coverage and so forth right. So when people look at you - and they say - oh here's - somebody with a history of successfully contributing accepted, pull requests to open-source libraries.
That's a great part of your portfolio, okay and you can specifically refer to it right so either I'm the person who built Python waterfall here is my repo or you know, I'm the person who contributed currency, number formatting to Python waterfall, here's my pull request. You know that anytime, you see something that doesn't work right in any open source software you use is not a problem, it's a great opportunity because you can fix it and send in the pull requests so yeah give it a go. It actually feels great. The first time you have a pull request accepted and, of course, one big opportunity is the faster, a library and thank you who it was the person here, the person who added all the docs to fast a I structured in the other class okay. So, thanks to one of our students, we now have docstrings for most of the FASTA. I dot structured library again came by a pull request. So thank you. Okay, does anybody have any questions about how to calculate any of these random forest interpretation methods or why we might want to use any of these random first interpretation methods towards the end of the week you're going to need to be able to build all of these Yourself from scratch over there can you pass that test? That's right! Just looking at the tree interpreter, I noticed that some of the the values are in a ends. How I got I get my you keep them in the tree, but how can an NA and have a future importance? Okay, let me pass it back to you.
Why not so, in other words, how is ni n handled in pandas and therefore in the tree, such as some default value, everybody remember how pandas these are. Notices are all in categorical variables. How does pandas handle an NA ends in categorical variables and how does fast? Ai deal with them can somebody pass it to the person who's talking negative one through yeah pandas sets into negative one category code, and do you have to remember what we then do? Doesn't matter really, we add one to all of the category codes, so it ends up being zero, so in other words, we have a category with remember by the time it hits the random forest is just a number, and it's just it's just the number, zero right And we map it back to the descriptions back here. So the question really is: why shouldn't the random first be able to split on zero? It's just it's just another number, so it could be na n high, medium or low zero. One two three four, and so you know missing values are one of these things that are generally taught really badly. Like often people get taught like here are some ways to remove columns with missing values or remove rows with missing values or to replace missing values. That's like never what we want, because missingness is very, very, very often interesting, and so we actually learnt that from our future importance that cupola system n, a n is like one of the most important features, and so for some reason. Well, I could.
I could guess right: coupla system n am presumably means. This is a kind of industrial equipment that doesn't have a coupler system. Now I don't know what kind that is, but apparently it's more it's a more expensive kind. That makes sense. Yeah. Ok, ok! So I did this competition for university grant research success where, by far the most important predictors were whether or not some of the fields were a null, and it turned out that this was data leakage that these fields only got filled in most of the time. After a research grant was accepted right, so you know it allowed me to win that cowgirl competition, but didn't actually help the university very much okay, great
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
So, let's talk about um extrapolation and I am going to do something risky and dangerous, which is we're going to do some live coding and the reason we're going to do some live coding is I want to explore extrapolation together with you, and I kind of also Want to get kind of help give you a feel of. You know like how you might go about, like writing - writing code quickly in this notebook environment right, and this is the kind of stuff that you're going to need to be able to do. You know in the real world and in the exam, is kind of quickly create the kind of code that we're going to talk about. So I really like creating synthetic datasets anytime, I'm trying to like investigate the behavior of something, because if I have a synthetic data set, I know how it should behave which reminds me before we do this. I promised that we would talk about interaction importance, and I just about forgot, trie interpreter - tells us the contributions for our particular row. Based on the difference in the tree. We could calculate that for every road in our data set and add them up right, and that would tell us that would tell us feature importance and would tell us feature importance in a different way. Right. One way of doing feature importance is by shuffling the columns. One at a time, another way is by doing tree interpreter for every row and adding them up. Neither is right more right than the others, they're, actually both quite widely used.
So this is kind of type 1 and type 2 feature importance. So we could try to expand this a little bit. Do not just single variable feature importance, but interaction feature importance. Now, here's the thing what I'm going to describe is very easy to describe. It was described by Bremen right back when random forests were first invented, and it is part of the commercial software product from Salford systems who have the trademark on random forests. But it is not part of any open-source library, I'm aware of, and I've never seen an academic paper that actually studies it closely. So what I'm going to describe here is a huge opportunity, but it's also like there's lots and lots of details that kind of need to be fleshed out, but here's the basic idea this particular this particular difference here right is not just because of year made, but Because of a combination of year made and enclosure right, the fact that this is 9.7 is because enclosure was in this branch and year made was in this branch. So, in other words, we could say the contribution of enclosure interacted with year made is minus 0.3 yeah, and so what about that difference? Well, that's an interaction of year made and hours on the meter, so you made interacted with, am using star here not to mean times but to mean interacted with it's a it's a kind of a common way of doing things like ours formulas.
Do it this way, as well here made by interacted with meter, has a import has a importance, sorry, a contribution of minus 0.1. Perhaps we could also say from here to here that this also shows an interaction between meter and enclosure like with one thing in between them, so maybe we could say meter by enclosure equals and then what should it be? You know should it be 0.6? I sorry minus 0.6. I mean some ways that kinda seems unfair because we're also including like the impact of a vm aid right. So maybe it should be. Maybe it should be minus 0.6. You know - maybe we should add back this point to right, and these are like details that I actually don't know the answer to right like how should we best kind of assign a contribution to each pair of variables in this path right, but but clearly conceptually. We can write the pairs of variables in that path. All represent interactions, all right. Yes, Chris can you? Could you pass that to Christmas? Why don't you first have to be next to each other in the tree. I I mean I'm not gon na say it's the wrong approach. I I don't think it's the right approach, though, because it feels like this path here. Mayda and enclosure are interacting, so it seems like not recognizing that contribution is throwing away information, but I'm not sure you know.
I had one of my staff at cattle actually do some R & amp D on this a few years ago and actually found you know - and I wasn't close enough to know how they dealt with these details, but they got it working pretty well, but unfortunately it Never saw the light of day is a software product, but like this is something which you know. Maybe a group of you could get together and build. You know I mean, do some googling to check, but I really don't think that there are any interaction feature importance. Parts of any open source library can you cross the flip. Wouldn't this exclude interactions, though, between variables that don't matter until they interact so say your road never chooses to split down that path, but that variable interacting with another one becomes your most important split. I don't think that happens right, because if this, if there's an interaction, that's important only because it's an interaction and not in a univariate basis, it will appear. Sometimes they cut, assuming that you set max features to less than one, and so therefore it will appear in in some file what is meant by indirection or is it multiplication ratio addition? Interaction means appears. Branches appears on the same path through a tree like an interaction, in this case the tree there's an interaction between enclosure and ear made because we branched on enclosure, and then we branch on your mode. So to get to here.
We have to have some specific value of enclosure and some specific value of you made sorry. I just membranes kind of working on this right now. What if? What? If you went down the middle Leafs between the two things, you're trying to observe - and you could just sort of norm - and you would also take into account what the final measure is, so I mean if we extend the tree downwards - you'd have many measures both of Like the two things you're trying to look at, and also the in between steps, there seems to be a way to like average information out in between them. There could be so I think what we should do is talk about this on the forum. I think this is fascinating and I hope we build something great, but I need to do my live coding. So, let's yeah, that's a great discussion, thinking about it and news of experiments and so to experiment with that. You almost certainly want to create a synthetic data set first right.  There's this interaction effect and there isn't that interaction effect. And then you want to make sure that the feature importance you get at the end is what you expected right, and so probably the first step would be to do single variable feature importance using the tree interpreter, style approach, and one nice thing about this is like It's it doesn't really matter how much data you have like all you have to do to calculate feature.
Importance is just like slide through the tree right, so you should be able to write in a way. That's actually pretty fast, and so even writing. It in pure Python might be fast enough, depending on your tree size. Okay, so we're going to talk about extrapolation, and so the first thing I want to do is create a synthetic data set that has a simple linear relationship. We're going to pretend it's like a time series right, so we need to basically create some X values. So the easiest way to kind of create some synthetic data of this type is to use linspace, which just creates some evenly spaced. Some evenly spaced data right between start and stop with by default 50 observations. So if we just do that right there, it is okay and so then we're going to create a dependent variable. And so, let's assume there's just a linear relationship between x and y and, let's add a little bit of randomness to it right so uniform random between low and high. So we could like add somewhere between like minus 0.2 and 0.2, say okay, and so the next thing we need is is a shape right, which is basically how what dimensions do you want this, this randomness, these random numbers to be, and obviously we want them to Be the same shape as X's shape, so we can just say X, dot shape, okay, so, in other words, that's xscape.
Remember when you see something in parentheses with a comma, that's at Apple with just one thing in it: okay, so this is of shape 15 and so we've added 50 random numbers, and so now we could plot those okay, so shift-tab, x, comma y, all right! So there's no data, okay, so like when you were both working as a data scientist or for doing your exams in this course. You need to be able to like quickly whip up a data set like that, throw it up in our plot without thinking too much. Okay and like, as you can see, you don't have to really remember much if anything you just have to know how to like hit shift. I have to check the names of parameters, and you know everything in the exam will be open up and broke up and internet, so you can always like google her something to try and find linspace if you forgot what it's called all right. So, let's assume that's out data all right, and so we're now going to build a random first model and what I want to do is build a random first model that kind of acts as if this is a time series. So I'm going to take this as a training set right. I'm going to take of this as our validation or test set just like we did in you know, groceries or bulldozers or whatever okay. So we can use exactly the same kind of code that we used in split bells right.
So we can basically say X, train comma X, tau equals x up to 40 comma X from 40 okay, so that just fits it into the first audio since the last 10 right, and so we can do the same thing for Y and there we go okay. So the next thing to do is we want to create a random forest, okay and fit it, and that's going to require X's and Y's all right. Now, that's actually going to give an error and the reason why is that it expects X to be a matrix? Not a vector because it expects X to have a number of columns of data right, so it's important to know that a matrix with one column is not the same thing as a vector all right. So if I try to run this right, expect a 2d array got 1d array instead, so we need to convert our 2d array into a 1d array. So remember I said X, dot shape is 50 comma right, so X has one axis. So here's important to measure X is rank is 1. The rank of a variable is equal to the length of its shape. How many axes does it have so a vector we can think of as an array of Rank 1 matrix as an array of Rank 2? I very rarely used words like vector and matrix because, like they're kind of meaningless, specific examples of something more general, which is they're all n, dimensional, tensors, right or n, dimensional, arrays, ok, so an n dimensional array, we can say it's a tensor of Rank. They basically mean kind of the same thing: physicists get crazy when you say that, because to a physicist, a tensor has quite a specific meaning, but in machine learning we generally use it in the same way.
Okay. So how do we turn an array, a one-dimensional array, into a two-dimensional array? There's a couple of ways: we can do it, but basically we slice it right. So colon means give me everything in that axis: right, colon, comma, none means give me everything in the first axis, which is the only axis we have, and then none is a special indexer, which means add a unit axis here. So let me show you that is a sort of shape: 50, comma one. So it's a prank. It has two axes. One of them is a very boring access right. It's a length, one access. So, let's move this over here, there's 1, comma, 50, ok and then to remind you, the original is just 50 right, so you can see I can put none as a special indexer to introduce a new unit axis there. Ok, so this thing has one row in 50 columns. This thing has 50 rows of one column. So that's what we want right. We want 50 rows and one column. This kind of playing around with ranks and dimensions is going to become increasingly important in this course and in the deep learning course right. So I spend a lot of time slicing with non slicing, with other things, try to create three dimensional, four dimensional chances and so forth. I'll show you a trick I'll show you two tricks. The first is you never ever need to write karma, it's always assumed.
So if I delete that this is exactly the same thing, okay and you'll see that in code all the time, so you need to recognize it. The second trick is this is adding an axis in the second dimension right or I guess the index one dimension. What, if I always want to put it in the last dimension right and like often our tensors change dimensions without us? Looking because, like you, went from a one channel image to a three channel image, or you went from a single image to a mini batch of images like suddenly, you get new dimensions appearing so to make things general, I would say this dot: dot, dot, dot, dot, Dot means as many dimensions as you need to fill this up. Okay, and so in this case, it's exactly the same thing, but I would always try to write it that way, because it means it's going to continue to work. As I get you know, higher dimensional tenses alright, so in this case I want 50 rows in one column, so I'll call that say X, okay, so let's now use that here and so this is now a 2d array, and so I can create my random forest. Okay, so then I could plot that - and this is where you're going to have to turn your brains on, because the folks this morning got this very quickly, which was super impressive. I'm going to plot white rain against MDOT predict next rain. Okay, before I hit go, what is this going to look like yeah? It should basically be the same right. A predictions hopefully are the same as the actuals, so this should fall on a line, but there's some randomness, so it won't. Quite. I should have used scatter plot, okay, all right, so that's cool right.
That was the easy one. Let's now do the hard one, the fun one. What's that going to look like you? Okay, so I'm gon na say no, but nice try. You know it's like hey we're extrapolating to the to the validation. That's what I'd like it to look like, but that's not what it is going to look like think about what trees do and think about. Think about the fact that we have a validation set here and a training set here so think about a forest is just a bunch of trees. Well, the first tree is going to okay, Melissa is going to have a go. Can you pass that to Melissa? Well, let's start grouping yeah, that's what I mean: that's what it does okay, but you know, let's think about how it groups the dots so yeah, I'm guessing, since all the new data is actually outside of the original scope, it's all going to be. Basically the same. It's like one huge group, yeah right so like we make it like, forget the forest. Let's create one tree right, so we're probably going to split somewhere around here first and then we're kind of probably split somewhere around here and then we're going to split somewhere around here and somewhere around here right, and so our final split is here right. So our prediction: when we say okay, let's take this one, and so it's going to put that through the forest right and end up predicting this average. It can't predict you anything higher than that because there is nothing higher than that to average right.
So this is really important to realize if a random forest is not magic right, it's just returning the average of nearby observations where nearby is kind of in this like tree space. So, let's run it, let's see if Tim's right but holy, that's awful right and like if you don't know how, when firsts works - and this is going to totally screw right, if you think that it's actually going to be able to extrapolate to any kind of data, It hasn't seen before, like particularly like future time periods. It's just not like it. Just can't it's just averaging stuff. It's already seen. That's all it can do okay, so we're going to be talking about like how to avoid this problem. We talked a little bit in the last lesson about trying to avoid it by just like in avoiding unnecessary time, dependent variables where we can write, but in the end, if you really have a time series that looks like this, we actually have to deal with a Problem all right, so one way we could deal with the problem would be used like a neural net right use, something that actually has a function or shape that can actually like fit something that can she fit something like this right and so then it will extrapolate Nicely another approach would be to use all the time series techniques you guys are learning about in the morning class to fit some kind of time, series right and then D trend it right and so then you'll end up with D, trended dots and then use the Random chorus to predict those right and that's particularly cool right, because, if you're like imagine that your random forest was actually trying to predict data that, like I don't know, maybe it was two different states and so the blue ones. You know that down here and the red ones are up here right now.
If you try to use a random forest, it's going to do a pretty crappy job because, like time is going to see much more important, so it's basically still gon na. Like split like this, and it's going to split like this and then finally, once it kind of gets down to this piece, it'll be like oh okay. Now I can see the difference between the states right. So, in other words like when you've got this big timepiece going on you're, not gon na see the other relationships in the random forest until you've dealt and kill every tree deals with time. So one way to fix this would be with a gradient, boosting machine GBM right now. What a GBM does is. It creates a little tree right and runs everything through that first little tree, which could be like the time tree, and then it calculates the residuals and then the next little tree just predicts the residuals, so it'd be kind of like D trending it right. So GPM has handle this GBM still can't extrapolate to the future, but at least they can deal with time-dependent data more conveniently right. So we're going to be talking about this quite a lot more over the next couple of weeks, right and in the end that a solution is going to be just used neural nets right, but for now you know using it some kind of time series analysis, D Trend it and then use a random forest on that, isn't a bad technique at all and if you're playing around something like the Ecuador groceries competition.
That would be a really good thing to to fiddle around with alright see you next time.


--- layout: post status: publish published: true title: Episode 3.45 - Huxtable and Clampett author: display_name: Corey Fineran login: Corey email<EMAIL_ADDRESS>url: '' author_login: Corey author_email<EMAIL_ADDRESS>wordpress_id: 6314 wordpress_url: http://ivyenvy.com/?p=6314 date: '2011-10-29 07:38:33 +0000' date_gmt: '2011-10-29 12:38:33 +0000' categories: - Podcast tags: - Podcast - Tom Ricketts - Jed Hoyer - Jason McLeod comments: [] ---
Andy and Corey talk about the new Epstein regime, as Jed Hoyer and Jason McLeod are brought into the fold. We talk about the Epstein press conference and what stood out to us. We continue our praise of Tom Ricketts and his dedication to doing everything he can to help the Cubs build a winning organization.
We look at the Hendry years and predict how things will differ between the previous and present front offices. Then, we take a look at some specific players and situations and predict what Epstein and Hoyer will do.


File: ../src/away/library/naming/ConflictStrategyBase.ts
///<reference path="../../_definitions.ts"/> module away.library { /** * Abstract base class for naming conflict resolution classes. Extend this to create a * strategy class which the asset library can use to resolve asset naming conflicts, or * use one of the bundled concrete strategy classes: * *
 *
IgnoreConflictStrategy (ConflictStrategy.IGNORE)
 *
ErrorConflictStrategy (ConflictStrategy.THROW_ERROR)
 *
NumSuffixConflictStrategy (ConflictStrategy.APPEND_NUM_SUFFIX)
 *
 * * @see away3d.library.AssetLibrary.conflictStrategy * @see away3d.library.naming.ConflictStrategy * @see away3d.library.naming.IgnoreConflictStrategy * @see away3d.library.naming.ErrorConflictStrategy * @see away3d.library.naming.NumSuffixConflictStrategy */ export class ConflictStrategyBase { constructor() { } /** * Resolve a naming conflict between two assets. Must be implemented by concrete strategy * classes. */ public resolveConflict( changedAsset : away.library.IAsset , oldAsset : away.library.IAsset , assetsDictionary : Object , precedence : string ) { throw new away.errors.AbstractMethodError(); } /** * Create instance of this conflict strategy. Used internally by the AssetLibrary to * make sure the same strategy instance is not used in all AssetLibrary instances, which * would break any state caching that happens inside the strategy class. */ public create():ConflictStrategyBase { throw new away.errors.AbstractMethodError(); } /** * Provided as a convenience method for all conflict strategy classes, as a way to finalize * the conflict resolution by applying the new names and dispatching the correct events. */ public _pUpdateNames(ns : string , nonConflictingName : string, oldAsset : IAsset, newAsset : IAsset , assetsDictionary : Object , precedence : string ) { var loser_prev_name : string; var winner : IAsset; var loser : IAsset; winner = (precedence === away.library.ConflictPrecedence.FAVOR_NEW)? newAsset : oldAsset; loser = (precedence === away.library.ConflictPrecedence.FAVOR_NEW)? oldAsset : newAsset; loser_prev_name = loser.name; assetsDictionary[winner.name] = winner; assetsDictionary[nonConflictingName] = loser; loser.resetAssetPath(nonConflictingName, ns, false); loser.dispatchEvent(new away.events.AssetEvent( away.events.AssetEvent.ASSET_CONFLICT_RESOLVED, loser, loser_prev_name)); } } }


I'm a senior software engineer at Genesys.
I am the co-founder of rovercode, an open-source robotics platform for classrooms. We're always looking for new contributors, so check us out!
I like school and am fortunate to get to do a lot of it. I have a BS in Electrical Engineering and an MA in Teaching, and I'm working on an MSCS in Georgia Tech's revolutionary online program.
I write, mostly about non-coding things.
This best way to get in touch is to direct-message me on Twitter.


MarkedContentReference
1.3
Type
:: ( Name | =
"MCR"
) , required
1.3
MCID
:: Integer , required
1.3
Pg
:: Dictionary
PageObject
, indirect
1.3
Stm
:: PDFStream
Stream
, indirect
1.3
StmOwn
:: Array or Dictionary or PDFStream
Stream
, indirect


WALKING DOTS
Copyright (c) 2002 Walter Joris
This game is played on a 8x8 board:
- SETUP - Each player drops two friendly stones on empty cells.
- TURN - On each turn, each player passes or drops a friendly stone on an empty cell orthogonally adjacent to another friendly stone.
- GOAL - When both players pass, wins the player with more stones.
Notice that a player which encloses an area of empty cells, can automatically claim it.
An example This is an endgame, there's no need to continue playing. Black won 35-29.
This game can be player with just pencil and paper. It is presented at 100 Strategic Games for Pen and Paper by Walter Joris.


- calculateBitLength(String) - Static method in class de.passwordmanager.helpers.PasswordHelper
-
Calculates the bits of possible passwords from a given password.
- calculateCrackTime(String, int) - Static method in class de.passwordmanager.helpers.PasswordHelper
-
Calculates how many seconds it would take for a bruteforce-attack guess the
correct password given a number of guesses per second.
- calculatePossiblePasswords(String) - Static method in class de.passwordmanager.helpers.PasswordHelper
-
Calculates the number of possible passwords from a given password.
- charInCharset(char, char[]) - Static method in class de.passwordmanager.helpers.PasswordHelper
-
Determines if a character is in a given charset.
- checkPasswordHash(String, String) - Method in class de.passwordmanager.services.CryptService
-
Hashes the input string and checks if it is equal to a given hash.
- ClipboardHelper - Class in de.passwordmanager.helpers
-
Class for loading saving Strings to the clipboard.
- ClipboardHelper() - Constructor for class de.passwordmanager.helpers.ClipboardHelper
-
- containsCharset(String, char[]) - Static method in class de.passwordmanager.helpers.PasswordHelper
-
Determines if a String has a character that is contained in a given charset.
- convertSecondsTo(TimeName, BigDecimal) - Static method in class de.passwordmanager.helpers.TimeCalculator
-
Converts seconds to another time unit.
- convertSecondsToPrintableTime(BigDecimal) - Static method in class de.passwordmanager.helpers.PasswordHelper
-
Converts an arbitrary amount of seconds to a printable time estimation.
- countCharacters(String, char[]) - Static method in class de.passwordmanager.helpers.PasswordHelper
-
Counts the number of characters in a String that are in a specific charset.
- createNewList() - Method in class de.passwordmanager.services.TableService
-
Creates a new TableEntryList.
- createNewProfile() - Method in class de.passwordmanager.services.ProfileService
-
Creates an empty passwordgenerator profile and adds it to the list.
- createPasswordHash(String) - Method in class de.passwordmanager.crypt.PasswordHasher
-
Generates a password hash by generating a random salt.
- createPasswordHash(String, byte[]) - Method in class de.passwordmanager.crypt.PasswordHasher
-
Generates a password using the given salt.
- Cryptor - Class in de.passwordmanager.crypt
-
The purpose of this class is to encrypt or decrypt String messages using
AES-128.
- Cryptor() - Constructor for class de.passwordmanager.crypt.Cryptor
-
- CryptService - Class in de.passwordmanager.services
-
Service for encryption.
- CryptService() - Constructor for class de.passwordmanager.services.CryptService
-
Class contructor
- CryptUtils - Class in de.passwordmanager.crypt
-
Utility class for cryptography.
- CryptUtils() - Constructor for class de.passwordmanager.crypt.CryptUtils
-
- CSS_PATH - Static variable in class de.passwordmanager.helpers.IOHelper
-
The relative path leading to the resources.css package.


PCB FABRICATION |
Simple design of final product/Mechanical design:-When we started designing our project, we had a very sample image of our final product in our mind and we draw it on a page as it below. As it was a 3x3 led matrix (each matrix of 8x8 led) the best possible arranged was as below, for displaying analog clock on it beside many other sprites and messages. |
Printed Circuit Board (PCB) is easy and efficient way to make a complex circuit. It is printed on sheet of copper. First of all a schematic diagram of circuit is drawn on computer software. We are using eagle software but there are many others. Then the board of this schematic is made which is difficult and need practice. After that this board is converted into image form (.png) so the drilling machine could understand how to print the circuit on sheet. Here in Fab Lab SUKKUR IBA UNIVERSITY we have rolland machine (SRM-20) which perform printing of the circuit physically. After the complition of PCB devices are soldered onto the pcb.There are two types of devices.
The selection of these devices depend upon the requirement of the PCB and must be taken into account while making schematic of the circuit. Also the dimension of devices must be taken into account while making the board. |
This is small milling machine used for PCB fabrication and is available here in FAB LAB Sukkur IBA University. Engineered for optimum efficiency and productivity, the SRM-20 is a next-generation desktop mill that boasts a micro-step motor drive system for clean and precise contours and a phenomenal feed rate that's two times faster than previous generations Schematic and board made in eagle is converted into (.png) format and then is uploaded into http://fabmodules.org/ for further process. The final PCB is attached further in report. |
|
Schematic is basically electrical circuit diagram. It uses symbol to represent components and connection of components. |
This Gerber file is basically the image of a schematic. It shows each individual connection and type of component used |
In the below picture, the white lines are called traces. These traces are the physical paths that connect components and connectors on the PCB board. A PCB allows current and signals to be transmitted through it. Traces are made by 1/64 pin header. |
It is one of the important part of the PCB fabrication. DIP Components are dipped into these drills and then soldered. |
After all these processes (printing and soldering) the final product (board) will look as below. One thing that must be kept in mind is that every connection must be double checked after soldering and continuity check is compulsory so there is no short path provided. |


SweetAlert for Bootstrap
For more visit the
lipis.github.io/bootstrap-sweetalert
Try any of those!
Primary
Info
Success
Warning
Danger


All the tests
Here are all the tests associated with lispy:
The
interpreter
The
core of the language
itself
The
parser
The
executor


Home
Projects
HTML/CSS
C#
Javascript
About
Javascript Projects
Written with the guidance of
Epicodus


Current counting value: {{count}}
This counter is 10% faster, thanks to new technologies.
Current counting value: {{customCount}}
Current operated counting value: {{customOperatedData}}
Error: {{customError}}


Python, Django, MySQL, PostgreSQL, Git, SVN, jQuery, Javascript, Bootstrap, CSS3, HTML5, Linux/Unix, Agile, Social Media APIs(Facebook, Twitter, Linkedin and Flickr)
Web Apps built using Python/Django, MySql, SQLite, HTML5, CSS3, jQuery, Javascript.
QRCoder FREE QR Code generator.
QR Code Generator FREE QR Code generator.
Aloe Vera of Forever Affiliate Apps, Calculators Apps and more.
Pysoldev Pysoldev website.
Web Apps built using Python/Django, Postgresql, HTML5, CSS3, jQuery, Javascript.
Royalorchard Python/Django Hotel Booking and Reports
FBYC Python/Django Blog and Reports
Sayah Python/Django Survey Site
GottenCard FREE Card Creator for Discount Card, ID Card, Business Card, VIP Card.
PosterFast FREE Poster Creator for Home, School and Business Use.
Holisticab A Simple Collection of Alternative Medicine.
© {% now "Y" %} Arbin Bulaybulay. Built on top of Django .


I’ve been back home for a week after a very busy week in San Francisco. I hosted a one-day Watir conference and then attended a three-day Selenium conference where I gave a keynote talk. After the conference was over we parked our car in a sketchy part of town for dinner, and me and three others had our laptops stolen out of our rental van. Some one broke the window and snatched our bags.. I also spent a day in Convio’s Berkeley office where I got a chance to catch up with several of my colleagues there. This week I’ve been catching up on my day job and replacing the things that were stolen.
There has been increasing cooperation between the Watir and Selenium teams, both on our projects and in organizing the conferences. Although we sold our own tickets to Watir Day (Sunday), the Selenium conference team made the hotel arrangements for us (both conferences were at the same hotel); and in the end, ended up subsidizing some of our costs. They also wanted to make sure that some one from the Watir team was on the Selenium conference committee. Charley Baker was generous to volunteer to help them select talks for the program. And of course, they invited me to keynote the Selenium conference. That talk should show up on Youtube at some point. I’ll talk more about the points I made in my keynote some other day.
We had about 60 people for the Watir Day and some 250 for the Selenium Conference. Both events were sold out. They did a lot more promotion for the Selenium conference that we did, and ended up with a waiting list of a 100 people, many of whom were waiting for multiple tickets. They are planning to have a much larger event next year. We’ve also been talking about folding the Watir event into the Selenium conference, something that actually sounds very attractive to me. I forsee having a one-day Watir track and then having a track on the second day that would focus on topics like Cucumber and Jenkins (nee Hudson) that would appeal equally to Selenium and Watir users. I see a lot of benefits to pulling our communities more closely together. For example, the conferences had several talks on Page Objects, but I thought the best was during the Watir day, and that Selenium users would have benefitted from the chance to see it.
I have a whole lot more I’d like to talk about, but am going to post this now. I do need to give my thanks to Alister Scott who’s help was invaluable for getting the Watir Day organized and Ashley Wilson who ran the Selenium conference and was a big help for the Watir Day too.


There's not much here.
Just words.
And you're reading them.
We've become obsessed with fancy designs, responsive layouts, and scripts that do magical things.
But the most powerful tool on the web is still words.
I wrote these words, and you're reading them: that's magical.
I'm in a little city in British Columbia; you're probably somewhere else. I wrote this early in the morning, June 20th, 2013; you're probably reading it at a different time. I wrote this on my laptop; you could be reading this on your phone, a tablet or a desktop.
You and I have been able to connect because I wrote this and you're reading it. That's the web. Despite our different locations, devices, and time-zones we can connect here, on a simple HTML page.
I wrote this in a text editor. It's 6 kB. I didn't need a Content Management System, a graphic designer, or a software developer. There's not much code on this page at all, just simple markup for paragraphs, hierarchy, and emphasis.
I remember teaching my daughter to code HTML when she was 8. The first thing she wrote was a story about a squirrel. She wasn't "writing HTML"; she was sharing something with the world. She couldn't believe that she could write a story on our home computer, and then publish it for the world to see. She didn't really care about HTML, she cared about sharing her stories.
You are still reading.
Think about all the things you could communicate with a simple page like this. If you're a businessperson, you could sell something. If you're a teacher, you could teach something. If you're an artist, you could show something you've made. And if your words are good, people will read them.
 Instead of starting with a style guide or a Photoshop mockup, start with words on a page.
What do you have to say? If you don't know, there's not much use in adding all that other cruft. Just start with one page, with a single focus. Write it and publish it, and then iterate on that. Every time you're about to add something, ask yourself: does this help me communicate better? Will that additional styling, image, or hyperlink give my audience more understanding? If the answer's "no", don't add it.
At its heart, web design should be about words. Words don't come after the design is done. Words are the beginning, the core, the focus.
Start with words.
Cheers,
Justin Jackson
@mijustin
Do you like to make stuff on the web? Subscribe to my newsletter.


Resize Map
Resize Pie Chart
Resize Line Chart
Resize Lava Logo
Resize Heco Logo
Resize Legend
We noticed that you do not have a saved Map Layout. You can now position and resize your display as well as hide any elements that you do not want shown.
Each map element is represented by a box and icon here. You can drag an drop them wherever you want to and the cooresponding element that is being projected will move to the cooresponding position
Click on the Settings button of any element and you will see options to resize, hide, or show that element.
When you are finished, click Save Changes to save this layout for the next time you load the application or Cancel Changes if you want to start over.