nng episode 1 (part 1 + 2) script
note that some formatting might be a bit screwed up as the .docx conversion wasn't perfect with some unicode characters (like ').
Episode 1 - Welcome To Tokisadame [Part 1]
Plot
Nano fires her rocket punch at school by accident, she goes flying through the roof, Nakamura tries to get Nano before Hakase and Sakamoto, Nakamura has a breakdown therefore Hakase and Sakamoto get there first, they take Nano home, Yukko and Mio talk about the incident, then at midnight, Hakase is repairing Nano but spills some strawberry juice on the circuits during the process.
Script
[Fade in]
[Scene: Local Hardware store (tools aisle)]
Nakamura-Sensei, High School Science Teacher, looks tired and she always thinking to herself.
Coffee mug.
Nakamura
(thinking)
I've been trying to find a way to get Nano into my hands. Nothing I've tried so far has worked. My latest plan is more of a' a last resort.
Nakamura walks to the heavyweights aisle of the store, showing various weights and anvils of different shapes and sizes, until she finds the anvil she thinks will work. She attempts to pick it up, puts it on the floor, and pushes it back to the checkout desk.
The anvil faintly says ACME in it in reference to the old version of the script.
[Cut to checkout desk]
Nakamura
(Out of breath)
I'll purchase this anvil, please.
Cashier
(Curiously)
...Do you need something to carry that around?
Nakamura
N-no, I'm fine.
Cashier
Alright then, that'll be $450, please.
Nakamura gives the money to the cashier, which is tied together with a ribbon. The cashier prints out a receipt, and Nakamura takes it.
Nakamura
(Thinking)
Even if this is a price to pay to get ahold of Nano, it will definitely be worth it!
Nakamura pushes the anvil out of the store, dropping the receipt in the street.
[Cut to a calendar and a clock]
[One of the day pages fall off, and the clock turns forwards]
[The next morning]
[Scene: Shinonome Laboratory (bedroom)]
Hakase, a young, spoiled, and very young/childish (yet gifted) scientist. Roll cake.
[An alarm clock, strikes 8:20 AM and plays it's ringtone.]
[Hakase, whose in her mat, tries to stop the alarm with her hand, but decides to press the button next to the clock, crushing the clock with a hammer, and then a new alarm clock rises from the original spot of the crushed alarm clock, replacing it.]
Hakase
(tired)
*yawn* I feel like every morning just gets shorter and shorter.
[Hakase grabs a custom-made tablet with an antenna coming out of the top, that shows Nano's abilities, the options to toggle them, along with stability/health.]
The back of the tablet has a wind-up key for a logo, parodying the iPhone.
[Scene: Downstairs]
Include a glass filled with Strawberry juice as it's used later in the script
Hakase enters the scene, going downstairs.
Sakamoto
Well, you're awake early for once. What's the occasion?
Hakase
(slightly hysterical)
What's the occasion?!
Sakamoto
Calm down, what is it?
Hakase
What do you mean, 'what's the occasion?' There's a new season of 'Bobbie McSponge' and I really don't want to miss it.
Sakamoto
Wait, so you're all hyped about a new season of a show that probably only the minority cares about?
Hakase
Minority? Say that again, but slowly.
Sakamoto
(slowly)
Minority.
Hakase
*sigh* Not literally' Anyway, what does Nano even watch?
Sakamoto
(slightly annoyed)
I don't know. I didn't program her.
Hakase
Well, I don't know either, even though I programmed her. Blame neural networks.
Sakamoto
What's a neur-
The tablet starts beeping, Hakase looks and to her dismay, there is a message on it that says that the rocket punch is loose.
Hakase
(worried)
AI stuff, but that's the least of our problems. There's something wrong with Nano, and she's just about to leave!
Sakamoto
(concerned, angry)
...Alright, since when did you get a-
Hakase
Since always! I need to get to Nano before she leaves!
Sakamoto
'Okay.
Hakase rushes towards Nano, who is just about to open the door.
Hakase
(distant)
Nano!
Nano
(holding door open)
Hm?
Hakase
(worried)
Nano, your arm-
Nano
Hakase, I can't talk right now, school starts in an hour and a half!
Hakase
(still worried)
This is important!!
Nano
It's only a hour and a half, Professor. You can tell me when I'm back by 3:00 PM.
Hakase
(still worried, still)
But you-
Nano
If it's about your shark chocolates, then no.
Hakase
But-
Nano stares at Hakase.
Hakase sighs.
Hakase
(somewhat defeated)
...Fine, I'll tell you after.
Nano
Okay then.
Nano leaves the house and closes the door behind her.
Hakase
(frustrated)
Why did I even bother saying?
Sakamoto
Say about what?
Hakase jumps in surprise.
Hakase
(stuttering, and also startled)
What non-functional robot punch?
Sakamoto
'*sigh* What did you do this time with Nano?
Hakase
(turning back from Sakamoto, crossing arms)
I' I don't know what you mean!
Sakamoto
I can easily tell you're lying, now tell me what you did with Nano.
Hakase remains silent.
Sakamoto
When I walked up to you, you said something about a robot punch. What about that did you do with Nano?
(continued silence)
Hakase
I said n-nothing! I don't know what you're talking about!
 I'll spill your strawberry juice if you don't tell me, and I'll spill it on the rug.
Hakase gasps in shock.
(tense silence)
Hakase
(crying)
I'll talk!
Sakamoto
Alright, now tell me about this supposed rocket arm.
Hakase
I was doing some work with a rocket arm that I gave Nano last week.
Sakamoto
Didn't you just give Nano a toaster function last month?
Hakase
Uhhhh'
[Flashback: Kitchen]
Zoom out from eggs cooking, Sakamoto is just kind of standing there in the background
After a bit, Hakase presses a button on her tablet, causing the top ring of Nano to slide open a bit, shooting 3 slices of toast onto Hakase's plate.
Hakase
I thought I said 4 slices-
Another slice of toast shoots out and lands on to Hakase's face.
Hakase
(muffled)
...Thanks.
[Zoom back into eggs]
[end of flashback]
Hakase
Oh yeah, now I remember. That was really good toast.
Sakamoto
(one word at a time)
But what about the rocket arm?
Hakase
That was last week-
Sakamoto
No, I mean what about it that's making you so worried?
Hakase
*sigh* I... made it too strong.
Sakamoto
...How strong?
Hakase
Several tons of force.
Sakamoto
(slightly worried)
Wouldn't that kill someone?
Hakase
Uh, It could blast someone into pieces and still have enough force to blow through an entire building or two.
Sakamoto
So' Where's Nano off to?
Hakase
School.
Sakamoto
(breaking down)
Oh no. Ohhh no.
Hakase
Is something wr- wrong?
Sakamoto
(ultimately worried)
Of course there's something wrong, kid!! Nano could possibly brutally kill someone with that kind of force! We have to get to her and tell her about her arm now!
Hakase
Should we try and chase after her then?
Sakamoto
You're seriously asking that question: Isn't it obvious? Of course we need to chase her!
Hakase
Then we should go then!
Sakamoto
Then let's go then!
Hakase and Sakamoto head for the door, but Hakase stops for a second and grabs a juice pack (strawberry flavor)
Sakamoto
...Really?
Hakase
What? You said you would spill it, and I am not-
Sakamoto pushes Hakase out the door so they can get to Nano.
Sakamoto
(while pushing)
Let's just get going already.
Hakase
Fine.
[Scene: Outside]
[Halfway from Shinonome Laboratory to School]
[Cut to Nakamura standing on a streetlamp, tying a rope to the anvil from earlier]
Nakamura
(thinking)
What part of me thought that using an anvil would work? '*sigh* Oh well, I'm this far in and there's no going back.
Nakamura puts the knife close to the rope, waiting for Nano.
Moments pass.
[Zoom out (no slide, instant scene change)]
Nakamura
(thinking)
Any moment now.
More moments pass.
The camera slowly zooms out of the shot in a fading sequence.
[Short intermission bumper]
[Cut to Tokisadame High School: Bottom floor, hallway]
(CHARACTER INTRODUCTIONS WHATEVER)
Weboshi, Fe-chan and Misato are walking down the hallway.
Weboshi
So, Fe-chan, what are you going to do when the weekend comes?
Fe-chan
I'm planning to host a house party this saturday night.
Misato
(questioning)
Why would you do that? By what I've seen, house parties usually end up as a wreck.
[Background - Nano walks in through a door, Weboshi, Misato and Fe-chan don't notice]
Weboshi
Misato's right, if you want to plan a house party, at least know what you're doing.
Misato
For example, do you invite via posters or via invitations?
Fe-chan
(curious)
I'm not sure, actually! What do you two think would be the best?
Weboshi
Definitely via invitations, if you're just going to go and invite people who're interested to come over to your house, it's probably going to be a disaster in the end.
Fe-chan
So' more of a 'girl's night out' then?
Misato
Whatever floats your boat, I guess. However, I'm not the kind of person who enjoys shopping or y'know, that kind of stuff.
Fe-chan
Hmm. Well, do you want to come then?
Weboshi
Sure.
Misato
I'll also come if it bothers you that much.
Fe-chan
No, no, it's fine. Who else should I invite?
Misato
Possibly those three at class 1-Q.
Fe-chan
You mean Yuuko, Mio, and Mai?
Weboshi
Isn't Nano there too?
Misato
Yeah, her too.
Fe-chan
I'll see if I can invite them!
Weboshi
Sounds good.
As the three walk, the camera slides to the 1-Q door, and the scene goes to inside the class.
Sakurai
Today, we have a test on Japanese.
Background chatter is heard in the background.
Yukko sighs.
Mio
What is it, Yukko?
Yukko
I didn't study'
Mio
(unamused)
...Again.
Yukko
Yeah, again.
Mio
Yukko, look. One day you'll have to do homework yourself, and not rely on copying others homework, like mine or Mai's. One day, we won't be there.
Yukko
(disappointed)
...Oh.
Mio
...What I'm trying to say is to believe in yourself, just try and do the test anyways.
Yukko
(slowly becoming determined)
...I'll try!
Mio
There you go!
Sakurai
Everyone, please remain seated and silent: the tests are being handed out.
Papers are passed out, and everyone but Yukko starts writing. Yukko looks at her paper, stern to complete the test.
Yukko
(thinking)
I can't do this.
[Cut back to Nakamura]
Nakamura
(tired, thinking)
I've been waiting for an entire hour now, where is she??
Annaka walks past Nakamura, and then stops and turns around.
Annaka
(confused)
...What are you doing?
Nakamura
(startled, still a bit tired)
*yawn* Huh, what?
Annaka
L-look, whatever it is, school does start in about half an hour, you should get going.
Nakamura
(starting to fall asleep)
I'm doing this' for a'
Nakamura falls asleep, and accidentally cuts the rope, breaking the streetlamp along with it due to the pressure, as Nakamura falls down and lands on the anvil.
Nano finally walks up to the broken streetlight.
Nano
(confused)
What happened here? ...And why is Nakamura unconscious?
Nano's inbuilt watch beeps.
Nano
(surprised)
30 minutes until school starts?! I gotta get going!
Nano continues walking, and looks back at Nakamura, not waking up.
Nano
...I should bring her as well.
Nano goes back to Nakamura, and starts dragging her as Nano walks to school.
Annaka
(confused)
Ehhhhh??
[Cut to Hakase and Sakamoto]
Hakase and Sakamoto are running, out of breath, and tired.
Hakase
(whining)
I want to go back home!
Sakamoto
(annoyed)
What about that rocket punch?
Hakase
But I want to sleep!
Sakamoto
Well, that's just too bad, isn't it?
Sakamoto and Hakase run past a juice stand
Hakase slows down slightly, looking back at the stand
Sakamoto
You're not having any juice until we sort Nano out.
Hakase
Aw!
Sakamoto
Wait, didn't you bring a pack of strawberry juice earlier?
Hakase
(looking away)
...No.
Sakamoto
I can tell you're lying.
Hakase
I'm not lying!!
Sakamoto
*sigh* Can we just continue on? We'll get the juice after.
Hakase
Fine!
Hakase and Sakamoto dash out of the scene, only for Hakase to run right for the stand only a few seconds later, with Sakamoto trying to stop Hakase.
[Short intermission bumper]
[Calendar cutscene, but the time changes instead of the calendar]
Sakamoto
So, kid. Do you know where the school is?
Hakase
(thinking)
Uhh' No.
Sakamoto stops.
Sakamoto
Wait, so you're telling me that you don't know where the school is?
Hakase
Nope.
Sakamoto
Then where are we supposed to go in the first place to get there?
Hakase looks around and notices a broken streetlamp with a trail of coffee.
Hakase
(curious)
How about that trail?
Sakamoto
Why? How can we be so sure it leads to the school?
Hakase
What else do we have?
Sakamoto
A GPS?
Hakase
The GPS doesn't work.
Sakamoto
Why not?
Hakase
How should I know? I thought the satellite I built was working the time I launched it into space.
Sakamoto
Wait, what?
[Cut to the satellite being stuck on the moon, then cut back to Hakase and Sakamoto]
Hakase
...Nevermind, let's just follow the trail!
Sakamoto
(bored)
...Whatever.
[Cut to Fe-chan]
[Highschool, bottom floor | hallway | corkboard]
Fe-chan pins a paper onto the corkboard.
Fe-chan
That should do!
Weboshi notices Fe-chan at the corkboard and walks up to her.
Weboshi
Hey Fe-chan.
Fe-chan
Hm?
Weboshi
I thought you were going to invite select people?
Fe-chan
Well, I was deciding between invitations or people who are interested, so I ultimately decided to go for the other option, simply because it would have more people come over. Also, it was hard choosing select people, since I think everyone here is great equally.
Weboshi
Well, at least you know what you're doing. Good luck with the house party!
Fe-chan
Thanks!
[Cut to Hakase and Sakamoto]
Hakase and Sakamoto both follow the trail of coffee, looking directly at the ground, soon later the trail ends.
Sakamoto
Seems like the trail ends here.
Both of them simultaneously look up and see Nakamura on the ground under a pool of coffee.
Hakase
Look!
Hakase points at the giant machine next to Nakamura.
Sakamoto
You're just going to ignore the woman on the ground?
Hakase
It's a recovery unit!
Sakamoto
A what.
Hakase
You don't know what a recovery unit is?
Sakamoto
Listen here, it's no time to play games, we have to find Na-
Hakase
It brings dead people back to life!
Sakamoto
Stop talking nonsense, I said we have to find Na-
Hakase
But it does bring people back to life!
Sakamoto
Stop talking about this 'Recovery unit', it's not real.
Hakase
Then what is it?!
Sakamoto
I don't know, maybe it just burns garbage! What I do know is that that isn't a 'recovery unit'-
Before Sakamoto can finish, the machine turns on and starts shaking rapidly.
Sakamoto
(nervously)
Um' Should we run?
Hakase
Don't worry, it's safe.
Sakamoto
How do you even know that?!
The machine stops shaking and it opens, Makoto leaves the machine.
Makoto
Jeez, I gotta be careful while training Igo Soccer, I don't want to die a third time.
Sakamoto stares intensely at the situation he had just witnessed.
Sakamoto
Wha' Wha, wha, what in the'
Hakase
See? I told you so! You owe me a box of apple juice now!
Sakamoto
When did we agree on that?! Wha-what was that?! What happened?!
Hakase
The machine regenerated a school boy.
Sakamoto
Bu-but how?! How even!?
Hakase walks forwards.
Sakamoto
Don't ignore me! Hey!
Hakase
'Sakamoto you're being too noisy, we have to find Nano now, come on.
Sakamoto
He-hey! Arrrrgh! Forget it! Wh-what should we do now?
Hakase
Ah.
Sakamoto
You don't know, do you?
Hakase
I, umm' How about we ask the boy?
Sakamoto
*sigh* It's the only thing we can do now, isn't it?
Hakase
Hey mister!
Makoto
(turns around)
Eh? Me?
Sakamoto
Hey, kid! Could you tell us where Tokisadame school is located?
Makoto
(raises eyebrow)
'Pfft' Hahahahaha!!
Sakamoto and Hakase
(tilt heads)
Eh?
Hakase
Um, mister?
Makoto
(keeps laughing)
Sakamoto
Le-let's get out of here, I think we asked the wrong person.
Hakase
(scared)
Ye-yeah'
Sakamoto and Hakase start walking away slowly.
Makoto
(while laughing)
W-wait! Pfft- It's just, it's just'
[Hakase and Sakamoto turn around]
Makoto
You are at Tokisadame school, that's all!  Pfft! I just couldn't hold it!
Sakamoto
(embarrassed)
It- It's not that funny!
Hakase
(embarrassed)
Y-yeah!
Makoto leaves before they know it.
Hakase
(angry)
Where did he go?!
Sakamoto
(angry)
You know what? Forget him! He said this was Tokisadame School. If it really is, then let's search for an entrance.
Hakase
(angry)
Al-alright'
[Cut to Makoto]
Makoto is seen on the other side of the school, he walks towards a door with a weird device where the handle should be, he takes out a card and puts it in the machine, opening the door.
Makoto
Ah, I'm lucky that I didn't forget my student ID at home, otherwise I'd be in too much trouble.
Makoto walks through the door, which leads to the school's hallway on the first floor.
Makoto
...I wouldn't be able to get in after dying like it normally happens.
Makoto
(thinking)
It's weird though, why do I have to go through the back door instead of the front door? ' Meh, who cares, I have to get back to class.
[Cut to Makoto in the hallway, a few feet away from the corkboard]
Before Weboshi and Fe-chan can leave, Makoto sees the paper Fe-chan placed on the corkboard, Makoto looks at the paper
(Show a childish house party invitation)
Makoto
Pfft...
Fe-chan
Eh?
Makoto
Who made this?
Fe-chan
(getting the wrong idea)
E-eh? Um, so-sorry, but I think mainly girls are going to the house party and, uh, I don't know if you'd feel comforta-
Makoto
(smirking)
It looks so childish.
Fe-chan and Weboshi
Ah.
[Cut to Fe-chan's heart]
Fe-chan's heart shatters in two, glass shattering is heard.
[Cut back to Makoto, Weboshi and Fe-chan]
Makoto
Are these supposed to be people?

Fe-chan
Ah.
Makoto
(turns around, unaware)
What is it?
Fe-chan
...I'
Makoto raises his eyebrow and looks at Fe-chan
Fe-chan
(heartbroken)
I' I know ri-right? Th-they look so, so bad!
Makoto
I know right? Oh! I forgot I was supposed to go!
 He probably was unaware that you made it.
Fe-chan
'
Fe-chan takes off the paper
Fe-chan
(sad)
Y-you know what' Maybe I won't make the house party.
Weboshi
Eh?! Bu-but you've been talking about it the whole day!
Fe-chan
(sad)
Maybe' Maybe next week. Can we go back to classes now?
Weboshi
'If it makes you happy, I can buy juice for you.
Fe-chan
(forgets past events)
Gasp! Really?
Weboshi
(smiles)
Yeah'
Weboshi
(thinking)
She completely changed after I told her I'd buy her juice'
Fe-chan
(excited)
I'm kind of thirsty right now, can we buy it already?
Weboshi
(happy)
Sure.
Weboshi and Fe-chan walk away from the scene. After a short bit, Fe-chan hears Hakase and Sakamoto, she stops and looks out the window out of curiosity to figure out where the noise is coming from.
[Outside the School]
Hakase
The door is locked!
Sakamoto
Well, that's just great.
Hakase
No it's not!!
Sakamoto
Kid, when are you going to realize when I'm sarcastic and when I'm not sarcastic?
Hakase
...Are you being sarcastic now?
Sakamoto
About earlier, yes.
Hakase
Oh' Well, how do you think we're going to get across this locked door?
Sakamoto
We can't.
Hakase
What do you mean?
Sakamoto
I mean, there's no possible way we can get through this door. It's locked, we can't go over it, and there's no form of doorbell or anything to use to get someone to open the door to begin with.
[Cut back to Fe-chan and Weboshi]
Fe-chan
Wait, Weboshi, there's someone out the window trying to get in the building!
Weboshi
Huh? Who are they?
Fe-chan
(trying to look out the window farther)
I don't know, they're too far to recognize!
[Room 1-Q]
(Yukko is face-planting the table due to not being able to finish the test)
Sakurai
The test is over, remember not to talk until all of the tests have been taken in.
(All the tests are taken in- except Nano's)
(Nano raises her hand, which starts to charge the rocket punch)
Nano
Uh, my test hasn-.... Nevermind, sorry.
Nano
(thinking)
Oh no. This can't be happening.
Nano's hand/arm starts to glow red (and makes a kind of charging noise)
Mio
Uh...
Yukko
(confused)
Since when did her arm become re-
[end: black screen, explosion sound]
Episode 2 - Welcome To Tokisadame [Part 2]
Script
(huge explosion, cut to Sakamoto and Hakase just looking at the explosion in shock and total horror)
(cut back to Nano, the camera follows Nano as she breaks through several classroom floors (CAMEOS AND RECOMMENDED CHARACTERS!) until she stops in the ceiling of a class)
Annaka
(distressed)
Ehhhhh??
Yukko
(paralyzed)
...That happened?
Mio
Looks like Nano broke through multiple floors for' Whatever reason.
Yukko
I hope she's fine'
Mio
Don't worry about it. If she is, then Hakase could simply just repair her.
Misato opens the door quickly
Misato
(confused/peeved)
What happened here? Why is there a giant hole in the ceiling?
Yukko
(annoyed)
I was just about to ask myself that!!
Mio
...What now?
Yukko
Well, I guess we just' continue on with the day?
Mio
How would we do that when there's those giant craters in the building now?
Yukko
...Beats me, actually.
Mio sighs in frustration.
[cut to Hakase and Sakamoto]
Sakamoto
(sarcastic)
And there goes the evening.
Hakase
(spiraling, walking around, breaking down, etc etc)
What're we gonna do, what am I going to do, what am I supposed to do?!
Sakamoto
Now what?
Hakase
(still obviously stressed out)
I don't know! We can't get in to actually see what's happened!
Sakamoto
Uh, then we just wait here?
Hakase
No, but at the same time yes' Wait, what did I mean?
Sakamoto looks around for something to get him and Hakase inside.
Sakamoto
...Hey, I think I see an open window in the building.
Hakase
Wait, so you're saying we should get someone's attention?
Sakamoto
That was the point.
 The engine's made of compressed air.
Sakamoto
Well, what are you going to do with that rocket? And don't try to blast a hole through the gate to pass through, that'd make things worse.
Hakase
Why would I blast through the gate?
Sakamoto
So what's the point of the rocket then? Is it like a firework?
Hakase
Just you find out and see!
Hakase throws the rocket at the ground and it bounces back up. The rocket flickers on and it flies straight through the open window of 1-Q
Sound effects of breaking glass and then other destruction noises, Sakamoto looking in horror
Sakamoto
(distressed)
You just said you wouldn't blast through the window! What are you doing?!
Hakase
Getting attention, Besides, I only said through the gate.
Sakamoto
But you're still destroying other property!
Hakase
It's only about the size of a toy anyway!
[Cut to the school hallway]
Fe-chan is impaled by the rocket.
Fe-chan
Urgh...
Weboshi
(startled)
Uh' Fe-chan?
Fe-chan falls onto the ground face first
Misato
...I think she's dead.
Weboshi looks outside the window
[Cut to Hakase and Sakamoto]
Hakase
(shouting in order for Weboshi to hear her)
Did I break anything??
[Back to Weboshi]
Weboshi
(also shouting in order for Hakase to hear her)
Well, you killed Fe-chan!
Sakamoto stares sternly at Hakase
Hakase
(shouting)
Fair enough!
[Short intermission bumper, then cut to Yukko and Mio]
Yukko
(trying to tape together test)
Mio, should we go check on Nano?
Mio
We're in the middle of a practice test for midterms, so I'm not sure if its-
Yukko points at the hole in the ceiling and coughs
Mio
...Fine, what's your idea on getting out of this?
Yukko
The idea I have is that we're going to come up with an excuse so we can get out the door.
Mio
What excuse will I use?
Yukko thinks for a moment.
Yukko
(a tad bit slower)
Uhhhh, just try and go with the flow.
Mio
'(sigh) Okay.
Yukko intentionally smudges some pencil graphite onto her hand
Mio
(thinking)
I see.
Yukko raises her hand, waiting for Sakurai to notice.
A short moment passes.
Mio
...How long does this take?
Yukko
(slowly getting impatient)
Any moment now'
Mio rolls her eyes.
Mio
Hang on, wait a minute.
Yukko
(arm starting to shake)
Huh?
Mio
Notice how Sakurai hasn't seemed to have told the class to be quiet yet? She hasn't even said anything at all for that matter.
Yukko
(puts down hand)
Only now, actually.
Mio
...That's because she's asleep.
Yukko
Teachers can do that?
Mio
...Well, the school bell rings at the end of the test.
Yukko
So, we should sneak out?
Mio
Either that or we go for your plan.
Yukko looks at Sakurai, whos sleeping.
Yukko
Your idea is better, come to think of it.
Yukko and Mio get up out of their chairs and crawl to the door while everyone else stares at them.
Annaka
...E-
Mio
Shhhhh.
Annaka
(very quietly)
Ehhhh??
Mio and Yukko eventually reach the door, before very quickly dashing out of it
Yukko
(relieved)
Phew. Now that that's done, what next?
Mio
We go and find Nano, obviously.. Do you have the memory of a goldfish?
Yukko
Uh, no.
Mio
*sigh of disbelief* Seems like it.
Yukko
(fake relief)
Phew. Now that that's done, what next?
Mio facepalms
Mio
There's no time for this, come on!
Yukko follows Mio.
Yukko
So uh, wouldn't Nano just be straight up?
Mio
That's what I'm thinking.
Yukko
So if Nano went straight up, we'll have to go up some floors to figure out where she stopped at!
Mio
The question is, how many floors?
Yukko
Better not be a lot, I'm not a fan of stairs.
Mio
We have an elevator, you know.
Yukko
Oh. I always took the stairs'
Mio
Well at least you're getting exercise, however we have to take the quick way up.
Yukko
No arguing with that!
[Cut to inside the school elevator]
The elevator door opens, and Yukko and Mio walk in, and proceed to look at the array of buttons.
Yukko
Which floor should we go to first?
Mio
Hmm' Well, I saw that the hole in the ceiling lasted for roughly 3 or so floors, so let's start at Floor 4.
Yukko
Okay then.
Yukko presses the button tagged with '4', meaning floor 4. The elevator dings once the button is pressed, and the doors close. Then, the elevator goes up, while some music plays in the background.
[Scene: Nakamura is asleep in the school field, close to the school regeneration unit]
Fe-chan gets regenerated by the SGU (school regeneration unit) and notices Nakamura on the ground.
Fe-chan
Nakamura?
Nakamura doesn't respond.
Fe-chan
Hm, must be asleep.
Nakamura then wakes up.
Nakamura
Huh? What? Where am I?
Fe-chan
(startled)
AH! Man, you scared me!
Nakamura
Oh, good afternoon, Fe-chan. How long was I here for, and what time is it?
Fe-chan
I just got back from.. Being dead, and'
Fe-chan looks at the clock located at the top of the entrance wall.
Fe-chan
It's about 1:49 PM.
Nakamura
Period 4? Isn't that when I have to do Science?
Fe-chan
Well, you did pass out, so Sakurai suggested we'd do a test we were originally going to do on Friday today instead because we already knew the material well enough!
Nakamura
...Oh. Then what are you doing, then?
Fe-chan
I was originally going to return back to the lesson, but then you were just sort of laying there on the ground'
Nakamura
I still haven't figured out why I was here instead of on the sidewalk.
Fe-chan
Well, I haven't seen you anywhere else today, so I'm unsure how you ended up here'
Nakamura
Oh.
Fe-chan
Moving on, I heard something happened in 1-Q earlier,
Nakamura
And...what would that be?
Fe-chan
Apparently, Nano accidentally fired her rocket punch and blasted herself through several floors...
Nakamura
(thinking)
Nano? This gives me an idea'
Nano? I have just the perfect plan...
Nakamura
(slightly anxious)
Fe-chan, what floor is she on?
Fe-chan
Unsure, but she definitely did go through 2 or more floors.
Nakamura
Alright, I'll go check if she's alright.
Fe-chan
...Alright. I'm going back to class.
Nakamura
You do that.
[Scene: Art Class (Floor 4, Class A [4-A])]
Everyone in the class Is working on paintings.
Misato
(relaxed)
Working on art sure does get my mind off of certain things..
Weboshi
So, what are you painting?
Misato
Haven't gotten a full picture yet, but I'm sure I'll figure it out.
Fe-chan opens the class door.
Fe-chan
Hey, I'm back.
Weboshi
Fe-chan! Where have you been? I thought you would've been here sooner!
Fe-chan
Well, I saw Nakamura outside, and that's the only thing that delayed me getting here.
Weboshi
...Oh.
Misato
Why was she outside?
Fe-chan
I personally think it should be best left a secret.
Misato
(confused)
...Ok?
Weboshi
Well, at least she's fine, right?
Fe-chan
(positive)
No worries, no worries. She's fine.
Misato
Positive?
Fe-chan
Positive.
Weboshi
Well, that concludes that then.
Well, that's all fine/good then.
Scene: Front of class 4-A's door (camera zooms out through door)
Mio and Yukko open the door.
Yukko
Is Nano here?
Misato
(startled)
AAAH-
Misato is holding an open bucket of (blue) paint, (with two eyes floating around near the edge, barely visible [BFB CAMEO]) attempting to pour some of it onto her palette: the sudden arrival of Mio and Yukko startles her and throws the paint up and it lands straight on her work, along with making her launch a rocket by accident, and then the rocket aims for the paint bucket.
Mio
Oh-
[Webcomic: Page that shows the moment before the rocket hits the paint.]
[Animated: Slow motion before the rocket hits the paint.]
The rocket hits the paint, causing an explosion, and everyone in the room is covered in paint [people closer obviously more than people further away]
Mio
...No.
Yukko
*cough*
Yukko and Mio step backwards and close the door.
Misato stares at her canvas, entirely covered in blue paint, and then groans in anger.
Mio
Well, that wasn't the room. What one do you think it is, Yukko?
Yukko
How about the room next door to 4-A?
Mio
4-B?
Yukko
Yeah, 4-B. It's the woodworking class!
Mio
*groan* I've had bad experiences in there.
Yukko
(confused)
What do you mean?
Mio
(reminded of previous events)
Nothing, nothing.
Yukko
...Alright, should I go in first?
Mio
I'd probably have a nervous breakdown if I went in first, so you go on right ahead.
Yukko
Alright.
Yukko opens the door, to find Nano embedded in the ceiling.
Yukko
Nano!
Mio
(walking in, relieved)
Thank goodness'
Yukko
(looks around)
Well Mio, I don't see why you're scared of this place, it seems fine.
Mio
That's because they cleaned up the accident, far as I see.
Yukko
(questioning)
Also, is it just me or is Nano still conscious? You got a stick, Mio?
Mio
Not really, but there's some small wood chunks over there near the wood cutters that could be taped together.
Yukko
That works too, I guess'
[Class 4-A]
Fe-chan
(Drying self with towel)
I don't blame you, Misato. I was startled too.
Weboshi
Well, I have to do my painting all over again, since it's covered in blue paint.
Fe-chan
Don't worry, Weboshi! At least you have a fair idea of what you're painting this time.
Misato
(looking at her painting, still half covered in blue paint)
...Yeah.
[Class 4-B]
Yukko slowly walks up to Nano and prods Nano with the stick, made out of taped together wood chunks.
Mio
Well, she seems unconscious by the looks of it.
Yukko
How should we get her out of the ceiling?
Mio
Maybe Hakase would know how. She did build Nano, after all.
Yukko
That's a good idea, but the question is' Where is Hakase? Has she already gone to get to her?
Mio
Should we go back down to Floor 1 and see? The gate to enter here is usually closed at this point, so she could be stuck right outside the gate. Maybe she also brought Sakamoto, since he's, y'know, older than her, making him a 'supervisor' to her.
Yukko
(in agreement)
I don't see why not.
[Cut back to Hakase and Sakamoto]
Hakase throws a small pebble at the gate, in desperate attempts to open it.
Hakase
(sits with her arms around her legs, feeling hopeless )
Well, I've done all I can do.
Sakamoto
What about just trying to get someones- No, wait, that ended out as a disaster.
Hakase
Maybe we could try going over?
Sakamoto
I don't see how you could have any more materials to make something else. There's nothing here for a distance.
Hakase
If I can make a rocket out of sticks and whatever else, this is nothing!
Sakamoto
You do you, then.
[Cut to Nakamura]
Nakamura is walking to a door that says 'AUTHORIZED PERSONNEL/STAFF ONLY', and notices Yukko and Mio running towards the elevator.
Nakamura
(concerned)
What are you doing?
Yukko is startled by Nakamura's appearance, and both Mio and Yukko stop.
Yukko
I' Um...
Mio
We're' going to uhh' the...
Nakamura
...Shouldn't you be in class?
Mio's wristwatch beeps, signalling the end of the test.
Mio
That's not good.
Yukko
Uh, yeah, we should be in class. Nice to meet you.
Mio tugs Yukko, signalling them to get back to Q-1,
[brief cut to 1-Q]
Sakurai wakes up, startled but still seems half-asleep
Sakurai
(somewhat slurred)
(yawn) Uh, the test is over, please hand in your tests,
Annaka
Mine's dust'
Sakurai
(still slurred)
You mean dusted?
Annaka
(holding dust of her test)
I meant, actual dust.
Sakurai
...Oh.
[cut back to Yukko, Mio, and Nakamura]
Yukko
(whispering)
But what about Na-
Mio
(whispering)
What about us?
Yukko
(whispering)
Alright, fine..
Yukko and Mio dash back to the elevator and the doors close.
Nakamura
...Okay then.
Nakamura enters the authorised personnel room.
[Floor 1 - Authorised Personnel Room/Security]
Nakamura
(thinking)
With this security camera footage, I should figure out where Nano is located.
Nakamura walks up to the monitor, and the flips through the footage until she eventually finds video of Nano embedded in the ceiling.
Nakamura
(thinking)
Bingo' Ah...
Finally I can figure out what's inside of her and make tons of money- or build my own tools and contraptions'
Nakamura leaves the room
[Room 4-B]
(silence for a moment)
Nano regains consciousness.
Nano
Where-
Nakamura opens the door.
Nakamura
Oh. There you are. Was wondering where you went.
Nano
N-Nakamura! What're you doing here? I'm stuck in the ceiling!
Nakamura
Well, you've been stuck in there for a longer time than usual, and the bell just rang, so I thought you were still here as a conclusion.
Nano
Well, c-can you get me out of here?
Nakamura
I'm going to do that. Anyways, by the looks of it, the best possible option to get you out of the floor and onto the floor is to have something weigh you down, like a block of concrete or a gym weight.
Nano
But' I-I don't see anything that could possibly do that!
Nakamura
Well, besides a taped together stick of wood chunks, but that's not even close to capable. I'll go get something from the gym to get you down. I already have my time off due to my unconsciousness from earlier today.
Nano
Alright, I'll just' S-stay here, I g-guess.
Nakamura exits 4-B.
[Outside Gate]
Sakamoto
Kid, we've been standing here all morning. Do you have even the slightest idea on how to get over?
Hakase
The only way now is to break it. Nothing else worked!
Sakamoto
But you said you wouldnt break the gate!
Hakase
Well, uh,
Sakamoto
Kid, if you really want to go through property damage and the res-
Hakase pulls what looks like a pen out of her pocket and it folds out into a laser gun.
Hakase
Property shmoperty. I'll just blast the door with a laser gun.
Sakamoto
You have a laser gun?! All this time wasted and NOW you say you have a laser gun?
Hakase
(ecstatically ignorant)
'Yes!
Hakase shoots the gate with the laser gun, and it explodes in half, one flying towards the school entrance: It flies through another open door and slices Fe-chan in half.
[cut to Fe-chan's dead corpse]
Misato
This again?
(sigh)
Weboshi
I wonder what Fe-chan's actual fla- flavour-
Misato
Don't.
Weboshi
Alright, fine, I'll resist my temptation.
[back to Hakase and Sakamoto]
Sakamoto
Congratulations. You killed someone' again.
Hakase
But the gate's open now.
Sakamoto
Yeah, but killing people-
Hakase
' The gate's open, and we have no time to waste!
Sakamoto
But-
Hakase
No.
Hakase walks through the gate and Sakamoto reluctantly follows.
[cut to school recovery unit]
Fe-chan
I wonder if I'm going to be accused of skipping lessons due to all this time I've been outside'
The recovery unit shakes again, and the Principal comes out of it.
Principal
(walking past)
Don't worry, you're not the only one who dies more than once.
Fe-chan
Eh?! Principal?!
Principal
I suggest you go back to class now.
Fe-chan
Bu-but' Okay...
[cut to Nano]
Nakamura opens the door, holding a magnet
Nakamura
This magnet should pull you down.
Nano
You sure?
Nakamura
If this magnet isn't strong enough, I'll have to use force to get you unstuck or pry you out
Nakamura throws the magnet, and instead of attracting Nano, it attracts every other piece of metal
Nakamura
Ugh.
Nano
Any other ideas?
Nakamura
Maybe I could try getting something so I can pull you down instead of using magnetic force.
Nano
I guess?
Nakamura
Alright, hang in there while you still can.
Nakamura closes the door.
Nano
...I wonder if I can get myself out of here.
Nano switches one of her arms into a gun and slowly aims to the ceiling she's stuck in.
Nano
(careful, arm shaking)
Nice and easy'
[cut to Yukko and Mio]
[faint bang sound, scream. Sakurai is startled by the explosion, along with the rest of the class]
Nano
(distant, in pain)
Ow, my eye!
Yukko
What was that?
Mio gets up and looks through the hole in the ceiling
Mio
Looks like Nano is trying to get down herself.
Yukko
So that would explain the bang'
Mio
Yeah, now we should worry on what happens if Nano gets herself down, because she'll fall from the ceiling.
Yukko
Wouldn't it be a ceiling?
Mio
From the floor below it.
Yukko
Then wouldn't it be a Cloor?
Mio
What type of made up name is that?
Yukko
A name I would come up with.
Mio
(to herself)
Cloor. What an odd word.
Yukko
What?
Mio
Nothing.
Yukko
(unaware)
Okie Dokie!
Mio has a concerned look on her face.
[cut to Nano]
Nano
(switches arm back to normal)
...Right, maybe a gun wasn't the best idea' What else could possibly work?
Nano's hand switches through various items until it becomes a chainsaw
Nano
A chainsaw?! Since when did I have this? ...I guess it'll work in this case.
Nano begins to slice the ceiling before Nakamura opens the door
Nakamura
Uh, What are you doing?
Nano
(running chainsaw through ceiling)
Trying to get myself out of this ceiling.
Nakamura
Well, it seems to be work-
Nano suddenly falls and screams, along with the piece of wood
Nakamura
'-ing...
[Brief cut to Hakase and Sakamoto looking through a window to see Nano falling.]
[Cut back to Nakamura]
Nakamura walks forward, looking down the hole to see Nano landing on a desk
[silence]
Sakamoto and Hakase open the door of 1-Q
Hakase
Uh...
Silence continues- until it is broken by Nano's CPU overload
Sakamoto
Right, we should definitely take her home now.
Hakase
Probably a good idea.
[silence]
Yukko
...I would say something about this, but I'm out of ideas.
Mio
Yeah' me too.
Nakamura walks through the door but realises Hakase and Sakamoto got there first
Nakamura
Uh'
Nakamura walks out and closes the door
[cut to Nakamura outside the class (hallway)]
Nakamura
(defeated, to self)
Of course, things always have to go wrong last second.
Suddenly, the bell goes, and a voice comes on
idea: cameo reference from bfdi regarding the voice and possible appearance of the speakers themselves
Principal
(from school speaker)
Due to major damage, School has been closed early for the day. Please return home.
Everyone swarms down the hallway, towards Nakamura
Nakamura
Uh oh...
[cut to Fe-chan/Weboshi/Misato, in a library]
Fe-chan
Thanks for helping me make the posters digitally guys. I don't know anything about computers, compared to you guys!
Fe-chan takes sips from her juice.
Weboshi
No need to thank us, we're your friends after all!
Misato
You're talking as if we're computer experts, Fe-chan, you should really learn a bit more, this is basic stuff.
Fe-chan
I know, I know. I should listen more at class.
Fe-chan takes more sips.
Weboshi
You'll have to pay me for that juice later, you know that?
Fe-chan
(spits out drink)
Eh?! I thought it was free!
Weboshi
I was planning to buy my own juice with that money, so you'll have to pay me back tomorrow.
Fe-chan
(sad)
Awwwww...
Misato
I still can't believe you took off the paper simply because some guy made fun of it.
Fe-chan
What?
Misato
What I'm saying is that you should be more tolerant, I'm pretty sure everyone else would've gone to your house party without caring about the poster's quality.
Fe-chan
Who cares? Now the poster will look cooler! Is that not better?
Misato
I, I guess? What you did was still pretty childish though. I would've kept the poster on the corkboard.
Fe-chan
Eh?! It's not childish!
Misato
It kinda is though...
Fe-chan
Weboshi! Help me on this!
Weboshi
Well' Cancelling a party simply because of someone's opinion is kinda childish.
Fe-chan
Eh?!
Misato
See? I told you so.
Fe-chan
(angry)
We-well, you-you'd be shooting missiles left and right if, if your crush, Sa-Sasahara trashed your painting! Wouldn't that be more chi-childish?!
[Cut to the librarian reading a book]
Sound of multiple explosions
Camera shakes violently and the librarian's glasses and book fall off
Librarian
Wha-what?!
Shinonome Laboratory, Night
Sakamoto and Hakase are inspecting the damage on Nano.
Sakamoto
I'm amazed at how Nano hasn't just broken in half, I mean, she blasted herself through several floors, apparently shot her own eye, falling back down and then finally landing on a table, breaking it in the process.
Hakase
(applying square eyepatch to Nano's eye)
I'm just thankful she's still alive. If she had any more damage, she would've just broke into pieces!!
Sakamoto
Well, maybe next time you should take better care for her too, and not just yourself, kid.
Hakase
(grabbing a strawberry juice carton)
But Nano didn't listen to me when I tried saying.
Sakamoto
You're overwhelming her, since she also has school and her own persona to worry about, too. That also explains why she passed out when we found her on a broken desk.
Hakase
(sounding defeated)
Let's just hope this doesn't happen again'
Sakamoto
Yeah. Also, make sure she's ready for tomorrow this time.
Hakase sighs and opens the carton
A small droplet lands on the motherboard, without anyone noticing
[focus on the droplet]
Hakase
Everything appears to relatively fine, only a few parts have taken damage which is just-
Sakamoto
How do you keep breaking physics like this?
Hakase
...I dunno, ask physics.
Hakase removes the damaged parts and opens a box full of working parts
Sakamoto
I never thought we'd have to use that box.
Hakase
Me neither, but it seems like some important parts are broke as well, and need replacement.
Sakamoto
As I said, just make sure she's ready for tomorrow.
Hakase
Alright.
[later timecard]
Hakase
And that should fix it.
Hakase looks at Sakamoto, who's sleeping.
Hakase
(tired)
*yawn* I should go sleep as well'
Hakase closes the front of Nano, and carefully brings her over to her bed, trying not to cause any further damage.
Hakase then turns off the light, gets to her own bed, and falls asleep.
[episode end credits]
[after credits scene, shinonome lab]
Hakase, Sakamoto, and Nano are sleeping. Nano electrocutes for a split second, but no one is woke up by it.


class: center, middle ### W4995 Applied Machine Learning # Linear models for Regression 02/10/20 Andreas C. Müller ??? So today we'll talk about linear models for regression. You're probably familiar with some of the basics, and that's where we'll start. But we'll also look into how to tune these models, what the trade-offs are, and some more complex techniques. But first we'll start with some more preprocessing that we didn't get to last week, in particular dealing with missing values, which you'll need for the homework. FIXME show how to get feature names out of complex pipelines FIXME remove isocontour slides FIXME sync with workshop FIXME imputation for categorical data FIXME notebook needs sorting FIXME better motivation for log FIXME add missing value indicator FIXME better explanations for imputation, in particular iterative! too short, this fixes it FIXME L1 explanation unclear FIXME adding features: use pipeline --- class: center, middle # Sidebar: Dealing with Missing Values ??? This is for the homework. I didn't realize there were missing values in the continuous data. --- class: spacious # Dealing with missing values - Missing values can be encoded in many ways - Numpy has no standard format for it (often np.NaN) - pandas does - Sometimes: 999, ???, ?, np.inf, “N/A”, “Unknown“ … - Not discussing “missing output” - that’s semi-supervised learning. - Often missingness is informative (Use `MissingIndicator`) ??? So first we're going to talk about imputation, which means basically dealing with missing values. Before that, we're going to talk about different methods to deal with missing values. The first thing about missing values is that you need to figure out whether your dataset has them and how they encode it. If you're on Python, numpy has no standard way to represent missing values while Pandas does and it's usually nan. But the problem is usually more in the data source. So depending on where your data comes from, missing values might be encoded as anything. They might be differently encoded for different parts of the data set. So if you see some question marks somewhere, it doesn't mean that all missing values are encoded as question marks. There might be different reasons why data is missing. If you look into like theoretical analysis missingness, often there you can see something that's called missing at random or missing completely at random, meaning that data was retracted randomly from the dataset. That's not usually what happens. Usually, if the data is missing, it's missing because something when differently in a process, like someone didn't fill out the form correctly, or someone didn't reply in a survey. And very often the fact that someone didn't reply or that something was not measured is actually informative. Whenever you have missing values, it's often a good idea to keep the information about whether the value was missing or not. If a feature was missing, while we're going to do imputation and we're going to try to fill in the missing values. It's often really useful information that's something was missing and you should record the fact and represent it in the dataset somehow. We're only going to talk about missing input today. You can also have missing values in the outputs that are usually called semi-supervised learning where you have the true target or the true class only for some data points, but not for all of them. So there's the first method which is very obvious. Let's say your data looks like this. All my illustrations today we'll be adding randomness to the iris data set. FIXME: Better digram for feature selection. What are the hypothesises for the tests --- .center[ ![:scale 80%](images/row_nan_col_nan.png) ] ??? So if my dataset looks like something on the left-hand side here, then you can see that there are only missing values in the first feature and it's mostly missing. One of the ways to deal with this is just completely dropped the first feature, and that might be a reasonable thing to do. If there are so few values here that you don't think there's any information in this just drop it I always like to compare everything to a baseline approach. So your baseline approach should be if there's only missing values in some of the columns, just drop these columns, see what happens. You can always iterate and improve on that but that should be the baseline. A little bit trickier situation is that there might be some missing values only for a few rows, the rows are data points. You can kick out these data points and train the model on the rest and that might be fine. There's a bit of a problem with this though, that if you want to make predictions on new data and the data that arrives has missing values you'll not be able to make predictions because you don't have a way to deal with missing values. If you're guaranteed that any new test point that comes in will not have missing values then doing this might make sense. Another issue with dropping the rows with missing values is that if this was related to the actual outcome then it might be that you biased how well you think you're doing. Maybe all the hard data points are the ones that have missing values. And so by excluding them from your training data, you're also excluding them from the validation. So it means you think you're doing very well because you discarded all the hard data points. Discarding data points is a little bit trickier and it depends on your situation. --- .center[ ![:scale 100%](images/imputation-schema.png) ] ??? The other solution is to impute them. So the idea is that you have your training data set, you build some model of the training data set, and then you fill in the missing values using the information from the other rows and columns. And you built a model for this and then you can also apply the same imputation model on the test data set if you want to make predictions. Question is what if it has all missing values? Then you have no choice but drop that. If in the dataset that happens, you need to figure out what you are going to do. Like, if you have a production system, and something comes in with all missing values, you need to decide what you're going to do. But you probably cannot use this data point for training model. You could like use the outputs and train to find the mean outcome of all the values that are missing. --- class: spacious # Imputation Methods - Mean / Median - kNN - Regression models - Matrix factorization (not in this lecture) ??? So let's talk about the general methods for data imputation. So these are the ones that we are going to talk through. The easiest one is me doing a constant value per column. Imputing the mean or the medium of the column that we're trying to compute. kNN means taking the average of KNearest Neighbors. Regression means I build a regression model from some of the features trying to rip the missing future. And finally, elaborate probabilistic models. They try to build a probabilistic model of the dataset and complete the missing values based on this probabilistic model --- # Baseline: Dropping Columns .smaller[ ```python from sklearn.linear_model import LogisticRegressionCV from sklearn.model_selection import train_test_split, cross_val_score X_train, X_test, y_train, y_test = train_test_split(X_, y, stratify=y) nan_columns = np.any(np.isnan(X_train), axis=0) X_drop_columns = X_train[:, ~nan_columns] scores = cross_val_score(LogisticRegressionCV(v=5), X_drop_columns, y_train, cv=10) np.mean(scores) ``` 0.772 ] ??? Here’s my baseline, which is dropping the columns. I used iris dataset where I had to put some missing values into the second and third column. Here, x_score has like some missing values, I split into test and training test data set and then I drop all columns that have missing values and then I can train logistic regression, which is sort of the model I'm going to use to tell me how well does this imputation work for classification. For my baseline, using a logistic regression model and 10 fold baseline, I get 77.2% accuracy. --- # Mean and Median .center[ ![:scale 100%](images/imputation-median-schema.png) ] ??? The simplest imputation is mean or medium. Unfortunately, only one is implemented in scikit-learn right now. If this is our dataset, the imputation will look like this. For a column, each missing value is replaced by the mean of the column. The imputer is the only transformer in scikit-learn that can handle missing data. Using the imputer you can specify the strategy, mean or median or constant and then you can call the method transform and that imputes missing values. --- .center[ ![:scale 90%](images/median_imputation.png) ] ??? Here is a graphical illustration of what does this. So the iris dataset is four-dimensional, and I'm plotting it in two dimensions in which I added missing values. So, there are two other dimensions which had no missing values, which you can't see. The original data set is basically blue points here, orange points here, green points here, and it's relatively easy to separate. But here you can see that the green points they have a lot of missingness, so they were replaced by the mean of the whole dataset. So there are two things about this. One, I kind of lost the class information a lot. Two, the data is now in places where there was no data before, which is not so great. You could do smart things like doing the mean per class, but that's actually not something that I've seen a lot. --- ```python from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScalar nan_columns = np.any(np.isnan(X_train), axis = 0) X_drop_columns = X_train[:,~nan_columns] logreg = make_pipeline(StandardScalar(), LogisticRegression()) scores = cross_val_score(logreg, X_drop_columns, y_train, cv=10) np.mean(scores) ``` 0.794 ```python mean_pipe = make_pipeline(SimpleImputer(strategy='median'), StandardScalar(), LogisticRegression()) scores = cross_val_score(mean_pipe, X_train, y_train, cv=10) np.mean(scores) ``` 0.729 ??? Here's the comparison of dropping the columns versus doing the mean amputation. We actually see that the mean imputation is worse. In general, if you have very few missing values, it might actually not be so bad and putting in the mean might work. Here I basically designed the dataset so that mean imputation fails. This is not very realistic. --- class: spacious # KNN Imputation - Find k nearest neighbors that have non-missing values. - Fill in all missing values using the average of the neighbors. ??? FIXME illustration In terms of complexity is using Nearest Neighbors. So the idea is for each data point, we find the kNearest neighbors and then fill in the missing values as the average of the features of these neighbors. The difficulty here is measuring distances if there are missing values. If there are missing values, I can't just compute Euclidean distances. --- .smaller[ ```python knn_pipe = make_pipeline(KNNImputer(), StandardScaler(), LogisticRegression()) np.mean(scores) ``` ``` 0.849 ``` ] .center[ ![:scale 60%](images/knn_imputation.png) ] ??? --- # Model-Driven Imputation - Train regression model for missing values - Iterate: retrain after filling in - IterativeImputer in next sklearn release .smaller[ ```python rf_imp = IterativeImputer(predictor=RandomForestRegressor()) rf_pipe = make_pipeline(rf_imp, StandardScaler(), LogisticRegression()) scores = cross_val_score(rf_pipe, X_rf_imp, y_train, cv=10) np.mean(scores) ``` ``` 0.845 ``` ] ??? The next step up in complexity is using an arbitrary regression model for imputation. I mean, arguably, the kNN is also a regression model, but there's like some intricacies that make it a little bit different. So the idea with using a regression model is you do the first pass and impute data using the mean. And then you try to predict the missing features using a regression model trained on the non-missing features and then you iterate this until stuff doesn't change anymore. You can use any model you like, and this is very flexible, and it can be fast if you have a fast model. --- class: spacious # Comparision of Imputation Methods .center[ ![:scale 100%](images/med_knn_rf_comparison.png) ] ??? FIXME better illustration/ graph Here's a comparison on the iris dataset again. This is a very artificial example and is only meant to illustrate the ideas. In practice, using the mean or median is actually quite decent. --- class: center, middle # Linear Models --- class:center # Linear Models for Regression ![:scale 50%](images/linear_regression_1d.png) $$\hat{y} = w^T \mathbf{x} + b = \sum_{i=1}^p w_i x_i +b$$ ??? Predictions in all linear models for regression are of the form shown here: It's an inner product of the features with some coefficient or weight vector w, and some bias or intercept b. In other words, the output is a weighted sum of the inputs, possibly with a shift. here i runs over the features and x_i is one feature of the data point x. These models are called linear models because they are linear in the parameters w. The way I wrote it down here they are also linear in the features x_i. However, you can replace the features by any non-linear function of the inputs, and it'll still be a linear model. There are many differnt linear models for regression, and they all share this formula for making predictions. The difference between them is in how they find w and b based on the training data. --- # Ordinary Least Squares $$\hat{y} = w^T \mathbf{x} + b = \sum_{i=1}^p w_i x_i +b $$ `$$\min_{w \in \mathbb{R}^p, b\in\mathbb{R}} \sum_{i=1}^n (w^T\mathbf{x}_i + b - y_i)^2$$` Unique solution if $\mathbf{X} = (\mathbf{x}_1, ... \mathbf{x}_n)^T$ has full column rank. ??? The most straight-forward solution that goes back to Gauss is ordinary least squares. In ordinary least squares, find w and b such that the predictions on the training set are as accurate as possible according the the squared error. That intuitively makes sense: we want the predictions to be good on the training set. If there is more samples than features (and the samples span the whole feature space), then there is a unique solution. The problem is what's called a least squares problem, which is particularly easy to optimize and get the unique solution to. However, if there are more features than samples, there are usually many perfect solutions that lead to 0 error on the training set. Then it's not clear which solution to pick. Even if there are more samples than features, if there are strong correlations among features the results might be unstable, and we'll see some examples of that soon. Before we look at examples, I want to introduce a popular alternative. --- # Ridge Regression `$$ \min_{w \in \mathbb{R}^p, b\in\mathbb{R}} \sum_{i=1}^n (w^T\mathbf{x}_i + b - y_i)^2 + \alpha ||w||^2 $$` Always has a unique solution. Tuning parameter alpha. ??? In Ridge regression we add another term to the optimization problem. Not only do we want to fit the training data well, we also want w to have a small squared l2 norm or squared euclidean norm. The idea here is that we're decreasing the "slope" along each of the feature by pushing the coefficients towards zero. This constraings the model to be more simple. So there are two terms in this optimization problem, which is also called the objective function of the model: the data fitting term here that wants to be close to the training data according to the squared norm, and the prenalty or regularization term here that wants w to have small norm, and that doesn't depend on the data. Usually these two goals are somewhat opposing. If we made w zero, the second term would be zero, but the predictions would be bad. So we need to trade off between these two. The trade off is problem specific and is specified by the user. If we set alpha to zero, we get linear regression, if we set alpha to infinity we get a constant model. Obviously usually we want something in between. This is a very typical example of a general principle in machine learning, called regularized empirical risk minimization. --- # (regularized) Empirical Risk Minimization `$$ min_{f \in F} \sum_{i=1}^n L(f(\mathbf{x}_i), y_i) + \alpha R(f) $$` ??? FIXME pointers data fitting / regularization! Many models in machine learning, like linear models, SVMs and neural networks follow the general framework of empirical risk minimization, which you can see here. We formulate the machine learning problem as an optimization problem over a family of functions. In our case that was the family of linear functions parametrized by w and b. The minimization problem consists of two parts, the data fitting part and the model complexity part. The data fitting part says that the predictions mad eby our functions should be accurate according to some loss L. For our regression problems that was the squared loss. The model complexity part says that we prefer simple models and penalizes complicated f. Most machine learning algorithms can be cast into this, with a particular choice of family of functions f, loss function L and regularizer R. And most of machine learning theory is build around this framework. People proof for differnt choices of F and L and R that if you minimize this, you'll be able to generalize well. And that makes intuitive sense. To do well on the test set, we definitely want to do reasonably well on the training set. We don't expect that we can do better on a test set than the training set. But we also want to minimize the performance difference between training and test set. If we restrict our model to be simple via the regularizer R, we have better chances of the model generalizing. --- # Reminder on model complexity ![:scale 80%](images/overfitting_underfitting_cartoon_full.png) ??? I hope this sounds familiar from what we talked about last time. This is a particular way of dealing with overfitting and underfitting. For this framework in general, or for ridge regression in particular, trading off the data fitting and the regularization changes the model complexity. If we set alpha high we restrict the model, and we will be on the left side of the graph. If we make alpha small, we allow the model to fit the data more, and we're on the right side of the graph. --- # Ames Housing Dataset ![:scale 80%](images/ames_housing_scatter.png) .tiny[ ```python print(X.shape) print(y.shape) ``` ``` (2195, 79) (2195,) ``` ] ??? Ok after all this pretty abstract talk, let's make this concrete. Let's do some regression on the boston housing dataset. After the last homework you're hopefully familiar with it. The idea is to predict prices of property in the boston area in different neighborhoods. This is a dataset from the 70s I think, so everything is pretty cheap. Most of the features you can see are continuous, with the exception of the charlston river variable which says whether the neighborhood is on the river. Keep in mind that this data lives in a 13 dimensional space and these univariate plots only look at 13 different projections of the data, and can't capture any of the interactions. But still we can see that the price clearly depends on some of these variables. It's also pretty clear that the dependency is non-linear for some of the variables. We'll still start with a linear model, because its a very simple class of models, and I'd always star approaching any model from the simplest baseline. In this case it's linear regression. We're having 506 samples and 13 features. We have much more samples than features. Linear regression should work just fine. Also it's a tiny dataset, so basically anything we'll try will run instantaneously, which is also good to keep in mind. Another thing that you can see in this graph is that the features have very different scales. Here's a box plot that shows that even more clearly. --- ![:scale 100%](images/ames_scaling.png) ??? That's something that will trip up the distance based models models we talked about last time, as well as the linear models we're talking about today. For the penalized models the different scales mean that different features are penalized differently, which you usually want to avoid. Usually there is no particular semantics attached to the fact that one feature has different magnitutes than another. We could measure something in inches instead of miles, and that would change the outcome of the model. That's certainly not something we want. A good idea is to scale the data to get rid of this effect. We'll talk about that and other preprocessing methods in-depth on Wednesday next week. Today I'm mostly gonna ignore this. But let's get started with Linear Regression --- # Coefficient of determination R^2 `$$ R^2(y, \hat{y}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \hat{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2} $$` `$$ \bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i$$` Can be negative for biased estimators - or the test set! ??? The scores are R squared or coefficient of determination. This is basically a score that's usually between zero and one, where one means perfect prediction or perfect correlation and zero means a random prediction. What it does is it computes the mean of the targets over the data you're evaluating it on and then it looks at the distance between the prediction and the ground truth relative to the mean. If it's negative, it means you do a worse job at predicting and just predicting the mean. It can happen if your model was really bad and bias. The other reason is if you use a test set. This is guaranteed to be positive on the data it was fit on with an unbiased linear model, which will nearly never apply to what we’re doing. The R^2 can be misleading if there's outliers in the training data and some consider it a bad metric. Max Kuhn, author of APM thinks it's a bad metric. It's not clear to me that MSE is much better in general, though. Reducing anything to a single number is tricky. --- class: smaller # Preprocessing ```python cat_preprocessing = make_pipeline( SimpleImputer(strategy='constant', fill_value='NA'), OneHotEncoder(handle_unknown='ignore')) cont_preprocessing = make_pipeline( SimpleImputer(), StandardScaler()) preprocess = make_column_transformer( (cat_preprocessing, make_column_selector(dtype_include='object')), remainder=cont_preprocessing) X_train, X_test, y_train, y_test = train_test_split( X, y, random_state=0) cross_val_score( make_pipeline(preprocess, LinearRegression()), X_train, y_train, cv=5) ``` ``` array([0.928, 0.927, 0.932, 0.898, 0.884]) ``` --- class: smaller # Skewed targets .left-column[ ```python y_train.hist(bins='auto') ``` ![:scale 100%](images/ames_housing_price_hist.png) ] .right-column[ ```python np.log(y_train).hist(bins='auto') ``` ![:scale 100%](images/ames_housing_price_hist_log.png) ] --- class: middle, spacious ```python cross_val_score(make_pipeline(preprocess, LinearRegression()), X_train, y_train, cv=5) ``` ``` array([0.928, 0.927, 0.932, 0.898, 0.884]) ``` ```python from sklearn.compose import TransformedTargetRegressor log_regressor = TransformedTargetRegressor( LinearRegression(), func=np.log, inverse_func=np.exp) cross_val_score(make_pipeline(preprocess, log_regressor), X_train, y_train, cv=5) ``` ``` array([0.95 , 0.943, 0.941, 0.913, 0.922]) ``` --- # Linear Regression vs Ridge ```python log_regressor = TransformedTargetRegressor( LinearRegression(), func=np.log, inverse_func=np.exp) cross_val_score(make_pipeline(preprocess, log_regressor), X_train, y_train, cv=5) ``` ``` array([0.95 , 0.943, 0.941, 0.913, 0.922]) ``` ```python log_ridge = TransformedTargetRegressor( Ridge(), func=np.log, inverse_func=np.exp) cross_val_score(make_pipeline(preprocess, log_ridge), X_train, y_train, cv=5) ``` ``` array([0.948, 0.95 , 0.941, 0.915, 0.931]) ``` ??? Let’s look at two simple models. Linear regression and Ridge regression. What I've done is I’ve split the data into training and test set and used 10 fold cross-validation to evaluate them. Here I use cross_val_score together with the model, the training data, training labels, and 10 fold cross-validation. This will return 10 scores and I'm going to compute the mean of them. I'm doing this for both linear regression and Ridge regression. Here is ridge regression uses a default value of alpha of 1. Here these two scores are quite similar. --- .smallest[ ```python pipe = Pipeline([('preprocessing', preprocess), ('ridge', log_ridge)]) param_grid = {'ridge__regressor__alpha': np.logspace(-3, 3, 13)} grid = GridSearchCV(pipe, param_grid, cv=RepeatedKFold(10, 5), return_train_score=True) ``` ``` {'ridge__regressor__alpha': array([ 0.001, 0.003, 0.01, 0.032, 0.1, 0.316, 1., 3.162, 10., 31.623, 100., 316.228, 1000.])} ``` ```python grid.fit(X_train, y_train) ``` ] .center[ ![:scale 50%](images/ridge_alpha_search.png) ] ??? Coming back to the ridge regression we used the standard alpha of one which is a reasonable default, but by no means, this is guaranteed to make sense in this particular problem. Here I’ve done the grid search. As we talked about on Monday, I defined a parameter grid where the key is the parameter I want to search (alpha in Ridge) and the parameters I want to try. For regularization parameters, like alpha, it’s usually good to do them on the logarithmic grid. I do a relatively fine grid here with 13 different points mostly because I wanted to have a nice plot. In reality, I would use a three or six or something like that. I’ve instantiated GridSearchCV with Ridge(), the parameter grid and do 10 fold cross-validation and then I called grid.fit. I’ve reported the mean training accuracy and mean test accuracy over 10 cross-validation folds for each of the parameter settings. Okay, you can see a couple things here. A) There's a lot of uncertainty B) The training set is always better than the test set. C) The most important thing that you can see here is that regularization didn't help. Making alpha as small as possible is the best. What I'm going to do next is I'm going to modify this dataset a little bit so that we can see the effect of the regularization. I’m going to modifying by using a polynomial expansion, again we're going to talk about a little bit more on Wednesday. --- .padding-top[ .left-column[ ![:scale 100%](images/ridge_alpha_search.png) ] .right-column[ ![:scale 100%](images/ridge_alpha_search_cv_runs.png) ] ] --- # Triazine Dataset ```python triazines = fetch_openml('triazines') triazines.data.shape ``` ``` (186, 60) ``` ```python pd.Series(triazines.target).hist() ``` .center[ ![:scale 40%](images/triazine_bar.png) ] --- ```python X_train, X_test, y_train, y_test = train_test_split(triazines.data, triazines.target, random_state=0) cross_val_score(LinearRegression(), X_train, y_train, cv=5) ``` ``` array([-4.749e+24, -9.224e+24, -7.317e+23, -2.318e+23, -2.733e+22]) ``` ```python cross_val_score(Ridge(), X_train, y_train, cv=5) ``` ``` array([0.263, 0.455, 0.024, 0.23 , 0.036]) ``` --- ```python param_grid = {'alpha': np.logspace(-3, 3, 13)} grid = GridSearchCV(Ridge(), param_grid, cv=RepeatedKFold(10, 5), return_train_score=True) grid.fit(X_train, y_train) ``` .center[ ![:scale 40%](images/ridge_alpha_triazine.png) ] --- # Plotting coefficient values (LR) ```python lr = LinearRegression().fit(X_train, y_train) plt.scatter(range(X_train.shape[1]), lr.coef_, c=np.sign(lr.coef_), cmap='bwr_r') ``` .center[ ![:scale 55%](images/lr_coefficients_large.png) ] ??? In the previous slide, linear regression did nearly as well as the Ridge Regression with the default parameters but let's look at the coefficients of a linear model. These are the coefficients of the linear model trained on the polynomial futures. I plot them adding a color to represent positive and negative. The magnitude here is 1 and then 13 zeros. We have two features, one is like, probably more than trillions times two and then there's another one that's very negative. What I conclude from this is, these 2 features are very highly correlated. The model makes both of them really, really big and then they cancel each other out. This is not a very nice model, because it relates to American stability and also it tells me that these 2 features are like extremely important, but they might not be important at all. Like the other features might be more important, but they're nullified by this cancellation effect. They (0.2 and -0.8) need to cancel each other out because the predictions are reasonable and all the houses only cost like $70,000. --- # Ridge Coefficients ```python ridge = grid.best_estimator_ plt.scatter(range(X_train.shape[1]), ridge.coef_, c=np.sign(ridge.coef_), cmap="bwr_r") ``` .center[ ![:scale 55%](images/ridge_coefficients.png) ] ??? Let's look at the Ridge model. This is the best estimator, which is the model that was found in a grid search with the best parameter settings. This looks much more reasonable. This feature, which was a very negative one, still is very negative. But now this is actually three and minus three. So this is a much more reasonable range. We can also look at the features and the effect of different values of alpha. Here is a Ridge with 3 different values of alpha. In the previous random seat, alpha equal to 14 was the best now and now we have it equal to 30 something. The green one is more or less the best setting and then there's a smaller and a bigger one. You can see that basically what alpha does is, on average, it pushes all the coefficients toward zero. So here you can see this coefficient shrank going from 1 to 14 going to 0 and the same here. So basically, they all push the different features towards 0. If you look at this long enough, you can see things that are interesting, the first one with alpha equal to one it's positive, and with alpha equal to 100, it's negative. That means depending on how much you regularize the direction of effect goes in opposite directions, what that tells me is don't interpret your models too much because clearly, either it has a positive or negative effect, it can't have both. --- ```python ridge100 = Ridge(alpha=100).fit(X_train, y_train) ridge1 = Ridge(alpha=.1).fit(X_train, y_train) plt.figure(figsize=(8, 4)) plt.plot(ridge1.coef_, 'o', label="alpha=.1") plt.plot(ridge.coef_, 'o', label=f"alpha={ridge.alpha:.2f}") plt.plot(ridge100.coef_, 'o', label="alpha=100") plt.legend() ``` .center[ ![:scale 60%](images/ridge_coefficients_alpha.png) ] ??? One other way to visualize the coefficient is to look at the coefficient path or regularization path. On the x-axis is the alpha, and on the y-axis, the coefficient magnitude. Basically, I looped over all of the different alphas and you can see how they shrink towards zero to increase alpha. There are some very big coefficients that go to zero very quickly and some coefficients here that stay the same for a long time. --- # Learning Curves ![:scale 90%](images/ridge_learning_curve.png) ??? The thing I want to illustrate is if you have less data, regularization helps more. You can see the effects here between the orange and the red are very drastic. The green one probably has too much regularization. If you have very little data, even regularizing way too much is okay, but there's sort of a sweet spot here which is the orange one and the orange one does best. The more data you add, the less important regularization becomes. If you take the blue curve at the beginning it’s much higher than the red curve, but then, in the end, they overlap. --- # Lasso Regression `$$ \min_{w \in \mathbb{R}^p, b\in\mathbb{R}} \sum_{i=1}^n (w^T\mathbf{x}_i + b - y_i)^2 + \alpha ||w||_1 $$` - Shrinks w towards zero like Ridge - Sets some w exactly to zero - automatic feature selection! ??? Lasso Regression looks very similar to Ridge Regression. The only thing that is changed is we use the L1 norm instead of the L2 norm. L2 norm is the sum of squares, the L1 norm is the sum of the absolute values. So again, we are shrinking w towards 0, but we're shrinking it in a different way. The L2 norm penalizes very large coefficients more, the L1 norm penalizes all coefficients equally. What this does in practice is its sets some entries of W to exactly 0. It does automatic feature selection if the coefficient of zero means it doesn't influence the prediction and so you can just drop it out of the model. This model does features selection together with prediction. Ideally what you would want is, let's say you want a model that does features selections. The goal is to make our model automatically select the features that are good. What you would want to penalize the number of features that it uses, that would be L0 norm. --- # Understanding L1 and L2 Penalties .wide-left-column[ ![:scale 100%](images/l2_l1_l0.png) ] .narrow-right-column[ .smaller[ `$$ \ell_2(w) = \sqrt{\sum_i w_i ^ 2}$$` `$$ \ell_1(w) = \sum_i |w_i|$$` `$$ \ell_0(w) = \sum_i 1_{w_i != 0}$$` ]] ??? The L0 norm penalizes the number of features that are not zero. The L1 norm penalizes the absolute values. L2 norm penalizes the squared norm. The problem with L0 norm is it's one everywhere except at zero. And that's not differentiable, it's not even continuous, so this is very hard to optimize. Basically, we just gave up on this and we use the next best thing we can do, which is the L1 norm. The L1 norm is not differentiable, there’s like this kink in zero, but we can still optimize it pretty well. The L2 norm penalizes large values much more than small values. --- # Understanding L1 and L2 Penalties .wide-left-column[ ![:scale 100%](images/l1_kink.png) ] .narrow-right-column[ .padding-top[ .smaller[ `$ f(x) = (2 x - 1)^2 $` `$ f(x) + L2 = (2 x - 1)^2 + \alpha x^2$` `$ f(x) + L1= (2 x - 1)^2 + \alpha |x|$` ]]] ??? Let's say I want to do a one-dimensional linear regression problem. The problem to solve is something of this form. I want my coefficients to be such that, let's say x, the one coefficient I'm looking for and this is sort of the squared loss function that I have. F is the daycare fitting term here, and this is would be like some quadratic. The loss function is in blue, its quadratic. The loss function plus our L2 regularization, the ridge is in orange. The data fitting plus L1 regularization is in green. The blue is a squared quadratic function, the orange is a quadratic function that sort of moved a little bit toward zero and the green one also moved towards zero but there's a kink here. Basically, this is just sort of the kink at the absolute value of zero and that makes it much more likely that the optimum will actually be at zero. So here the optimum for orange is somewhere here, so it was pushed towards zero but not exactly zero. --- class: center # Understanding L1 and L2 Penalties ![:scale 70%](images/l1l2ball.png) ??? A different way to visualize this is in two dimensions. Let's say we have 2 coefficients. So one way to look at these norms, let’s say we want to minimize the norm but say we want to fix the norm, like more or less equivalent, these are the L1 and L2 balls. So this is all the 2D vectors that have Euclidean norm 1 or the ball and all the vectors that have L1 norm is the diamond. Let's say we restrict ourselves to a solution that lies on this ball. --- class: center # Understanding L1 and L2 Penalties ![:scale 70%](images/l1l2ball_intersect.png) ??? And then we have a quadratic function like this, this is sort of the data fitting term again and so now if we want a solution with L2 norm equal to zero, we get the solution here, the intersection of the height lines of the data fitting term and constraint to L2 norm. But for this diamond, we're going to hit very likely one of the corners just because of the geometry of the space, we're likely to either of these corners. The corner means one of the coefficients is exactly zero, and the other one is one. So this is another sort of geometric realization of trying to understand why does this lead to exact zeros hop. --- # Grid-Search for Lasso ```python param_grid = {'alpha': np.logspace(-5, 0, 10)} grid = GridSearchCV(Lasso(normalize=True), param_grid, cv=10) grid.fit(X_train, y_train) print(grid.best_params_) print(grid.best_score_) ``` ``` {'alpha': 0.0016} 0.163 ``` ??? Now we can do Grid Search again, the default parameters usually don't work very well for Lasso. I use alpha on the logarithmic grid. I fitted and then I get the best score. --- ![:scale 90%](images/lasso_alpha_triazine.png) ??? Looking at the training test set performance, you can see that if you increase the regularization, this model gets really bad. The ridge regression didn't go this badly. If you set the realization to one, all coefficients become zero. Other than that there's reasonable performance, which is about as good as the ridge performance. --- .center[ ![:scale 60%](images/lasso_coefficients.png) ] ```python print(X_train.shape) np.sum(lasso.coef_ != 0) ``` ``` (139, 60) 13 ``` ??? These are the coefficients of the model. Out of the 104 features, it only selected 64 that are non-zero, the other ones are exactly zero. You can see this visualized here. The white ones are exactly zero and the other ones on non-zero. If I wanted I could prune the future space a lot and that makes the model possibly more interpretable. There's a slight caveat here if two of the features that are very correlated, Lasso will pick one of them at random and make the other one zero. Just because something's zero doesn't mean it's not important. It means you can drop it out of this model. If you have two features that are identical, one of them will be zero and one of them will be not zero and it's going to be randomly selected. That makes interpretation a little bit harder. --- # Elastic Net - Combines benefits of Ridge and Lasso - two parameters to tune. `$$\min_{w \in \mathbb{R}^p, b\in\mathbb{R}} \sum_{i=1}^n ||w^T\mathbf{x}_i + b - y_i||^2 + \alpha_1 ||w||_1 + \alpha_2 ||w||^2_2 $$` ??? You can also combine them. This actually what works best in practice. This is what's called the Elastic Net. Elastic Net tries to combine both of these penalizations together. You now have both terms, you have the L1 norm and the L2 norm and you have different values of alpha. Basically, this generalizes both. If you choose both these are alpha, it can become ridge and it can become Lasso, it can become any anything in between. Generally, ridge helps generalization. So it's a good idea to have the ridge penalty in there, but also maybe if there are some features that are really not useful, the L1 penalty helps makes the same exactly zero. --- class: center # Comparing unit balls ![:scale 70%](images/l1l2_elasticnet.png) ??? Looking at the unit balls, if you zoom in here you would see the Elastic Net ball is kind of round, but also has a corner. Important things are the corners because that allows you to hit the exact zeros. --- # Parametrization in scikit-learn `$$\min_{w \in \mathbb{R}^p, b\in\mathbb{R}} \sum_{i=1}^n (w^T\mathbf{x}_i + b - y_i)^2 + \alpha \eta ||w||_1 + \alpha (1 - \eta) ||w||^2_2 $$` Where $\eta$ is the relative amount of l1 penalty (`l1_ratio` in the code). ??? The way this parameterize in scikit-learn is slightly different. In scikit-learn, you have a parameter alpha, which is the amount of regularization and then there's a parameter called l1_ratio, that says how much of the penalty should be L1 and L2. If you make this one, you have Lasso, if you make it zero, you have a Ridge. Don't use Lasso or Ridge and set alpha zero, because the solver will not handle it well. If you actually want alpha equal to zero, use linear regression. Now we have more parameters to tune, but we just have a more general model. This actually works pretty well often. --- # Grid-searching ElasticNet ```python from sklearn.linear_model import ElasticNet param_grid = {'alpha': np.logspace(-4, -1, 10), 'l1_ratio': [0.01, .1, .5, .8, .9, .95, .98, 1]} grid = GridSearchCV(ElasticNet(), param_grid, cv=10) grid.fit(X_train, y_train) print(grid.best_params_) print(grid.best_score_) ``` ``` {'alpha': 0.001, 'l1_ratio': 0.9} 0.100 ``` ```python (grid.best_estimator_.coef_!= 0).sum() ``` ``` 10 ``` ??? Here is me doing a grid search. If you have two parameters for the grid search it will do all possible combinations. Here I do a logarithmic space for alpha and for the l1_ratio, I use something that’s very close to zero and something that's very close to one and some stuff in between. If you want to analyze the output of a 2D grid search a little bit harder we can’t do the nice curve anymore. --- class: smaller # Analyzing grid-search results ```python import pandas as pd res = pd.pivot_table(pd.DataFrame(grid.cv_results_), values='mean_test_score', index='param_alpha', columns='param_l1_ratio') ``` .center[ ![:scale 60%](images/elasticnet_search.png) ] ??? The way that I like to do it is, here's the grip.cv results. And I put it in a data frame and then I'll make a pivot table where the values are test score, the index is one parameter and the columns are the other parameter. It allows me to visualize the grid search nicely. This is alpha and this is l1_ratio and you can see that if the l1_ratio is pretty high, there are some pretty good results, if you set the alpha accordingly. So here's like the diagonal of pretty good things. This is the model that did best. There's a slight caveat here that right now I did this with cross-validation and so this is the cross-validation accuracy. Last time I said, this is not really a good measure of generalization performance. So here, I searched way more parameters, I tried like 5 or 10 times as many models. So it's likely that by chance, I'll get better results. I didn't do this here in particular, because the data set is small and very noisy but in practice, if you want to compare models, you should evaluate it on a test set and see which of the models actually are better on the test. One more thing, why this is helpful is if the best value is on the edge of this graph that means my ranges were too small. Question is why we're using r square instead of the squared loss, one of the answers is that's the default in scikit-learn and the other answer is it's nice to know the range so you know that perfect prediction is one and you have some idea of what 0.5 means, the RMSE (the other norm that you usually use is the RMSE) depends on the scale of the output. So for example for the housing prices, it might be interesting to see what is the standard error in terms of dollars. If you want, like something that is in the units of the output, RMSE is good or mean absolute error might even be better. If you want something that is independent of the units of the output r square is pretty good because you know it's going to be between zero and one and it's measure something like the correlation and so if it's like 0.9, you know it’s a pretty good model. If my RMSE is 10,000 I don't know if have a good model or a bad model depends on what the range of the outputs is. The last thing I want to talk about today is this was basically changing the regularization parts. The two most times regularization we looked at is Ridge which is L2 penalty, Lasso which is an L1 penalty and combining two of them which is Elastic Net. So now I want to talk about changing the first part, which was the squared loss of the predictions, basically. --- class: middle # Questions ?


ACT ADVANCED CARE PLAQUE GUARD ANTIGINGIVITIS ANTIPLAQUE CLEAN MINT- cetylpyridinium chloride mouthwash
ACT ADVANCED CARE PLAQUE GUARD ANTIGINGIVITIS ANTIPLAQUE FROSTED MINT- cetylpyridinium chloride mouthwash
Chattem, Inc.
Disclaimer: Most OTC drugs are not reviewed and approved by FDA, however they may be marketed if they comply with applicable regulations and policies. FDA has not evaluated whether this product complies.
ACT Advanced Care Plaque Guard
Antigingivits/Antiplaque Mouthwash- Clean Mint
Drug Facts
Active ingredient
Cetylpyridinium chloride 0.07%
Purpose
Antigingivitis/Antiplaque
Uses
- helps prevent and reduce plaque and gingivitis
- helps control plaque bacteria that contribute to the development of gingivitis, an early form of gum disease, and bleeding gums
Warnings
Stop
use and ask a dentist if
- gingivitis, bleeding, or redness persists for more than 2 weeks
- you have painful or swollen gums, pus from the gum line, loose teeth, or increasing spacing between the teeth. These
may be signs or symptoms of periodontitis, a serous form of gum disease.
Keep out of reach of children under 6 years of
age.
If more than used for rinsing is accidentally swallowed, get medical help or contact a Poison Control Center right away.
Directions
- use after your normal brushing and flossing routine; rinse toothpaste from mouth prior to using this rinse
adults and children 6 years of age and older:
- vigorously swish 20 mL (4 teaspoonfuls) of rinse between your teeth for 30 seconds twice a day and then spit out. Do not swallow the rinse.
children 6 to 12 years of age: adults should supervise use
children under 6 years of age: consult a dentist or doctor
Other information
- do not use if safety seal is broken or missing
Inactive ingredients
water, xylitol, propylene glycol, flavor, sucralose, PEG‑35 castor oil, dipotassium glycyrrhizate, yellow 5, green 3 (309-112)
ACT Advanced Care Plaque Guard
Antigingivits/Antiplaque Mouthwash- Frosted Mint
Drug Facts
Active ingredient
Cetylpyridinium chloride 0.07%
Purpose
Antigingivitis/Antiplaque
Uses
- helps prevent and reduce plaque and gingivitis
- helps control plaque bacteria that contribute to the development of gingivitis, an early form of gum disease, and bleeding gums
Warnings
Stop use and ask a dentist if
- gingivitis, bleeding, or redness persists for more than 2 weeks
- you have painful or swollen gums, pus from the gum line, loose teeth, or increasing spacing between the teeth. These
may be signs or symptoms of periodontitis, a serous form of gum disease.
Keep out of reach of children under 6 years of age.
If more than used for rinsing is accidentally swallowed, get medical help or contact a Poison Control Center right away.
Directions
- use after your normal brushing and flossing routine; rinse toothpaste from mouth prior to using this rinse
adults and children 6 years of age and older:
- vigorously swish 20 mL (4 teaspoonfuls) of rinse between your teeth for 30 seconds twice a day and then spit out. Do not swallow the rinse.
children 6 to 12 years of age: adults should supervise use
children under 6 years of age: consult a dentist or doctor
Other information
- do not use if safety seal is broken or missing
Inactive ingredients
water, alcohol (11%), xylitol, propylene glycol, flavor, sucralose, PEG‑35 castor oil, dipotassium glycyrrhizate, blue 1 (309-113)
Principal Display Panel
ACT
Advanced Care Plaque Guard
Antigingivits/Antiplaque Mouthwash
Kills Germs
Helps Prevent Plaque
Helps Control Gingivitis
Freshens Breath
Clean Mint
18 fl oz (532 mL)
Gingivitis Freshens Breath Clean Mint 18 fl oz (532 mL)" src="http://dailymed.nlm.nih.gov/dailymed/image.cfm?setid=5db380e6-f2f2-434d-b2d7-c19fedb6e821&name=act-advanced-care-plaque-guard-antigingivitis-anti-1.jpg">
Principal Display Panel
ACT
Advanced Care Plaque Guard
Antigingivits/Antiplaque Mouthwash
Kills Germs
Helps Prevent Plaque
Helps Control Gingivitis
Freshens Breath
Frosted Mint
18 fl oz (532 mL)
Chattem, Inc.


Getting KUnit¶
In order to use KUnit, you must first determine which branch you would like to use. What do we mean by that? We are in the process of upstreaming KUnit into the Linux kernel, so that means we have code under development that is not in the upstream repository or otherwise doesn’t match what is currently upstream.
For more information on why there are multiple versions of KUnit, see Why are there multiple versions?.
For information on deciding which version you should use see Which version should I use?.
For the actual URL you need, see Where are these versions?
Why are there multiple versions?¶
Ideally, upstreaming code to the Linux kernel would be fast, we would be able to develop features entirely on LKML, no one would use code that had not yet been accepted upstream, and everything would have been done this way from the start. Unfortunately, we do not live in a perfect world, and consequently none of these are true.
This means that there are some features that we’ve developed which aren’t yet going upstream. We’ve developed these in an experimental branch. This is basically a staging branch for features we’re still actively experimenting with, or which depend on code which isn’t yet upstream.
But wait, I see more than two branches!¶
For the upstream version, just use torvalds/master. Our experimental branch is hosted at https://kunit.googlesource.com/linux/+/kunit/alpha/master.
We will be periodically replacing the experimental branch with one based on newer upstream kernels, in order to minimise the divergence between experimental and upstream. For more details on how this will work, see<EMAIL_ADDRESS>Which version should I use?¶
As for which branch you should use, upstream should be generally preferred (just as upstreaming your own code should be generally preferred); nevertheless, there are a lot of features that are not upstream yet, and will not be upstream for some time (possibly more than a year), if you need these features for your tests then you have no choice other than to use the experimental branch.
What features are available upstream?¶
Basic test structure for defining test cases and test suites
Basic expectations and assertions
KUnit managed resources
Error reporting
What features are not upstream yet?¶
Class mocking
Function mocking
Hardware/platform faking
Error recovery
Where are these versions?¶
Upstream¶
KUnit can be found in Linux 5.5 and newer.
Code: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/
Usage - for a more detailed explanation of KUnit features
API - for the list of KUnit APIs used for testing
kunitconfig file: https://kunit.googlesource.com/kunitconfig/+/refs/heads/master/kunitconfig
The KUnit code itself is primarily found in the lib/kunit directory.
Experimental¶
The experimental branch is stored in a Google-hosted gerrit instance. Going forward, there will be a set of versioned experimental branches, of which only the latest will be actively supported. Until this is available, however, there is a historic kunit/alpha/master branch containing our experimental code.
Code: https://kunit.googlesource.com/linux/+/kunit/alpha/master.
Usage - for a more detailed explanation of KUnit features
API - for the list of KUnit APIs used for testing
kunitconfig file: https://kunit.googlesource.com/kunitconfig/+/refs/heads/kunit/alpha/master/kunitconfig


Refresh Your Ride
Client
Home
UserName
Logout
Black dot is you
Blue dot is providers
Sort Providers by:
check_box_outline_blank
Gas
check_box_outline_blank
Windshield Wiper Fluid
check_box_outline_blank
Jumper Cables
check_box
Vacuum
check_box_outline_blank
Air Fresheners
Service Providers
Driver 1
Gas
Vacuum


g2 Tests
g2.js, g2.ext.js, g2.mec.js are referenced. canvas and image should look alike.
Error at number:
#
name
src
canvas
image
error


Our device is installed! DoseNet has added James Logan High School to our network.
We had a great visit with chemistry and physics teachers, Mr. Ting and Ms. Chuang, to talk about bringing
DoseNet to their school and classrooms. Special thanks to the Calculus teacher, Mr. Prucha, for organizing this visit after meeting us during a tour of our lab with his own students! We are looking forward to future visits to talk to students in classes and science clubs about nuclear science.


Dow Futures Rise as US & China Trade Verbal Attacks
- Dow futures point to moderate gains for the US stock market on Monday.
- The US and China traded attacks over the weekend, despite allegedly being “very close” to a trade deal.
- The upcoming Thanksgiving holiday means US economic data releases will be packed into a dizzying two-day window on Tuesday and Wednesday.
Dow Jones Industrial Average futures point to more gains ahead of a holiday-shortened trading week.
With just three weeks remaining before new tariffs escalate the trade war further, relations between the US and China appear to be moving in the wrong direction.
Dow Futures Point to Gains on Monday
Futures tracking Wall Street’s three major indices all rose when the markets opened Sunday night.
Dow Jones Industrial Average futures climbed 50 points or 0.18%, implying a 63.38 point advance at the open.
S&P 500 futures rose 0.2%, while Nasdaq futures jumped 0.25% to round out a bullish pre-market session.
US & China Trade Barbs as Tariff Deadline Looms
President Donald Trump may say the US and China are “very close” to finalizing a trade deal, but the two economic superpowers continue to trade barbs three weeks before the next round of tariff hikes.
China’s Top Diplomat Torches Washington in Scathing Rebuke
Speaking at a G20 meeting in Japan, Chinese State Councillor Wang Yi torched the US as the “world’s biggest destabilizing factor.”
“The United States is broadly engaged in unilateralism and protectionism, and is damaging multilateralism and the multilateral trading system. It has already become the world’s biggest destabilizing factor,” Wang said, according to China’s Foreign Ministry.
Wang, China’s top-ranking diplomat, alleged that the US must stop playing “zero-sum games” and pursue a compromise that benefits both Washington and Beijing.
There is no way out for the zero-sum games of the United States. Only win-win cooperation between China and the United States is the right path.
Trump Adviser Warns that Hong Kong Threatens Trade Deal
Not to be outdone, the Trump administration levied its own assault on Beijing. During a visit to Canada on Saturday, US national security adviser Robert O’Brien said that a trade deal was “still…possible.” But gone was the confidence from exuding from Trump’s remarks on Friday.
Adding further pressure to the Dow, O’Brien warned Beijing that the White House would not ignore the plight of Hong Kong’s pro-democracy protesters, who continue to demonstrate under the ever-present threat of violent government suppression.
“We were hoping to have (a phase one) deal done by the end of the year. I still think that’s possible,” he said. “At the same time, we’re not going to turn a blind eye to what’s happening in Hong Kong…or other areas of the world where we’re concerned about China’s activity.”
Already, the US Congress has passed a bill that condemns China for its suppression of democracy in Hong Kong.
The Hong Kong Human Rights and Democracy Act, which received unanimous support in the Senate and all but one vote in the House, would impose sanctions on Chinese officials accused of perpetrating human rights abuses against protesters.
Trump has said that he may veto the legislation, though it should have enough support in Congress for lawmakers to override his veto. The president is concerned it could negatively affect trade negotiations, which is confirmed by Beijing’s withering response to the act’s passage.
Economic Data Releases to Watch
There are no US data releases scheduled for Monday, and due to the Thanksgiving holiday, none will drop on Thursday or Friday either. But that doesn’t mean that Wall Street will have any less data to pore over this week.
Here are the highlights (all times ET):
Nov. 26 (Tuesday)
- 8:30 am: Goods Trade Balance
- 8:30 am: Wholesale Inventories
- 9 am: FHFA House Price Index
- 9 am: S&P CoreLogic Case-Shiller Home Price Indices
- 10 am: Conference Board Consumer Confidence
- 10 am: Richmond Manufacturing Index
- 10 am: New Home Sales
Nov. 27 (Wednesday)
- 8:30 am: Durable Goods Orders
- 8:30 am: GDP
- 8:30 am: Unemployment claims
- 9:45 am: Chicago PMI
- 10 am: Core PCE Price Index
- 10 am: Personal Spending
- 10 am: Pending Home Sales
- 10 am: Personal Income
- 10 am: Crude Oil Inventories
- 11 am: Natural Gas Storage
- 2 pm: Federal Reserve Beige Book
This article was edited by Sam Bourgi.


--- layout: plain title: Home ---
I am a front-end web developer building web applications. I run events in Bournemouth for the tech community, helping to solve problems and get people working together.


Hi there. You can use this app to register and search for blogs by Australian GLAM people. We use the list of feeds to power @AusGLAMBlogs. Registrations are subject to an approval process so we can keep our feed clean.
If you have any questions or suggestions about this service contact us - we'd love to hear your ideas.