fast.ai: Deep Learning Part 2 (v2) (2018)
We start today with a deep dive into the DarkNet architecture used in YOLOv3, and use it to better understand all the details and choices that you can make when implementing a resnet-ish architecture. The basic approach discussed here is what we used to win the DAWNBench competition!
Then we’ll learn about Generative Adversarial Networks (GANs). This is, at its heart, a different kind of loss function. GANs have a generator and a discriminator that battle it out, and in the process combine to create a generative model that can create highly realistic outputs. We’ll be looking at the Wasserstein GAN variant, since it’s easier to train and more resilient to a range of hyperparameters.
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
So we're going to be talking about Ganz today, who has heard of Ganz yeah most of you very hot technology, but definitely deserving to be in the cutting edge, deep learning part of the course because they're not quite proven to be necessarily useful for anything but they're. Nearly there they're definitely going to get there and we're going to focus on the things where they're definitely definitely going to be useful in practice and there's a number of areas where they may turn out to be useful in practice. But we don't know yet. So I think the area that we're going to be that they're, definitely going to be useful in practice, is the kind of thing you see on the left here, which is, for example, turning drawings into rendered pictures. This comes from a paper. That's just came out two days ago, so there's a very active research going on right now before we get there, though, let's talk about some interesting stuff from the last class. This was an interesting thing that one of our diversity fellows Christine Payne did Christine. Has a master's in medicine from Stanford, and so she OC had an interest in thinking. What would it look like if we built a language model of medicine and one of the things that we briefly touched on back in less than four, but didn't really talk? Much about last time is this idea: you can actually seed a generative language model which basically means you've trained a language model on some corpus and then you're going to generate some text from that language model, and so you can start off by feeding it.
A few words you know to basically say here's the first few words to create the hidden state in the language model and then generate from there. Please, and so, when Christine so Christine did. Something clever which was to kind of pick up was to seed it with a question and then repeat the question. So it's 3 times Christine three times and then where'd it generate from there, and so she fed a language model, lots of different medical texts and then fed at this question. What is the prevalence of malaria and the model said in the US about 10 percent of the population? Has the virus, but only about one percent is infected with a virus about 50 to 80 million infected. She said: what's the treatment for ectopic pregnancy and it said it's: a safe and safe treatment for women with the history or symptoms may have a significant impact and clinical response. Most important factor is development and management of ectopic privacy, etc, and so what I find interesting about this is, you know it's it's pretty close to it being a to me, as somebody who doesn't have a masters in medicine from Stanford are pretty kind of close to Being a believable answer to the question, but it really has no bearing on reality whatsoever and I kind of think it's an interesting kind of ethical and user experience quandary.
So actually, I'm involved also in a company called dr., a that's trying to basically doing a number of things, but in the end provide an app for doctors and patients which can help kind of create a conversation or user interface around helping them with their medical issues. And I've been kind of continually saying to the software engineers on that team. Please don't try to create a generative model using like an LST em or something because they're going to be really good at creating bad advice. That sounds impressive.  You know people who can say with great Authority, so I think yeah. So I thought it was really. I thought it was a really interesting experiment and great to see. You know what what about diversity fellows doing? I mean this is why we have this program. I suppose I should just say masters in medicine, actually, a Juilliard trained, classical musician or on actually also a Princeton valedictorian in physics, so also a woman's a computing expert, yeah. Okay, so she does a bit of everything, so yeah, really impressive group of people and great to see such exciting kind of ideas coming out and if you're wondering you know, I've done some interesting experiments should I let people know about it. Well, I Kristin mentioned this. In the forum I went on to mention it on Twitter, too, which I got this response.
You may be wondering whose AVM aerotrain is well. He is the founder of a hot new medical, AI startup. He was previously the head of engineering at Quora before that he was the guy net flips around the data science team and built their recommender systems, and so this is what happens if you do something cool. Let people know about it and get get noticed by awesome. People like stevia, so let's talk about sci-fi 10 and the reason I'm going to talk about so far 10. Is that we're going to be looking at some more? You know: bare-bones pytorch stuff today to build these generative adversarial models. There's there's no really fast. Ai support to speak up at all Dan's at the moment. I'm sure there will be soon enough. Apparently there isn't so we're going to be building a lot of models from scratch. Now it's been a while, since we've done much, you know,
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Serious model building a little bit of model building, I guess for our bounding box staff, but really all the interesting stuff there was the loss function. So we looked at sci-fi 10 in the plat 1 of the course, and we built something which was getting about 85 % accuracy, and I remember a couple of hours to Train. Interestingly, there's a competition going on now to see who can actually train sci-fi turn the fastest going through this earth, Stepford Dawn bench and currently, so the goal is to get it to train to 94 % accuracy. So everything to see if we can build an architecture then can get four percent accuracy, because that's a lot better than our previous attempt, and so hopefully in doing so, we'll learn something about creating good architectures. That will be then useful for looking at these gams today, but I think also it's useful, because I've been kind of looking much more deeply into the last few years papers about different kinds of CN n architectures and realize that a lot of the insights in those Papers are not being widely leveraged and clearly not widely understood, so I want to show you what happens if we can leverage some of that understanding. So I've got this note book called sci-fi 10 darknet. That's because the the particular architecture we're going to look at is it's quite, is really very close to the darknet architecture, but you'll see in the process that the darknet architecture has in not the whole euro version, three end-to-end thing, but just the part of it that They pre trained on imagenet to do classification.
It's almost like the most generic simple architecture. Almost you could come up with, and so it's a really great starting point for experiments. So we're going to call it dark net, but it's not quite that net and you can fiddle around with it to create things that definitely aren't dark. Now it's really just the basis of nearly any modern ResNet based architecture. So so far 10 remember is a fairly small data set. The images are only 32 by 32 in size, and I think it's a really great data set to work with, because it's you can, you can train it. You know relatively quickly, unlike image net, it's a relatively small amount of data, unlike image net. Now it's actually quite hard to recognize the images, because 32 by 32 is it's kind of too small to easily see what's going on, so it's it's somewhat challenging. So I think it's a really underappreciated data set because it's old, you know - and you know who a deep mind or even an AI - wants to work with a small old data set when they could use their entire server room to process something much bigger. But you know to me: I think this is a really great data set to focus on so so go ahead and and kind of import, our usual stuff and we're going to try and build a network from scratch to train this with one thing that I think Is a really good exercise for anybody who's, not a hundred percent, confident with their kind of broadcasting and pytorch and so forth. Basic skills is figure out how I came up with these numbers.
Okay, so these numbers are the averages for each channel and the standard deviations for each channel insofar cap. So try and that's a bit of a homework just make sure you can recreate those numbers and see if you can do it and you know no more than a couple of lines of code. You know no loops all right. Ideally, I want to kind of do it in one: go Ken alright, because these are fairly small. We can use a larger batch size, unusual, 256 and the size of these images is 32 transformations. Normally we kind of have this standard set of sidon transformations. We used for photos of normal objects, we're not going to use that here, though, because these images are so small that trying to rotate a 32 by 32 image, a bit is going to introduce a lot of you know: blocky kind of distortions, so the kind of Standard transforms that people tend to use is a random horizontal flip and then we add size divided by 8. So 4 pixels of padding on each side and one thing which I find works really well, is by default fast. Ai doesn't add black padding, which basically every other library does. We actually take the last 4 pixels of the existing photo and flip it and reflect it, and we find that we get much better results by using this reflection padding by default. So now that we've got a 36 by 36 image, this sort of transforms in training will randomly pick a 32 by 32 crop. So we got a little bit of variation, but not he's all right, so we can use a normal from past grab our data.
So we now need an architecture, and what we're going to do is we've got to create an architecture which fits in one screen. Okay, so this is from scratch. As you can see, the only you know, I'm using the predefined come 2d veteran onto a daily key value modules, but I'm not using any blocks or anything they're all being defined. So the entire thing is here on one screen, so if you're ever wondering can I understand a modern good quality architecture? Absolutely, let's study. Let's study this one okay, so my basic starting point with an architecture is to say: okay, it's it's! It's a stacked bunch of layers and, generally speaking, there's going to be some kind of hierarchy of layers.  But, generally speaking any time you have a convolution you're, probably going to have some standard sequence and normally it's going to be con fetch norm. Then a nonlinear activation like a value right. So you know I try to start kind of right from the top by saying. Okay, what are my basic units going to be, and so by defining it here that way, I don't have to worry about. I don't have to worry about kind of trying to try to keep everything consistent, it's going to make everything a lot simpler. So here's my con flavor and so anytime, I say, come flare. I mean cons batch norm rally.
Now, I'm not quite saying value, I'm saying leaky value and that's I think we've briefly mentioned it before, but the basic idea is that normally Lu looks like that right. Hopefully, you all know that now a leaky rail you looks like that right. So this part as before, has a gradient of 1, and this part has a gradient of. It can vary, but something around point one or point zero. One is common now and the idea behind it is that, when you're in this negative zone here, you don't end up with a zero gradient, which makes it very hard to update it. In practice, people have found leaky value more useful on smaller datasets and less useful and big datasets. But it's interesting that for the Yola version 3 paper they did use early key value and got great performance from it. So it really makes things worse and it often makes things better. So it's probably not bad. If you need to create your own architecture to make that your default go to is to use leaky value, ok, you'll notice, I don't define a pipe launch module here. I just go ahead and go sequential. This is something that if you read other people's height watch code, it's really underutilized. People tend to write. Everything is applied, watch module with an inert and a forward, but if you're, if the thing you want is just a sequence of things, one after the other, it's much more concise and easy to understand to just make it a sequential right.
So I just got a simple plain function: it just returns a sequential model. Alright, so I mentioned that there's generally kind of a number of hierarchies of kind of units in most modern networks, and I think we know now that the the kind of next level in this unit hierarchy for rez nets and kind of this. This is a type of resin, it is the is the the res block or the residual block. I quoted here as Leia and back when we lasted sci-fi 10. I over simplify this. I cheated a little bit we had X coming in and we put that through a conf and then we added it back up to X to go out okay, so we ended up so but in general you know, we've got your output is equal to your input, Plus some function of your input right and the thing we did last year was: we meant we made F was a 2d conf, okay, but actually the in the real res block is actually two of them: okay, so it's actually Khan of Khan's of X, okay, and When I say conf, I'm using this as a shortcut for outcome flare - in other words, in other words, calm Vectren on real you, okay, so you can see here. I've created two coms, and here it is. I take my ex put it through the first com. Put it through the second con and add it back up to my input again to get my basic Reds block, okay, so one kind of interesting approach or one interesting insight here - is kind of what are the number of channels in these convolutions right. So we've got coming in some ni.
Some number of input channels number of inputs well number of input filters, okay, the way that the darknet folks set things up is they said: okay, we're going to make every one of these rez layers spit out the same number of channels that came in, and I Kind of liked that that's why I used it here because it makes life simpler right and so what they did is they said. Okay, let's have the first cons, have the number of channels and then the second con double it again. So ni goes to ni, / and then ni / goes to ni right. So you've kind of got this like funneling thing where, if you've got like 64 channels coming in you kind of get squished down with a first come down to 32 channels and then taken back up again to 64 channels coming out. Yes, retro, why is in place equals true in the leaky rally? Oh thanks for asking a lot of people forget this. I don't know about it, but this is a really important memory technique. If you think about it, this cone flower, it's like the lowest level thing. So pretty much everything in our resin it once it's all put together is going to be complex, complex, complex if you don't have in place, equals true. It's going to create a whole separate piece of memory for the output of the value, so like it's gon na allocate a whole bunch of memory. That's that's totally unnecessary and actually, since I wrote this, I come up came up with an another idea. The other day which I'll now implement, which is you could do the same thing for the res layer rather than going.
Let's just reorder this to say X, plus that you can actually do the same thing here yeah. Hopefully, some of you might remember that in pytorch, pretty much every function has an underscore suffix version which says do that in place. So plus there's also a add, and so that's add in place, and so that's now suddenly reduced my memory there as well. So these are, these are really handy, little tricks and I actually forgot the in place equals true at first for this. In my literally, he was having to decrease my batch size to much lower amounts, and I mean you should be possible and it was driving me crazy, and then I realized that that was missing. You can also do that with drop out by the way. If you have dropped out so drop out and all the activation functions you can do in place and then generally any arithmetic operation you can do in place as well. Why is bias usually like in ResNet set to false in the conflate yeah? So if you're watching the video pause now and see if you can figure this out right, because this is a really interesting question is like: why don't we need bias? Okay, so I wait for you to pause. Okay, welcome back! So if you figured it out, here's the thing right immediately after the Kahn is a batch not and remember. Batch norm has to learn herbal parameters for each activation.
The the kind of the thing you multiply by and the thing you add so since we're if we, if we had bias here to add - and then we add another thing here, we're adding to you things, which is totally pointless like that's two weights, where one would Do right, so if you have a batch norm after a cons, then you can, you can either say in the batch norm. Don't don't include the add bit there. Please or easier is just to say: don't include the bias in it. There's no particular harm. But again it's it's. It's gon na take more memory because that's more gradients that it has to keep track of so best. To avoid also another thing: little trick is most people's comm players have padding as a parameter, but, generally speaking, you should be able to calculate the padding basically enough right, and I see people like trying to like implement. You know special same padding modules and all kinds of stuff like that, but like if you've got a stride, one and you've got or pretty much any strategy and you've got padding of oh sorry and kernel size of three right. Then. Obviously, that's going to overlap by kind of one unit on each side, so we want padding of one or else if its stride one, then we don't need any padding. So in general, padding of kernel size, integer, divided by two: that's what you need. There's some tweaks.
Sometimes, but in this case this works perfectly well so again trying to simplify my code by having the computer calculates stuff for me rather than me having to do it myself. Another thing here with the two common players: so we we kind of have this idea of a bottleneck. This idea of reducing the channels and then increasing them again is also what kernel size we use. So here's a one by one conf right, and so this is again something you might want to pause the video now and think about. What's the one by one con really what actually happens in a one by one con. So if we've got, you know a little 4x4 grid here right and, of course, there's a filters or channels access as well - maybe that's like 32 okay and we're going to do a one by one cons. So, what's the kernel for a one by one, Kong's gon na look like it's gon na be 1 by 32 right, so remember when we talked about the the kernel size, we never mention that last piece, but let's say it's 1 by 1 by 32, because that's Part of the filters in and filters out so in other words, then what happens is this this one thing gets placed first of all here on the first cell and we basically get a dot product of that 32 deep bit with this 32 bit deep bit, and That's going to give us our first oops and that's going to give us our first output, alright and then we're going to take that 32 bit bit and put it with the second one to get the second output right.
So it's basically going to be a bunch of little dot, pallets, ok for each point in the grid. So it's what it basically is then, is a it's basically something which is allowing us to to to kind of change the dimensionality in whatever way we want in the in the channel dimension, and so that would be. That would be one of our filters right and so, in this case we're creating an AI divided by two of these
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Right so we're going to have ni divided by two of these dot products, always different, basically different weighted averages of the input channels. Okay, so it basically lets us. You know with very little computation at this additional step of calculations and nonlinearities. So that's what that's a cool trick! You know this idea of taking advantage of these one by one cons, creating this bottleneck and then pulling it out again with three by three comms. So that's actually going to take advantage of the you know. The 2d nature of the input properly real so one by one comm, doesn't take advantage of that at all. So these two lines of code there's not much in it, but it's a really great test of your understanding and kind of your intuition about what's going on. Why is it that a one by one cond, going from ni to NI over two channels, followed by a three by three conferring from ni over to de and i and i channels like? Why does it work? Why do the tensor ranks line up? Why do the dimensions or line up nicely? Why is it a good idea? What's it really doing like it's a really good thing to fiddle a fiddle around with maybe create some small ones in jupiter notebook, you know run them yourself, see what inputs now, let's come in and out. You know really get a feel for that once you've done so
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
You can then play around with different things right and there's actually one of the really unappreciated papers. Is this one wide residual networks? Okay and it's really quite a simple paper, but what they do is they basically fiddle around with with these two lines of code right and what they do is they say well what if this wasn't 2/2, but what if it was times, two like that'd, be totally Allowable all, right, that's going to line up nicely or what, if we had another comfrey after this, and so this was actually ni over to ni /, and then this is an AO. To again that's going to work right, kernel say is one three one going to half the number of kernels leave it at half and then double it again at the end, and so they come up with this kind of simple notation for basically defining what this can Look like, and then they show lots of experiments and basically what they show is that this approach of a bottlenecking of decreasing the number of channels which is like almost universal and resinates, is probably not a good idea. In fact, from the experiments definitely not a good idea, because what happens is it lets you create really deep networks right and the guys who created resinates got particularly famous recruiting a 1001 bio network? Ok, but the thing about a thousand and one layers is you can't calculate layer, two until you're finished layer 1 from jakarta calculate layer 3 until you finish calculating where so, it's sequential GPUs don't like sequential.
So what they showed is that if you have less layers fat with more active with more calculations per layer - and so one easy way to do, that would be to remove the /. No other changes all right like try. This at home try running, sci-fi and see what happens right or make even more apply it back to you or fiddle around and that basically lets your GPU do more work. And it's very interesting because the vast majority of papers that talk about performance of different architectures never actually time how long it takes to run a batch throat. Like they literally say this one requires X number of floating-point operations per batch, but then they never actually bother to run the damn thing like a proper experimentalist and find out whether it's faster or slower, and so a lot of the architectures that are really famous.  So the wide ResNet paper is unusual in that it actually times how long it takes, as does the yellow version 3 paper, which which made the same insight. I'm not sure they might have missed the wide resonance paper, because that the yellow version 3 paper came to a lot of the same conclusions, but I'm not even sure
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
They cited the wide resinous paper, so they might not be aware that all that works being done, but they're both both great to see people actually timing, things and noticing what actually makes sense. Yes, which cell you looked really hot in the paper which came out. But I noticed that you don't use it. What's your opinion on cell you so sell. You is something largely for fully connected layers which allows you to get rid of batch norm, and the basic idea is that if you use this different activation function, it's it's kind of self normalizing, that's what yes, and so it stands for so self. Normalizing means that all always remain at a unit. Standard deviation and zero mean, and therefore you don't need that pattern on it hasn't really gone anywhere and the reason it hasn't really gone anywhere is because it's incredibly finicky, you have to use a very specific initialization. Otherwise, it doesn't start with exactly the right standard deviation of mean very hard to use it with things like embeddings. If you do, then you have to use a particular kind of embedding initialization, which doesn't necessarily actually make sense for embeddings. So you know - and you do all this work very hard to get it right and if you do finally get it right, what's the point? Well, you've managed to get rid of some batch norm layers which weren't really hurting you anyway, and it's interesting because that paper that sell me a paper.
I think one of the reasons people noticed it or, in my experience the main reason people noticed it was because it was created by the inventor of LST, m/s and also it had a huge mathematical appendix and people were like lots of maths from a famous guy. This must be great, you know, but in practice I don't see anybody using it to get any state-of-the-art results or win any competitions or anything like that. Okay. So this is like some of the tiniest bits of code we've seen, but there's so much here and it's fascinating to play with so now, we've brought this block, which is built on this block and then we're going to create another block on top of that block. Okay, so we're going to call this a group layer and it's going to create a bucket. It's going to contain a bunch of rez layers and so a group layer. It's going to have some number of channels or filters coming in okay, and what we're going to do is we're going to double the number of channels coming in by just using a standard, comm flare, optionally, we'll have the grid size by using a stride of two Okay and then we're going to do a whole bunch of rez blocks a whole bunch of rez layers. We can pick how many that could be two or three or eight back, because remember these rez layers, don't change the grid size and they don't change.
The number of channels - so you can add as many as you like anywhere you like, without causing any problems, and this is going to use more computation and more RAM, but there's no reason other than that. You can't add as many as you like. So a group layer, therefore, it's going to end up doubling the number of channels, because if this initial convolution, which doubles the number of channels, its initial convolution, doubles the number of channels and depending on what we pass in a stride, it may also have the grid Size if we put straight equals to and then we can do a whole bunch of res block computations as many as we like all right so then to define our darknet or whatever. We want to call this thing. We're just going to pass in something that looks like this and what this says is create five group layers. The first one will contain one of these extra rez layers. The second will contain two then 4, then 6, then 3, and I want you to start with 32 filters. Alright, so the first one of these res res layers will contain 32 filters and they'll just be one extra rez layer, the second one. It's going to double the number of filters, because that's what we do each time we have a new group layer, we double the number, so the second one will have 64 and then 128, then 256 and then 512 and then that'll. Be it all right. So that's going to be like nearly all of the network is going to be those bunches of layers and remember.
Every one of those group layers also has one convolution of this data. Okay, and so then all we have is before that all happens. We're going to have one convolutional layer at the very start and at the very end, we're going to do our standard adaptive average pooling flatten and a linear layer to create the number of classes out at the edge. Alright. So one convolution at the end that we're pulling and one linear layer at the other end and then in the middle, these group layers each one consisting of a convolution or layer, followed by n number of resumes, and that's that's it again. I think we've mentioned this. A few times, but I'm yet to see any code out there, any any exam Falls anything anywhere that uses adaptive average cooling, everyone, I've seen rats it like this and then spits a particular number here right, which means that it's now tied to a particular image size Which definitely isn't what you want, so most people, even the top researchers I speak to. Most of them are still under the impression that a specific architecture is tied to a specific size and that's a huge problem when people think that, because it really limits their ability to like use smaller sizes to kind of kick-start their modeling or to use smaller sizes. For doing experiments and stuff like that again, you'll notice, I'm using sequential here rather than a nice way to create architectures, is to start out by creating a list. In this case.
This is a list with just one comp layer in and then my function here make group layer. It just returns another list right. So then, I can just go plus equals patter pending that list to the previous list, and then I could go plus equals to a pen this bunch of things to that list and then, finally, sequential of all those layers right. So there's a very nice thing. So now my forward is just self dot layers. Okay, so here's a kind of you know. This is a nice kind of picture of how to make your architectures as simple as possible. Okay, so you can now go ahead and create this and, as I say you can fiddle around, you know you could even parameterize this too to make it a number that you kind of pass in here's to pass in different numbers. So it's not too. Maybe it's times two: instead, you could pass in things that change the kernel size or change the number of convolutional layers you know fiddle around with it, and maybe you can create something. I've actually got a version of this which I'm about to run for you, which kind of implements all of the different parameters. That's in that wide ResNet paper, so I could fiddle around. Let's see what worked well so, once we've got that we can use confluent from bottle data to take our pipe watch model module and that a model data object and turn them into a learner. Give it a criterion, that's a metric.
So if we like, and then we can call fit in a way go, could you please explain adaptive average pooling how does setting to one work sure before I do I just want to like, since we've only got a certain amount of time in this class. I wanted to see, I do want to see how we go. You know,
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
With this simple network against these state-of-the-art results, so to make life a little easier, so we can reconstruct later so I've got the command ready to go. So we've basically take taken all that stuff and put it into a simple little Python script and I've modified some of those parameters. I mentioned to create something I've caught of wrn 22 Network, which doesn't officially exist, but it's got a bunch of changes to the parameters. We talked about based on my experiments, we're going to use the new leslie smith, one cycle thing, so there's quite a bunch of cool stuff here, so the one cycle implementation was done by our students yoga. I think i don't know how to pronounce his name exactly so where this the trains life, our experiments were largely done by brett currents and stuff like getting the half position. Floating-Point implementation integrated into fastai, was done by Andrew Shaw, so it's been a cool kind of bunch of different student projects coming together to allow us to run this, so this is going to run actually on a AWS Amazon, AWS p3, which has eight GPUs the P3 has these newer Volta architecture GPUs, which actually have special support for half precision floating point. First, AI is the first library I know of to actually integrate the volatile, optimized half position floating point into the library. So we can just go, learn half now and get that support automatically and is also the first one to integrate one cycle.
So these are the parameters for the one cycle, so we can go ahead and get this running. So what this actually does is it's using pytorches multi-gpu support, since there are eight GPUs, it's actually going to fire off eight separate Python processors and each one's going to train on a little bit and then at the end, it's going to pass the gradient updates Back to kind of the master process, that's going to integrate them all together, so you'll see here they are right. Lots of
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Progress bars will pop up together and you can see it's training. You know three or four seconds when you do it this way. Where else when I had where else when I was training earlier, I was getting. Let's see 30 epochs in, I was getting about 30 seconds per clock so doing it. This way we can kind of train things like 10 times faster or so, which is pretty cool. Okay, so we'll leave that running. So you are asking about adaptive average pooling and I think specifically is what's the number one doing so. Normally, when we're doing average pooling - let's say: we've got four by four: let's say we did average pooling, comma okay, then that creates a 2x2 area and takes the average of those four right and then we can pass in the stride all right. So if we said stride one then the next one is we look at this block of 2x2 and take that average and so forth right, so that's like what a normal to go to average pooling would be, and so that would in that case, if we didn't Have any parting that would spit out a 3x3 okay, because it's true here to here to here, okay, and if we added padding, we can make it three by three as well. That's right! So if we wanted to spit out something, we didn't want 3x3. What? If we wanted one by one right, then we could say average pull four comma four right, and so that's gon na do 4, comma, 4 and average the whole lot right and that would spit out one by one. But that's just one way to do it rather than saying the size of the the kind of appalling filter.
Why don't we instead say? Well, I don't care what the size of the input grid is. I always want one by one right. So that's where, then you say: adaptive average Hall and now you don't say: what's the size of the palling filter, you instead say what's the size of the app. What I want - and so I want something that serve one by one and if you already put a single int, it assumes you mean one by one. So, in this case, adaptive average pooling one with a four by four grid coming in is the same as average pooling for Hummer four. If it was a 7x7 grid coming in, it would be the same as 7, comma 7 right. So it's the same operation. It's just expressing it in a way that says regardless of the input. I want something of that sized output.  Well, we got 294 and it took three minutes and 11 seconds and the previous state of the art was one hour and seven minutes. So was it worth fiddling around with those parameters and learning a little bit about how these architectures actually work and not just using what came out of the box well holy. We just used a publicly available instance. We use the spot instance, so it's cost you that so that cost us, like $ 8 per hour for three minutes - cost us a few cents to train this from scratch 20 times faster than anybody's ever done it before. So that's like the most crazy state at the at result.
I think we see we've seen many, but this one just blew it out of the water right, and so you know this is kind of partly thanks to just fiddling around with those parameters of the architecture. Mainly frankly, about using Leslie Smith's one cycle thing and swampers implementation of that, and remember not only so does to remind you of what that's doing. It's basically saying this is batches right, and this is mining right. Okay, it creates an upward path. That's equally long as the downward path right. So it's a true clr triangular cycle called learning rate. As per usual, you can pick. The ratio between those two numbers right so X, divided by Y in this case is, is a lumber that you get to pick in this case. We picked 50 okay, so we started out with a much smaller one here and then it's got this cool idea, which is you get to say what percentage of your epochs then is spent going from the bottom of this down all the way down pretty much to Zero and that's what this second number here is, so 15 percent of the batches are spent going from the bottom of our triangle even further. So importantly, though, with that's not the only thing one cycle does we also have momentum right and momentum goes from point. Nine. Five to 0.85 like this, in other words, when the loading rates really low, we use a lot of momentum and when the learning rates really high, we use very little momentum, which makes a lot of sense, but until Leslie Smith showed this in that paper, I've never Seen anybody do it before, so it's a really really cool trick, so you can now use that by using the UCLA beta parameter in in fastai, and you should be able to basically replicate this state-of-the-art result.
You can use it on your own computer or your paper space. Obviously, the only thing you won't get is the multi-gpu piece, but that makes it a bit easier to Train anyway, so in a single GPU, you should be able to beat this this. This on a single GPU yeah make group layer contains stride equals two. So this means stride is one for layer, one and two for everything else. What's the logic behind it, usually the strides I have seen our odd no strides are either one or two, I think you're thinking of kernel sizes, so strata calls to means that I jump to across and so stride of two means that you have your grid size. So I think you might've cos got confused between stride and kernel size there, and so, if we have a stride of one, the grid size doesn't change. If you have a stride of two, then it does - and so in this case, because this is for say, fire 10 32 by 32 is small and we don't get to have the grid size very often right, because pretty quickly we're going to run out of cells And so that's why the first layer has a stride of one. So we don't decrease the grid size straightaway, as you play, and it's kind of a nice way of doing it, because that's why we kind of have a low number here. So we can. We can start out with you know not too much computation on the big grid, and then we can gradually doing more and more computation as the grids get smaller and smaller, but because the smaller grid, the computation, will take less time.
Okay, so I think so that we can do all of our gaining in one go: let's take a slightly early break and come back at 7:30, okay, so we're going to talk about generative adversarial networks, also known as ganz, and specifically we're going to focus on the Vasa Stein gang caper, which included some guy called seumas chintala, who went on to create some piece of software called hide watch. The Vasa Stein Gann, was heavily influenced by the Sun is going to call this w again, that's the time the DC gain or deep convolution deep convolutional generative adversarial networks paper, which also seemeth was involved with. So it's a really interesting paper to read a lot of it looks like this. The good news is, you can skip those bits because there's also a bit that looks like this, which says: do these things all right now. I will say, though, but like a lot of papers have a theoretical section, which seems to be they're entirely to get past the reviewers need for theory, that's not true with a W again paper. The theory bit is actually really interesting. Like you, don't need to know what to use it, but if you want to learn about like some some cool ideas and see the thinking behind why this particular algorithm, it's absolutely fascinating, and almost nobody before this paper came out.
I didn't know literally I'd Lou knew nobody who had studied the math that it's based on so like everybody had the method is based on ins, and so the paper does a pretty good job of laying out all the pieces. You have to do a bunch of reading yourself, so if you're interested like in digging into the deeper math behind some paper to see what it's like to study it. I would pick this one because at the end of that theory, section you'll come away saying like okay, I can see now why they made this algorithm the way it is okay and then having come up with that idea. Like the other thing is often, these theoretical sections are very clearly added after they come up with the algorithm. They'll come up the algorithm based on intuition and experiments and then later on, post hoc, justify it, or else this one. You can clearly see it's like. Okay. Let's actually think about what's going on in Ganz and think about what they need to do and then come up with the algorithm. So the basic idea of a began is it's a generative model? Okay, so it's something that is going to create sentences or create images. Kind of generate stuff right and it's going to try and create stuff, which is very hard to tell the difference between generated stuff and real stuff. That's what generative model could be used to face. What a video you know, a very well-known, controversial thing of deep fakes and fake pornography and stuff happening at the moment could be used to fake somebody's voice.
It could be used to fake the answer to a medical question, but in that case it's not really a fake right. It could be a generative answer to a medical question. That's actually a good answer right, so you like generating language. You could generate a caption to a to an image, for example, so generative models have lots of apple, interesting adaptations, but generally speaking they're, they need to be good enough that, for example, if you're using it too, you know automatically create a new scene for Carrie Fisher. In the next Star, Wars, movies and she's not around to play that part anymore, you want to, you, know, try and generate an image of her. That looks the same. Then it has to fool the Star Wars, audience into thinking like okay, that that doesn't look like some weird Carrie Fisher. That looks like the real Carrie Fisher or, if you're, trying to generate an answer to a medical question, you want to generate English. That reads, you know nicely and clearly - and it sounds authority of meaningful, so the idea of a generative adversarial Network is we're going to create not just a generative model to create, say the the the the generated image that a second model. That's going to try to pick which ones are real and which ones are generated and we're going to call them fake, okay, so which ones are real and which ones are fake. So we've got a generator, that's going to create our fake content and a discriminator.
That's going to try to get good at recognizing, which ones are real and which ones are fake, so they're going to be two models and then they're going to be adversarial, meaning the generator is going to treat a tree trying to keep getting better at falling. The discriminator into thinking that fake is real and the discriminator is going to try to keep getting better at discriminating between the real and the fake and they're going to go head to head like that right and it's basically as easy as I just described. It really is, but it's going to build two models in play: torch we're going to create a training loop that first of all says the loss function for the discriminator is. Can you tell the density, real and fake and then update the weights of that and then we're going to create a loss function for the generator which is just going to say? Can you generate something which pulls the discriminator and update the weights? True from that loss and we're going to look through that a few times and see what happens, and so let's come back to the pseudo code here, if the algorithm and let's read the real code first, so there's lots of different things you can do with Gans And we're going to do something, that's kind of boring but easy to understand and and it's kind of cool that it's even possible, which is we're just going to generate some pictures from now. We're just going to get it to draw some pictures.
Okay and specifically, we're going to get it to draw pictures of bedrooms right, you'll,
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Find if you hopefully get a chance to play around of this during the week with your own data sets, if you pick a dataset, that's very varied, like imagenet and then get again to try and create image net pictures. It tends not to do so well because it's it's not really clear enough. What do you want a picture of alright? So it's better to give it. For example, the there's a data set called celeb, a which has pictures of celebrities faces that works great with dance. You create really clear celebrity faces that don't actually exist. The bedroom data set also a good one right: lots of things, pictures of the same kind of thing. Okay, so that's just a suggestion. So there's something called the L son same classification data set. You can download it using these steps. I've also it's pretty huge, and so I've actually created a kegel data set of a 20 % sample right. So, unless you're really excited about generating bedroom images, you might prefer to grab the 20 % sample. So then we do the normal steps of creating some different paths, and in this case you know I, as we do before. I find it much easier to kind of grow the CSV route when it comes to handling our our data. So I just generate a CSV with the list of files that we want and a fake label of 0, because we don't really have labels for these.
At all and I actually create two part, two CSV files, one that contains everything in that bedroom data set and one that just contains a random 10 %. That's just nice to do that because then I can most of the time use the sample when I'm experimenting because like because there's well over a million files, just reading in the list takes a while okay. So this will look pretty familiar. So here's a con block. This is before I realized that sequential models are much better. So if you compare this to my previous con, lock with a sequential model, there's just a lot more lines of code here, but you know it does the same thing of doing con rally better on okay and we calculate our padding and here's a bias pulse. So this is the same as before, basically, but with a little bit more code, alright, so the first thing we're going to do is we're going to build a discriminator, so a discriminator is going to receive as input an image. Okay and it's going to spit out a number and the number is meant to be lower if it thinks this image is real. Okay, now, of course the what does it do for a lower number thing doesn't appear in the architecture, that'll be in the loss function. So what we have to do is create something that takes an image and spits out a number okay. So a lot of this code is borrowed from the original authors of this paper of the paper. So some of the naming scheme and stuff is different to what we're used to so sorry about that.
But hopefully I've tried to make it look at least somewhat familiar.  This is not gon na use a residual right, so it looks very similar to before a bunch of extra layers, but these are going to become players rather than res layers and then at the end, we need to append enough stride to enough strive to calm players That we decrease the size, the grid size down to be no bigger than 4x4 right. So it's going to keep using straight to divide the size by twos, tried to divide by those by two and till our grid. Size is no bigger than four, and so this is quite a nice way of like creating as many layers as you need in a network to handle arbitrary sized images and turn them into a fixed loan grid size. Yes, Rachel does again need a lot more data than say dogs versus cats or NLP, or is it comparable? You know honestly, I'm kind of embarrassed to say I am NOT an expert practitioner in games, so you know the stuff i teach in part. One is stuff. I'm happy to say I know the best way to you know pretty close the best way to do these things, and so I can show you stayed at the out results like I just did with safe at ten, with the help of some of my students.
Of course, I'm not there at all with Ganz, so I'm not quite sure how much you need like in general Pitt seems unique quite a lot, but remember the only reason we didn't need too much and dogs and cats is because we had a pre trained model And could we leverage pre-trained and models and fine-tune them? Probably I don't think anybody's done it. As far as I know, that could be really interesting thing for people that are kind of think about an experiment with. Maybe people have done it and there's some literature there? I haven't come across, so I'm somewhat familiar with the main pieces of literature and Gans, but I don't know all of it. So maybe I've missed something about transfer learning in games, but that would be the trick to not needing too much data, and so it's the huge speed-up, a combination of one cycle, learning rate and momentum, annealing, plus the eight GPU parallel training in the half precision is That only possible to do the half precision calculation with consumer GPU. Another question: why is the calculation eight times faster from single to half precision while from double the single is only two times fast? Okay, so the SyFy? Ten result: it's not eight times faster from single to half it's about two or three times as fast from single to half the Nvidia claims about the about the flops performance of the tensor cause are academically correct, but in practice meaningless, because it really depends on what Calls you need for what pieces so about two or three X improvement for half so yeah.
The half precision helps a bit the the extra GPUs helps a bit. The one cycle helps an enormous amount. Then another key piece was the playing around with the parameters that I told you about so kind of ten reading the wide ResNet paper carefully identifying the kinds of things that they found there and then writing a version of the architecture you just saw that made. It really easy for me to fiddle around with prep one action for me for Brett pans to fiddle around with parameters staying up all night, trying every possible combination of different kernel, sizes and numbers of kernels and numbers of layer groups and size of layer groups and The amount of remember we did a bottleneck, but actually we tended to focus not on bottlenecks bonds instead on widening, so we actually like things that increase the size and then decrease it, because it takes better advantage of the GPU. So, though, all those things combined together, I'd say the one cycle was perhaps the most critical, but but every one of those resulted in a big speed-up. That's why we were able to get this 30x improvement over the state-of-the-art so factor and we got some ideas for other things like after this Dorn bench finishes, and you know maybe we'll try and go even further see if we can beat one minute one day. That'll be fun okay, so so here's, okay, so here's our discriminator right.
I mean that the important thing to remember about an architecture is, it doesn't do anything rather than have some input, tensor size and rank and some output tensor size and range. So this is going to spit out. You see. The last calm here has one channel right. This is a bit different to what we're used to right, because normally our last thing is a linear block right. But our last thing here is a common block right and it's only got one channel, but it's got a grid size of something around 4 by 4, it's no more than 4 by 4, so we're gon na spit out a let's say: it's provoke or a 4 By 4 by 1 cancer, so what we then do is we then take the mean of that fat. So it goes from 4 by 4 by 1 to the scalar right. So this is kind of like the ultimate adaptive average pulling right, because we've got something with just one channel. We take the mean, so this is a bit yeah a bit different. Normally, we first do average pooling, and then we put it through a fully connected layer to get our one thing out. In this case, though, but getting one channel out and then taking the mean of with that, I haven't fiddled around with like. Why did we do it that way? What would instead happen? If we did the usual average Pauling, followed by a fully connected layer? Would it work better? Would it not I I don't know, I rather suspect it would work better if we did it like the normal way, but I haven't tried it and I don't really have a good enough intuition to know whether I'm missing something finished, the experimenter truck.
If somebody wants to stick an adaptive average Pauling layer here and a fully connected layer afterwards with a single output, it should keep working. You know we should do something. The loss will go down to see whether it works okay. So that's the discriminator right. So it's gon na be a training loop. Let's put, let's assume, we've already got a generator. Somebody says: okay, Jeremy, here's a generator, it generates bedrooms. Ok, I want you to build a model that can figure out which ones are real and which ones aren't.  I'm data set to stick a 1 or a 0 and h1 and then I'll try to get there discriminator to tell the difference. Ok, so that's gon na be simple enough, but I haven't been given a generator. I need to build one, so a generator, and we haven't talked about the last function. Yet, okay, we're just going to assume that there's some loss function. That does this thing. Okay, so a generator is also an architecture which doesn't do anything by itself until we have a loss, function and data. But what are the ranks and sizes of the tenses? Well, the input to the generator is going to be a vector of random numbers. Okay and in the paper, they call that the prior, but it's going to be a vector of random numbers.
How big? I don't know some big 64 128 right and the idea is that a different bunch of random numbers will generate a different bedroom. Okay. So that's the idea, so our our gann generator sorry has to take as input a vector and it's going to take that vector. So here's our import all right and it's going to stick it through in this case so sequential model and the sequential models going to take that vector it's going to turn it into a 2 by 2. Sorry, a turn it into a well. I rank for tensor or if we take off the batch bit around three tensor height by width by three okay, so you can see at the end here our final step here, NC number of channels, so I think that's going to have to end up being 3 Equals going to create a 3 channel image of some size, yes, Rachel in coms block forward. Is there a reason? My batch dorm comes after Lu, ie self dot batch norm, dot self dot right? No, that's not it's just what they had in the code. I borrowed from, I think that the order is reversed yeah. So again, unless my intuition about Dan's is all wrong and say some some reason need to be different to what I'm used to. I would normally expect to yeah. I would actually no sorry. I would normally expect to go rally your then batch norm that this is actually the order. That makes more sense to me, but I think the order I had in the darknet was what they used in the darknet paper.
So I don't know everybody seems to have a different order of these things and in fact most people for sci-fi 10 have a different order again, which is they actually go bien then rally then conf, which is kind of a quirky way of thinking about it. But it turns out that for often for residual blocks that works better, that's called a pre activation ResNet. So if you google for pre activation ResNet, you can see that so yeah there's there's a few, not so much papers, but more blog posts out there. Where people will have experimented with different orders of those things and yeah, it seems to depend a lot on what specific data said it is and what you're doing with, although in general, the difference in performance is small enough. You won't care unless it's through a competition. Okay, so the generator needs to start with a vector and end up with a rank. Three tensor. We don't really know how to do that yet. So, how do we do that? How do we start with a vector and turn it into a rank? Three tensor. We need to use something called a deconvolution and a deconvolution is, or as they call it, an fie torch, transposed, convolution same name, that's right same same thing, different name, and so a deconvolution is something which, rather than decreasing the grid size, it increases the grid size. So, as with all things, it's easiest to see in an Excel spreadsheet, so here's a convolution right.
We start, let's say with a four by four grid: cell: okay, with a single channel, a single photo and let's put it through a three by three kernel again with a single output filter. Okay, so we've got a single channel in a single filter kernel, and so, if we don't add any padding we're going to end up with two by two right, because that three by three you can go in one two. Three four places right can go in. One of two places across some one of two places down: if there's no padding. Okay, so there's our there's our convolution right, remember: the convolution is just the sum of the product of the kernel and the appropriate grid cell. So, there's a there's our standard 3x3 upon one channel one filter, so the idea now is. I want to go the opposite direction. I want to start with my 2x2 and I want to create a 4x4, and specifically, I want to create the same 4x4 that I started with, and I want to do that by using a convolution. So how would I do that? Well, if I have a 3x3 convolution, then if I want to create a 4x4 output, I'm going to need to create this much padding. That goes with this much padding, I'm going to end up with one two, three four by one, two, three four. You see why that is so. This filter can go at any one of four places across from four places up and down.  Then I can calculate my my error for each cell just by taking this attraction and then I can get the sum of absolute values.
They are one loss later summing up the absolute values of those errors, all right so now I could use optimization so in Excel. That's called solver to do a gradient descent. Okay, so I'm going to set that cell equal to a minimum, try and reduce my loss by changing my filter. Okay and I'll go solve okay, and you can see it's come up with a filter such that you know fifteen point seven compared to sixteen seventeen. That's right, seventeen point it. So it's not perfect right and in general you can't assume that a deconvolution can exactly create the same.  You're trying to create right, but it's it's made a pretty good attempt. Okay. So this is what a deconvolution looks like asteroid, one 3x3 deep deconvolution on a 2x2 grid cell input. Did you have a question? How difficult is it to create a discriminator to identify fake news versus real news? Well, you don't need anything special, that's just a classifier right, so you would just use the NLP classifier from previous previous to previous class and listen for right. It's there's! Nothing like it and in that case, there's no generative piece right, so you just need the data set. That says these are the things that we believe are fake news, and these are the things we consider to be real news and it should actually work very well.
 I don't know, as I say, that it's very hard very hard there's not a good solution that does that well, but I don't think anybody in our course has tried and nobody else outside our cause knows of this technique right so like there's, beaners we've as we've Learned we've just had a very significant jump in NLP classification capabilities and yeah. I mean I think it's obviously the best you could do. I think at this stage would be to generate a kind of a triage that says these things look pretty sketchy based on how they're written and then you know some human could go and in fact check them. You know I mean an NLP classifier and I run in kind of fact-check things, but it could recognize like all these. These are written in that kind of you know that kind of highly popularized style, which often fake news is written in, and so maybe these ones are worth paying attention to. I think that would probably the best you could hope for without drawing on some kind of external data sources yeah. But it's important to remember you know the discriminator is basically just a classifier and you don't need any special techniques beyond what we've already learned to do.
An LP classification, ok so to to do that kind of deconvolution in in pytorch, just say: chrome, transporters, 2d and in the normal way you say the number of input channels, the number of output channels, the kernel size, the stride, the padding the bias. So these parameters are all the same right and the reason it's called a comm transpose is because actually it turns out that this is. This is the same as the calculation of the gradient of convolution. That's that's! That's, basically, why they call it that. So this is a really nice example back on the older Theano website. That comes from a really nice paper, which actually shows you some visualizations. So this is actually the one we just saw of doing a 2x2 deconvolution. If there's a stride to, then you don't just have padding around the outside, but you actually have to put padding in the middle as well. Okay, they're not actually quite implemented this way, because this is slow to do in practice you in a different way, but it all happens behind the scenes. We don't have to worry better okay yeah this we've we've talked about this convolution arithmetic tutorial before and if you're still not comfortable with convolutions and in order to get comfortable with D convolutions. This is a great site to go to. If you want to see the paper, just Google for convolution arithmetic it'll be the first thing that comes up.
Let's do it now, so you know you've found it.
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Here it is, and so that Theano tutorial actually comes from this paper, but the paper doesn't have the animated gifs okay, so it's interesting there, no decomp block looks identical to a comp lock except it's got the word transpose written here. Okay, we just go comb for Elliot better on this before it's about input, filters, output, fit filters. The only difference is that stride 2 means that the grid size will double. Rather than half both nn comp transpose to D and n n dot up sample seemed to do the same thing: ie expand, grid, size, height and width from the previous layer. Can we say that Kampf transpose to D is always better than up sample since up sample? Is merely resizing and filling unknown unknowns by zeros or interpolation? No, you can't so there's a fantastic interactive paper on distilled pub called deconvolution and checkerboard artifacts, which points out that what we're doing right now is extremely suboptimal, but the good news is everybody else. Does it if you have a look here, could you see these checkerboard architect artifacts like it's all like that blue light, blue dark, blue light blue, you know you kind of, and so these are all from from actual papers right and basically they noticed every one of These papers, with generative models, has these checkerboard artifacts and what they realized is it's because, when you have a stride to convolution of size, three kernel, they overlap right, and so you basically get like some pixels get twice as much kind of active some grid cells.
I guess wise as much activation, and so even if you start with random weights, you end up with a check of what artifact. So you can kind of see it here right and so the deeper you get kind of the worse it gets. Their advice is actually less director than it ought to be. I've found that for most generative models up sampling is better right. So all if you do end up sample, then all it does is it's. It's basically doing pooling right. Basically, but it's kind of its kind of the opposite of Pauline right. It says: let's replace this one pixel or this one grid cell with four two by two and there's a number of ways to up sample. One is just to kind of copy it across to those four and other is to use kind of lit by linear or bicubic interpolation. There are various techniques to kind of try and create a smooth off sample version, and you can pretty much choose any of them in pytorch. So if you do a two by two up sample and then a regular stride, one three by three khans, that's like another way of doing the same kind of thing as a commons plus. Well, it's it's doubling the grid size and doing some convolutional arithmetic on it, and i found for generative models and it pretty much always works better and in that distorter pub publication. They kind of indicate that maybe that's a good approach, but they don't just come out and say just do this, whereas i would just say just do this.
Having said that for gains, i haven't had that much success with it yet, and I think it probably requires some tweaking to get it to work, as I'm sure some people have got it to work. The the issue, I think, is that in the earliest ages it doesn't create enough noise ahead. I don't have it here. I had a version actually where I tried to do it within an up sample and you could kind of see that the noise didn't look very noisy so anyway, it's an interesting question, but um next week, when we look at style, transfer and super-resolution and stuff, I Think you'll see and ended up sample really comes into its own okay, so the generator we can now basically start with the vector we can decide and say, like okay, does not think of it as a vector, but actually it's a one by one grid cell and Then we can turn it into a four by four and they by eight and so forth, and so that's why we have to make sure it's a it's a suitable multiple, so that we can actually create something of the right size. And so you can see it's doing the exact opposite as before, right it's making the cell size smaller and smaller by two at a time as long as it can sorry, bigger and bigger, so this cell size, bigger and bigger, as long as it can until it Gets to half the size that we want and then finally we add one more on at the end.
Sorry, we add n more on at the end of just with no stride and then we add one more calm transpose to finally get to the size that we wanted and we're done. Finally, we put that through a fan and that's got a force us to be in the zero to one range, because of course we don't want to spit out arbitrary size, pixel values. Okay, so that's so, we've got to generate our architecture, which spits out an image of some given size, with the correct number, with the correct, whatever a readout of all correct number of channels and with values between zero and one. So at this point we can now create our model data object. These things take a while to train, so I just made it 128 by 128. So this is just a convenient way to make it a faster and that's going to be the size of the input, but then we're going to use transformations to turn it into 64. By 64, okay, there's been more recent advances which have attempted to really increase this up to kind of like high resolution sizes, but they still tend to require either like a batch size of 1 or like lots and lots of GPUs or whatever. So we're kind of trying to do things that we can do 1 conceal all consumer, GPUs yeah, so here's an example of one of the 64 by 64 bedrooms. Okay, so we're gon na do if pretty much everything manually. So let's go ahead and create our two models: our generator and our discriminator and, as you can see, they're DC can so, in other words, they're. The same modules that came up were appeared in this paper.
So, if you're interested in reading the papers, you it's well worth going back and looking at the DC gain paper to see what these architectures are, because it's assumed that when you read the vasa stein gain paper that you already know that, yes, you don't. We use a sigmoid if we want values between 0 and 1, I always forget which ones which okay so sick - yes, so sigmoid is 0 to 1 fan is 1 to minus 1. I think what will happen is I'm gon na have to check that? I beg you remember thinking about this when I was writing this notebook and realizing that 1 2 minus 1 made sense for some reason, but I can't remember what that reason was now. So let me get back here about that during the week and remind me if I forget it's good question. Thank you. Ok, so we've got our generator in a discriminator, so we need a function that returns a prior vector, so a bunch of noise. So we do that by creating a bunch of
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Zeros NZ is the size of Zed, so like very often in our code. If you see a mysterious letter, it's because that's the letter they used in the paper that says ed is the size about noise, vector okay, so there's the size of our noise vector and then we use a normal distribution to generate random numbers inside that and that Needs to be a variable because it's going to be participating in the in the gradient updates, so here's an example of creating some noise, and so here are four different pieces of noise. Okay, so we need an optimizer in order to update our gradients in the vasa stein game paper. They told us to use rmsprop. So that's fine! That's! So! When you see this thing saying, do an rmsprop update in a paper. That's nice! We can just do an rmsprop update with pipe watch, okay and they suggested a learning rate of five Enid five. I think I found one a nigga four seem to work, so I was made a little bit bigger. So now we need a training loop, and so this is the thing that's going to implement this algorithm, so a training loop is going to go through some number of epochs that we get to pick. So that's going to be a parameter, and so remember when you do everything manually, you've got to remember all the manual steps to do so. One is that you have to set your modules into training mode when you're training them and into evaluation mode when you're evaluating them, because in training mode batch norm updates happen and dropout, happens in evaluation mode.
Those two things get turned off: okay, so it's basically difference. So put it into training mode, we're going to grab a iterator from our training data loader we're going to see how many steps we have to go through and then we'll use two qdm to give us a progress bar and then we're going to go through that. Many steps, okay, so the first step of this algorithm is to update the is to update the discriminator. So in this one I'm just trying to remember yes, they don't call it a discriminator, they call it a critic right. So W are the weights of the of the critic, so the first step is to train our critic a little bit and then we're going to train our generator a little bit, and then we go go back to the top of the loop right. So this inner. So we got a while loop on the outside okay, so here's our while loop on the outside and then inside that there's another loop for the critic and so here's a little loop inside that for the critic. Okay, we call it a discriminator. So what we're going to do now is we're going to try. We've got, we've got a generator and at the moment it's random right, so our generator is going to generate stuff. That looks something like this right, and so we need to first of all teach our discriminator to tell the difference between that and a bedroom right, which shouldn't be too hard. You would hope so we just do it in basically the usual way, but there's a few little tweaks.
So, first of all, we're going to grab a mini batch of real bedroom photos, so we can just grab the next batch from our iterator turn it into a variable. Okay, then we're going to calculate the loss for that right. So this is going to be. How much does the discriminator think this looks?  Okay and that's going to create a mini batch of fake images, okay and so then we'll put that, through the same discriminator module as before, okay to get the loss for that. So how fake to the fake ones look remember when you do everything manually, you have to zero the gradients in in your loop and, if you've forgotten about that, go back to the part. One lesson where we do everything from scratch. So now, finally, the total discriminator loss is equal to the real loss, the fake loss, okay, and so you can see that here they don't talk about the loss. They actually just talk about one of the gradient updates, so this here is a symbol for get the gradients alright, so inside here is the loss right and like trying to like learn to throw away in your head all of the boring stuff. So when you see sum over m / M, that means take the average so just throw that away and replace it with NP, don't mean in your head. There's another NP domain right.
So you want to get quick at like being able to see these common idioms. So anytime, you see one over m sum over m. You go okay and peed on me right so we're taking the mean of and we're taking the mean of. So that's all fine X, I what's X, I it looks like it's next to the power of I, but it's not right. The math notation is very overloaded. They showed us here what X I is, and it's a set of M samples from a batch of the
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Real data, so, in other words this is a mini batch right. So when that, when you see something same sample, it means just grab a row grab a row grab a row and you can see here, grab it m times and we'll call the first row X, parentheses, 1, the second row X, parentheses, 2. One of the annoying things about math notation is the the way that we index into arrays is everybody uses different approaches, subscript superscript things in brackets, combinations, commas square brackets, whatever right so you've just got to look in the paper and be like ok at some point. They're going to say, take the I throw from this matrix or the ithe image in this batch. How are they going to do it in this case? Let's say superscript in parentheses? Okay, so that's all sample means and curly brackets means it's just a set of them. This little squiggle, followed by something here, means according to some probability, distribution, and so in this case like and very very often in papers, it simply means hey. You've got a bunch of data right grab a bit from it at random. Okay, so that's like that's the the probability distribution of the data you have is the data you have right, so this says: grab em things at random from your real data. This says: grab em things at random from your prior samples, and so that means, in other words, call create noise to create m random vectors. So now we've got em real images. Each one gets put through our discriminator. We've got em bits of noise.
Each one gets put through our generator to create em generated images. Each one of those gets put through. Look FW, that's the same thing, so these ones of those gets put through our discriminator to try and figure out whether they're, fake or not. And so then it's this minus this and the mean of that and then finally get the gradient of that in order to figure out how to use our own s prop to update our weights using some learning rate. Okay, so in height or CH, we don't have to worry about getting the gradients. We can just specify their last bit. Okay and then just say: lost, stop backward discriminator, optimizer, diet, step. Okay, now there's one key step right, which is that we have to keep all of our activations, so all of our weights, which are the parameters in a page horch module in this small range between 0.01 negative 0.01 and point out one. Why? Because the mathematical assumptions that make this algorithm work only only apply in like a small ball all right, so I'm not going to tell I don't. I think it's kind of interesting to understand the math of why that's the case, but it's very specific to this one paper and understanding it won't help. You understand any other paper, so only study it. You know if you're interested in you know, I think it's nicely explained. I think it's fun, but it won't be information that you're reuse elsewhere.
Unless you get super into dance I'll also mention after the paper came out and improved froster stein, Gann came out. That said, hey there are better ways to ensure that your that your weight space is in this tight ball, which was basically that kind of penalize gradients that are too high, so actually nowadays, they're at there is slightly different ways to do this anyway. That's why this line of code there it's kind of the key contribution this you know this one line of code actually is the one line of code. You add to make it a versus time again, basically that the work was all in knowing like that. That's the thing that you can do that makes everything work better. Ok, so, at the end of this, we've got a discriminator that can recognize an industry in real bedrooms and our totally random crappy generated images. So, let's now try and create some better images so now set trainable discriminator to false set trainable the generator to true zero out the gradients of the generator, and now our loss again is fw. That's the did that remember. That's the discriminator of the generator applied to some more random noise. Okay, so here's our random noise, here's our generator, here's our discriminator. I think I can remove that now because I think I've put it inside the discriminator, but I won't change it now, because it's going to confuse me so it's
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
Exactly the same as before, where we did generator on the noise and then pass a discriminator, but this time the thing that's trainable is the generator, not the discriminator, so in other words, in this pseudocode, the thing they update is theta, which is the generators parameters rather Than W, which is the discriminators parameters - and so hopefully you'll see now that this this W down here is telling you these are the parameters of the discriminator. This theta down here is telling you. These are the actually better these. This theta here is telling you. These are the parameters of the generator okay, again, it's not as universal mathematical, notation, it's a thing they're doing in this particular paper, but it's kind of nice. When you see some some suffix like that, there's like try to think about what it's telling you. Okay, so it takes noise, generate some images, try and figure out if they're, fake or real, and use that to get gradients with respect to okay, the generator as opposed to earlier. We got them with respect to the discriminator and use that to update our weights with our s prop with an alpha learning rate. Okay, you'll see that it's kind of unfair that the discriminator is getting trained and critic times which they set to five for every time that we train the generator once and the paper talks a bit about this.
But the basic idea is like there's no point making the generator better if the discriminator doesn't know how to discriminate yet okay. So that's why we've got this while loop and here's that v right and see something which was added. I think in the later paper, or maybe a supplementary material - is the idea that from time to time and a bunch of times at the start, you should do more steps at the discriminator so kind of make sure that the discriminator is pretty capable from time to Time: okay, so do a bunch of epochs of training, the discriminator a bunch of times to get better at telling the dentistry in real and fake and then do one step of making the generator being better at generating. And that is an epoch. And so let's train that for one epoch and then let's create some noise, so we can generate some examples actually going to do that later. Let's first of all decrease the learning rate by 10 and do one more pass. So we've now done two epochs and now let's use our noise to pass it to our generator, okay and then put it through our denormalization to turn it back into something we can see and then plot it, and we have some bedrooms: okay, there's not real bedrooms And some of them don't particularly like bedrooms, but some of them look a lot like bedrooms. So that's that's the idea.
Okay, that's again, and I think, like the best way to think about again, is it's like an underlying technology that you'll probably never use like this, but you'll use in lots of interesting ways. For example, they're going to use it to create now a cycle game and we're going to use the cycle Gann to turn horses into zebras. You could also use it to turn mono prints into photos or to turn photos of Yosemite in summer into winter. So it's gon na be pretty yes, Rachel, two questions, one. Is there any reason for using rmsprop, specifically as the optimized optimizer, as opposed to Adam? I don't remember it being explicitly discussed in the paper. I don't know if it's just experimental or the theoretical reason yeah have a look in the paper and see what it says I don't recall and which could be a reasonable way of detecting overfitting, while training or of evaluating the performance of one of these Gann models. Once we are done training, in other words, how does the notion of train validation test sets translate to Ganz? That's an awesome question and there's a lot of people who make jokes about how Cannes is the one field where you don't need a test set and people take advantage of that by making stuff up and saying it looks great. There are some pretty famous problems with with ganz.
One of the famous problems with ganz is called mode collapse and mode collapse happens where you look at your bedrooms and it turns out that there's basically only three kinds of bedrooms - that every possible noise vector, mapped or you look at your gallery and it turns at All that turns out they're all just the same thing or there's just three different things. Mode collapse is easy to see. If you collapse down to a small number of modes, like you know three or four, but what if you have a mode collapse down to 10,000 modes, so there's only 10,000 possible bedrooms that all of your noise vectors collapse to that's not like you, wouldn't be able To see it here right because it's pretty unlikely, you would have two identical bedrooms out of 10,000 or what? If every one of these bedrooms is basically a direct copy of what? If the basically had never memorized some input, you know, could that be happening and the truth is most papers, don't do a good job or sometimes any job of checking those things. So the question of like how do we evaluate ganz, and even like the point of like hey, maybe we should actually evaluate games properly, is something that is not widely enough understood even now, and some people are trying to
(autogenerated subtitles follow, may contain gibberish/bad format - please proofread to improve - remove this note once proofread)
You know really push so Ian Goodfellow who a lot of you will know, because he came and spoke here at a lot of the book club meetings last year and of course was the first author on the most famous deep learning book. He is the inventor of games and he's been sending a continuous stream of tweets, reminding people about the importance of testing dance properly. So yeah. If you see a paper that claims exceptional, gaen results, then this is definitely something to look at. You know is: have they talked about mode collapse? Have they talked about memorization, okay, so um? This is going to be really straight forwards because it's just a neural net right. So all we're going to do is we're going to create an input containing lots of zebra photos and with each one we'll pair it with an equivalent horse photo and we'll just train a neural net. That goes from one to the other, or you could do the same thing for every Monet painting create a dataset containing the photo of the place. Oh wait: that's not possible, because the places that Monet painted aren't there anymore and there aren't exact zebra versions of horses and oh wait. How the hell is this kind of work. This seems to break everything we know about what neural nets can do and how they do them. Alright, make sure you know ask me a question. Just spoil a whole train of thought. Come on very good.
Can Ganz be used for data augmentation yeah? Absolutely you can use a gain for data augmentation should you I don't know like there are some papers that try to do semi-supervised learning with ganz I haven't found any that are like particularly compelling showing state-of-the-art results on really interesting data sets that have been widely studied. I'm a little skeptical and the reason I'm a little skeptical is because, in my experience, if you train a model with synthetic data, the neural net will become fantastically good at recognizing. The specific problems of your synthetic data and that'll be ended up what it's learning from, and there are lots of other ways of doing semi-supervised models which do work. Well. There are some places that can work. For example, you might remember Octavio gawd created that fantastic visualization in part, one of like the zooming ComNet, where it kind of showed or let her go through em list. He at least at that time had the was the number one autonomous remote-controlled car. Okay, I mean in in autonomous remote control car competitions and he trained his model using synthetically, augmented data where he basically took real videos of a car driving around the circuit and added like fake people and fake other cars and stuff. Like that - and I think that worked well because well a because he's kind of a genius and B, because I think he had a kind of a world to find kind of little subset that he had to work in but yeah in general. It's really hard.
It's really really hard to use synthetic data. I've tried using synthetic data and models for decades now, obviously not ganz cuz they're pretty new, but in general it's very hard to do very interesting research question all right, so somehow these folks at Berkeley created a model that can turn a horse into a zebra. Despite not having any photos unless they went out there and painted horses and took before-and-after shots, but I believe they didn't right, so how the hell did they do this? It's it's kind of it's kind of genius. I will say the person I know who's doing the most interesting practice of cycle. Gann right now is one of our students, Elena sarin she's, the only artist I know of who was a psychic, an artist. Here's an example: I love she created this little doodle in the top of left and then trained a psycho Gann to turn it into this beautiful painting in the bottom row. Here's some more of her amazing works, and I think it's really interesting, like I I mentioned at the start of this class that, like Dan's, are in the category of like stuff: that's not there yet, but it's nearly there and in this case, like there's at least One person in the world now who's creating beautiful and extraordinary artworks using gas and there's lots of pairs, specifically cycle games and there's actually like at least maybe a dozen people I know of who are just doing interesting.
Creative work with neural nets more generally and the field of creative area is going to expand dramatically, and I think it's interesting with Elena right. I mean I don't know her personally, but from what I understand of her background.   Combining these two by saying gosh. I wonder what this particular tool could bring to my art, and so, if you follow her Twitter account we'll make sure we add it on the wiki. Somebody can find it as for Lena sarin sarin. She basically posts a new almost every day and they're, always pretty amazing. So here's the basic trick. Okay - and this is from the cycle Gann paper - we're going to have to kind of two images assuming we're doing this with images right, but the key thing is they're, not paired images. So we're not.  One horse grab one zebra: okay, we've now got an X, so X, let's say X, is horse and Y is zebra. We're going to train a generator and what they call here. A mapping function that turns horse into zebra, we'll call that mapping function, G and we'll create one mapping function, generator that turns a zebra into a horse and we'll call that F well create a discriminator. Just like we did before, which is going to get as good as possible at recognizing real from fake horses, so that'll be DX and then another discriminator which is going to be as good as possible and recognizing real from fake zebras I'll call that dy okay.
So that's kind of our starting point, but then the key thing to making this worse work. Okay, so we're kind of generating a loss function here right. Here's one bit of the loss function here. The second bit of the loss function we're going to create something called cycle, consistency loss which says after you turn your horse into a zebra with your G generator and check whether or not I can recognize that it's real. So I keep forgetting which one's false and which one zebra I apologize. I forget my X's and Y's backwards, though my horse into a zebra and then going to try and turn that zebra back into the same horse that I started with okay and so then I'm going to have another function. That's going to check whether my this horse, which are generated knowing nothing about generated entirely from this zebra, is similar to the original horse or not right. So the idea would be if your generated zebra doesn't look anything like your original horse. You've got no chance of turning it back into the original horse, so a loss which compares X hat to X is going to be really bad unless you can go into Y and back out again and you're, probably only going to be able to do that. If you're able to create a zebra that looks like the original horse so that you know what the original horse looked like and vice versa, take the original.
Take your zebra turn it into a fake horse and check that you can recognize that and then try and turn it back into the original zebra and check that it looks like the original so notice here. This F right is our zebra two horse. This G is our horse, two zebra right, so so the G and the F are kind of doing two things they're both turning the original horse into the zebra and then turning the zebra back into the original horse. Okay, so notice that there's only two generators right: there isn't a separate generator for the reverse mapping. You have to use the same generator that was used for the original mapping. Okay, so this is the cycle consistence you lost, and I just think this is like this is genius. You know like the idea that this is a thing that could be even be possible honestly when this came out. It just never occurred to me as a thing that I could even try and solve. It seems so obviously impossible and then the idea that you can solve it like this. I just think it's it's so damn smart. So it's good to look at the equations in this paper because they're just good example like they're written pretty simply I you know, there's it's not like some of the stuff in the fastest time game paper, which is just like lots of theoretical proofs and whatever else In this case they're, you know they're just equations that just lay out what's going on and you really want to get to a point where you, where you can read them and understand so like let's kind of start talking through them.
So we've got a horse and a zebra okay. So for some mapping function, G, okay, which is our horse zebra mapping function. Then there's AG and loss right, which is the bit we're already familiar with. It says I've got a horse, a zebra, a fake zebra recognizer and a horse zebra generator. Okay and the loss is except it's what we saw before it's our ability to draw one zebra out of our zebras okay and recognize whether it's real or fake, okay and then generate a take a horse and turn it into a zebra and recognize whether that's real Or fake, okay and then you're, you then do one minus the other, and in this case they've got a log in there. The logs not terribly important. So this is this is the thing we just saw. That's that's why we did for sustained gain. First is this is just a standard gann loss in math form? Did you have a question right wrong? All of this sounds awfully like translating in one language to another, then back to the original, have gans or any equivalent been tried in translation. Not that I'm not that I know of , yeah yeah, because there's the Unversed is this: this unsupervised machine translation, which does kind of do something like this, but I don't. I haven't looked at it closely enough to know if it's nearly identical or if it's just vaguely similar yeah did so to kind of back up to what I do know normally with translation, you require this kind of paired input.
You require parallel text, so you know this is the French translation of this English? I dunno there's been a couple of recent papers that show the ability to create good quality translation models without paired data. I haven't implemented them and I don't understand anything I haven't implemented but yeah. They may well be doing the same basic idea, we'll look at it during the week and get back to you. Okay, all right, so we're going to again loss. The next piece is the cycle, consistency, loss right, and so the basic idea here is that we start with our horse, use our zebra generator on that to create a zebra use our horse generator on that to create a horse and then compare that to the original Horse and this double lines were the one we've seen this before this is the l1 loss? Okay, so this is the sum of the absolute value of differences. Where else. If this was a turn, it would be the l2 loss or the two norm, which would be the sum of squared differences, the square root of it actually and again. We now know this squiggle idea, okay, which is from our horses, grab a horse. Okay, that's so this is what we mean by sample from a distribution. There's all kinds of distributions, but most commonly in these papers were using an empirical distribution. In other words, we've got some rows of data grab a row okay.
So when you see this thing, squiggle other thing this thing here when it says P data that means grab something from the data and we're going to call that thing X. So from our horses pictures grab a horse turn it into a zebra turn it back into a horse, compare it to the original and some of the absolute values. Okay, do that for horse to zebra. Do it for zebra to horse as well add the two together, and that is our cycle consistency. You lost okay, so now we get our loss function and the whole loss function depends on our horse generator a zebra generator our horse, recognizer, our zebra recognizer discriminator and we're going to add up the gain loss for recognizing horses, the gain loss for recognizing zebras and The cycle cycle consistency, loss for our two generators, okay and then we've got a lambda here, which hopefully we're kind of used to this idea. Now that is when you've got two different kinds of loss. You chuck in a parameter there. You can multiply them by so they're of account about the same scale, okay, and we did a similar thing with our bounding box loss compared to our classifier loss, when we did that localization stuff. Okay, so then we're going to try to for this map for this loss function maximize the capability of the discriminators are to discriminate discriminating whilst minimizing that for the generators, so the generators and the discriminators are going to be facing off against each other.
So when you see this min max thing in papers, you'll see it a lot. That means it basically means this idea that in your training loop, one thing is trying to make something better. The other is trying to make something worse and you generally there's lots of ways to do it, but most commonly you'll alternate between the two and you'll often see this just referred to in math papers as minnows. Okay, so we see min max, you think you should immediately think okay adversarial training. So let's look at the code and we're only going to probably might be able to finish this today, but we're going to do something almost unheard of which is. I started looking at somebody else's code and I was not so disgusted that I threw the whole thing away and did it myself. I actually said I quite like this. I like it enough, I'm going to show it to my students, so this is where the code comes from. So this is one of the people that created the original code for cycle games and they've created a high-torque version, and I had to clean it up a little bit, but it's actually pretty damn good. It's I think, the first time I found code that I didn't feel the need to rewrite from scratch before I showed it to you, and so the cool thing about this is one of the reasons I liked doing it this way of like finally, finding something: that's Not awful is that you're now going to get to see almost all the bits of fastai or, like all the relevant bits of fastai written in a different way about somebody else right and so you're going to get to see like oh how they do.
Data sets and data loaders and models and training, loops and so forth. Okay, so you'll find there's a C game directory, which is basically nearly this with some cleanups, which I hope to submit as a PR sometime. It was written in a way that unfortunately made it a bit over connected to how they were using it as a script, but I so cleaned it up a little bit, so I could use it as a module, but other than that. It's pretty similar. So C can is basically their code copied from their from their github repo, with some minor changes. So the way the second mini library has been set up is that the configuration options they're assuming are being passed into like a script. So they've got this train options. Parser method, and so you can see I'm basically past passing in an array of like script options: okay, where's, my data: how many threads do I want to drop out? How many iterations? What am I going to call this model rich GPU? Okay? So that gives me a opt object which you can then see. Well, you know what it contains. You'll see that it contains some things I didn't mention. That's because it's got defaults for everything else that I didn't mention. Okay, so we're going to, rather than using fastai stuff, we're going to largely use CG and stuff. So the first thing we're going to need is a downloader, and so this is also a great opportunity for you again to practice your ability to navigate through code with your editor or IDE of choice, so we're going to start with create data loader. So you should be able to go, find symbol or in vim tag, to jump straight to create data loader, and we can see that's creating a custom data set loader and then we can see custom data set.
Loader is a base data loader, okay, so that doesn't really do anything. It creates okay. So basically we can see that it's going to use a standard, pytorch data loader. So that's good, and so we know if you're, to use a standard plate or data loader. You have to pass it a data set and we know that a data set is something that contains a length and a an indexer. So, presumably, when we look at create data set, it's going to do that here is create data set okay, so this library actually does more than just cycle gain it handles both aligned and unaligned image pairs. We know that our image pairs are unaligned, so we're going to an online data set okay here it is and as expected, it has a get item and a length good, and so the main, obviously the main the length is just whatever of our. So a and B is our horses and zebras got two sets. So, whichever one is longer is the length of the data loader and so getitem is just going to go ahead and randomly grab something from each of our two horses and zebras. Open them up with pillow or PIL, run them through some transformations, and then we could either be turning horses into zebras or zebras into horses. So there's some direction and then I'll just go ahead and then return our horse and our zebra and our path to the boss and the path of zebra so yeah. Hopefully, you can kind of see that this is looking pretty similar to the kind of stuff that fastai does first day.
I obviously does quite a lot more when it comes to transforms and performance and stuff like this, but you know remember: this is like research code for this one thing like it's pretty cool that they did all this work, so we've got a data loader, so we Can go and load our data into it, and so that'll tell us how many mini batches are in it. That's the length of the data loader in Piper watch next step. We've got a data. Loader is to create a model, so you can go to go tag for create model there. It is okay, same idea: we've got different kinds of models, so we're gon na be doing a cycle gain. So here's our cycle gain model. Okay, so there's quite a lot of stuff in a cycle gain model, so let's go through and find out what's going to be used, but basically at this stage we've just called initializer, and so when we initialize it, you can see it's going to go through and It's going to define two generators, we're just not surprising a generator for our horses and a generator for a zebras here. Okay, there's some way for it to generate a pool of fake data and then here we're going to grab our our Dan loss and, as we talked about our cycle, consistency loss is an l1 loss. That's interesting, they're going to use Adam so sleepy, recycle cans can Adam works pretty well, and so then we're going to have an optimizer for our horse, discriminator and optimizer for our zebra discriminator and an optimizer for our generator.
Okay, the optimizer for the generator is going to contain the parameters both for the horse generator and the zebra generator all in one place. So, okay, so the initializer is going to set up all of the different networks and wasp functions. We need and they're all going to be stored inside this model, and so then it prints out and shows us exactly the piped watch bottles we have, and so it's interesting to see that they're using resonance, and so you can see the resinates look pretty familiar. We've got con vet norm rally, you can fetch norm so instance. Norm is just the same as batch norm. Basically, but it applies it to one image at a time and the difference isn't particularly important. Okay and you can see they're doing on reflection, padding just like we are, so you can kind of see like when you when you try to build everything from scratch like this. It is a lot of work and you know you can kind of forget the little.  You kind of have to do all of them by hand, and only you end up with a subset of them. So you know over time, hopefully soon we'll get all of this Gans stuff into fast. Ai and it'll be nice and easy okay, so we've got our model and remember the model contains the loss functions. It contains the generators it contains, the discriminators all in one convenient place, so I've gone ahead and kind of copied and pasted and slightly refactored the training loop from their from their code so that we can run it inside the notebook.
So this one a lot pretty familiar right: loop to go through each epoch and a loop to go through the dealer. Before we did this, we set up a tower. This is actually not a play torch data set. I think this is what they used slightly confusingly to talk about their you know their combined. What we would call a model data object. I guess all the data that they need whip through that with TQ DM to get a progress bar and so now we can go through and see what happens in the model right so set input so set input. So so it's kind of a different approach to what we do in fastai. This is kind of neat. You know it's quite specific to cycle games, but basically, internally inside this model. Is this idea that we're going to go into our data and grab? You know we're knowing when we're either going horse to zebra or zebra to horse, depending on which way we go. We are the you know, a is over the horse or the zebra and vice versa, and, if necessary, put it on the appropriate GPU and then grab the appropriate paths. Okay. So the model now has a mini batch of horses and a mini batch of zebras, and so now we optimize the parameters. Okay, so it's kind of nice to see it like this. You can see each step right. So, first of all try to optimize the generators. Then try to optimize the horse discriminator then try to optimize the zebra discriminator. Zero grad is part of pipe torch.
Step is part of pytorch, so the interesting bit is the actual thing that captain twitch does the backpropagation on the generator. So here it is and let's jump to the key pieces, there's all the bits or the formula that we basically just saw yeah the paper. So let's take a horse and generate a zebra. So we know what a fake zebra and let's now use the discriminator to see if we can tell whether it's fake or not okay, so it spread fake and then, let's pop, that into our loss function, which we set up earlier to see if we can basically To get a loss function based on that prediction, then let's do the same thing to do. The gain loss so take a go in the opposite direction and then we need to use the opposite discriminator and then put that through the loss function again and then let's do the cycle. Consistence, you loss. Okay, so again we take our fake, which we created up here, okay and try and turn it back again into the original and then let's use that last function of cycle consistency, loss function. We created earlier to compare it to the real original and here's that lambda right, so there's some weight that we used and that would set up. Actually, we just use the default that I suggested in there options and then do the same for the opposite direction and then add them all together do the backward step and that's it.
So we can then do the same thing for the first discriminator and since basically, all the works being done now, there's much less to do here, okay, so there. That is so. I won't step all through it, but it's basically the same same basic stuff that we've already seen so optimized parameters basically is calculating the losses and doing the optimizer step from time to time, save and print out some results and then, from time to time, update the Learning rate so they've got some learning rate annealing built in here as well, isn't very exciting, but okay, so they've basically got some kind of like fastai they've got this idea of schedulers, which you can then use to update your learning rates. So I think kind of, like you know, for those of you, are interested in better understanding, deep learning, api's or interested in contributing more to fastai or interested in like creating. You know your own version of some of this stuff in some different back-end. It's cool to like look at a second kind of API that covers some subset, that some of the similar things to get a sense for how are they solving some of these problems and what are the similarities and what are the differences? So we train that for a little while and then we can just grab a few examples, and here we have them.
Again, here's a hot zebra into a horse back of a zebra, it's kind of thrown away its head for some reason, but not so much a quick get it back again. This is a really interesting one like this is obviously not what zebras look like, but if it's going to be a zebra version of that horse, it's also interesting to see it's foliar situations. I guess it doesn't very often see. Basically, just an eyeball have no idea how to do that one. So some of them don't work very well. This one's done a pretty good job, this one's interesting. It's done a good job with that one in that one, but for some reason the one in the middle didn't get go yeah this one's a really weird shape, but it's done a reasonable job of it. This one looks good this one's pretty sloppy again for just a head up there, so you know I didn't it took me quite a bit for me, like 24 hours, to train it even that far so it's kind of slow - and I know Hellena - is constantly complaining On Twitter about how long these things take, I don't know how she's so productive with them so yeah. I will mention one more thing that just came out yesterday, which is there's now a multi-modal image to image translation of unpaired, and so you can basically now create different cats, for instance from this dock. So this is basically not just creating one example of the output that you want, but creating multiple ones.  This is the paper yeah.
So this came out like yesterday or the day before. I think I think it's pretty amazing captain dog, so you can kind of see how this technology is developing and I think you know if you, oh there's so many opportunities to you know, maybe do this with music or speech or writing or to create kind of Tools for artists or alright thanks everybody and see you next week, [ Applause, ]